oeqa/core/case.py: Add OEPTestResultTestCase for ptestresult helpers
Add the OEPTestResultTestCase class as a mix-in class to provide helper
functions for interacting with ptestresults within the extraresults
object generated by the test case.
This class also provides default compression of log text and log files.
Also add support to resulttool for decoding/decompressing log files
embedded in the test results.
Signed-off-by: Nathan Rossi <nathan@nathanrossi.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
gcc-cross.inc: Process binaries in build dir to be relocatable
Process binaries within the build directory before stashing to be
relocatable with ORIGIN relative rpaths. This corrects issues with
rpaths being invalid when trying to use the binaries from an unstashed
build directory (e.g. gcc-runtime).
Signed-off-by: Nathan Rossi <nathan@nathanrossi.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
resulttool: Handle multiple series containing ptestresults
Handle multiple results series having ptestresults content. The contents
are merged on a per-result basis where duplicates are ignored (with a
warning message printed). The 'ptestresults.sections' collection is also
merged on a per-suite basis.
Signed-off-by: Nathan Rossi <nathan@nathanrossi.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
libevent: don't treat test stats line as pass/fail in ptest
Supplemental to commit fb17b46e2. The libevent "regress" test
outputs its own pass/fail results, e.g.
"2/300 TESTS FAILED. (31 skipped)", which will be miscounted
as an extra test fail in the ptest log. Fixed this to ignore
the libevent results line when counting actual pass/fail
results.
Also removed the for loop in run-ptest and targeted only the
libevent "regress" test, as the other tests being run were
related to performance and did not provide a relevant pass/fail
output.
Signed-off-by: Trevor Gamblin <trevor.gamblin@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
chrpath.bbclass: Add break_hardlinks kwarg to allow breaking hardlinks
Add the break_hardlinks kwarg to break hardlinks when modifying files.
This uses the bb.utils.break_hardlinks function to break hardlinks. The
default is to maintain existing behaviour and leave hardlinks in place.
Signed-off-by: Nathan Rossi <nathan@nathanrossi.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
oeqa/selftest/cases/gcc.py: Split into classes for parallelism
Split the gcc selftest cases into multiple classes one for each test.
This is done in order to make it easy to execute multiple gcc tests in
parallel when using oe-selftest with the '-j' arg.
Additionally tag the user tests with "toolchain-user" and the system
emulation (qemu system) tests with "toolchain-system".
Signed-off-by: Nathan Rossi <nathan@nathanrossi.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
oeqa/selftest: Use extraresults on self instead of self.tc
In order to take advantage of multiprocess execution of tests the
extraresults must be passed through the TestResult. With changes to how
oeqa/core handles test cases the extraresults attribute of the testcase
is passed to the TestResult, with passing across process boundaries
handled automatically.
Signed-off-by: Nathan Rossi <nathan@nathanrossi.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
oeqa/selftest: For toolchain tests do not fail if a test failed
Do not fail the running selftest test case if the test suite has a
failed test case. Currently toolchain tests suites (binutils, gcc,
glibc) fail but this does not indicate failure to execute the tests.
Also remove the logging of each test that failed.
Signed-off-by: Nathan Rossi <nathan@nathanrossi.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
oeqa/selftest/cases/glibc.py: Don't install python3 and pexpect
When running the system emulation test case, do not include python3 or
pexpect in the image. The test cases that use these also need gdb (with
python configured).
Signed-off-by: Nathan Rossi <nathan@nathanrossi.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
oeqa/selftest/glibc: Create selftest case for glibc test suite
Create a oeqa selftest test case to execute the glibc test suite and
report the results. The results are populated into the extraresults
variable of the test case which are written to testresults.json for
resulttool to analyse.
An additional subclass is created to separate the execution with qemu
linux-user and qemu system. The GlibcSelfTestSystemEmulated test case
handles setup of the target image, setup of and NFS server as well as
execution with runqemu.
Signed-off-by: Nathan Rossi <nathan@nathanrossi.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
oeqa/selftest/gcc: Create selftest case for gcc test suite
Create a oeqa selftest test case to execute the gcc test suites and
report the results. The results are populated into the extraresults
variable of the test case which are written to testresults.json for
resulttool to analyse.
An additional subclass is created to separate the execution with qemu
linux-user and qemu system. The GccSelfTestSystemEmulated test case
handles setup of the target image as well as execution with runqemu.
Signed-off-by: Nathan Rossi <nathan@nathanrossi.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
oeqa/selftest/binutils: Create selftest case for binutils test suite
Create a oeqa selftest test case to execute the binutils test suites and
report the results. The results are populated into the extraresults
variable of the test case which are written to testresults.json for
resulttool to analyse.
Signed-off-by: Nathan Rossi <nathan@nathanrossi.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Richard Purdie [Tue, 3 Sep 2019 21:25:40 +0000 (22:25 +0100)]
oeqa/selftest: Markup 'machine' specific test cases
These test cases are run by the autobuilder on a machine specific basis.
Add tags to these classes so they can be controlled by the metadata rather
than hardcoded in the autobuilder config.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Use the __oeqa_testtags attribute added by OETestTag and display no, one
or more tags separated by a comma. Also change the formatting of the
output so that the list of tests is formatted as "<test> (<tags>)" and
remove the table header for "id" (it is no longer available).
Signed-off-by: Nathan Rossi <nathan@nathanrossi.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
oeqa/selftest: Add test run filtering based on test tags
Add '--run-only-tags' for running tests which match any of the provided
tags, and '--run-exclude-tags' for running all tests except those that
have any of the provided tags.
Signed-off-by: Nathan Rossi <nathan@nathanrossi.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
oeqa/core: Rework OETestTag and remove unused OETestFilter
Rework OETestTag so that it does not rely on the existing decorator code
base and instead inserts the tags into an attribute on the decorated
target (e.g. class/type or method). This allows the use of OETestTag on
classes and method.
In order to filter tagged tests rework the loaders filtering code,
removing the generic-ness (with validation and attributes/etc.) and
replace it with a "tags_filter" parameter which is a function that
filters a test based on the tags it has. This allows the loader user to
filter on tags in more specific ways (e.g. include all untagged tests
and any tests tagged with foo). Plumb all this through the context code
and testing code.
Update the associated tests to pass correctly with the changes.
Signed-off-by: Nathan Rossi <nathan@nathanrossi.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
oeqa/core: Implement proper extra result collection and serialization
Implement handling of extra result (e.g. ptestresult) collection with
the addition of a "extraresults" extraction function in OETestResult. In
order to be able to serialize and deserialize the extraresults data,
allow OETestResult add* calls to take a details kwarg. The subunit
module can handle cross-process transfer of binary data for the details
kwarg. With a TestResult proxy class to sit inbetween to encode and
decode to and from json.
Signed-off-by: Nathan Rossi <nathan@nathanrossi.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The libevent ptest used to report only a global pass or a fail result.
Count individual PASS, FAIL, SKIP results. The SKIP results now
include tests that are disabled in the libevent code.
libevent's ptest output did not comply with the automake-style output
"result: testname", and reported a FAIL status at the end of the test
run if any of the libevent tests failed. This patch makes the log
consistent with the automake style:
Peiran Hong [Fri, 6 Sep 2019 20:44:19 +0000 (16:44 -0400)]
diffutils: Added perl to support ptest & Skipped one test case
Added perl to the run-time dependency of the recipe for diffutils since
it is required by the test "large-subpot".
The test "strip-trailing-cr" is skipped since it requires valgrind to
work, but valgrind is considered too heavy-weight for diffutils package.
Signed-off-by: Peiran Hong <peiran.hong@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Andre McCurdy [Fri, 23 Aug 2019 20:51:36 +0000 (13:51 -0700)]
feature-arm-thumb.inc: fix ARM_THUMB_SUFFIX
Since TUNE_FEATURES now either contains a CPU or an architecture (but
not both) we can't rely on finding the architecture in TUNE_FEATURES.
Use architecture specific over-rides instead.
Signed-off-by: Andre McCurdy <armccurdy@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Alex Kiernan [Wed, 4 Sep 2019 07:43:44 +0000 (08:43 +0100)]
iputils: Set prefix correctly for usrmerge
Fix build with usrmerge enabled:
WARNING: iputils-s20190709-r0 do_package: iputils: alternative target (/usr/bin/ping or /usr/bin/ping.iputils) does not exist, skipping...
WARNING: iputils-s20190709-r0 do_package: iputils: NOT adding alternative provide /usr/bin/ping: /usr/bin/ping.iputils does not exist
ERROR: iputils-s20190709-r0 do_package: QA Issue: iputils: Files/directories were installed but not shipped in any package:
/bin/tftpd
/bin/tracepath
/bin/arping
/bin/clockdiff
/bin/ping
/bin/traceroute6
/sbin/rarpd
/sbin/ninfod
/sbin/rdisc
Please set FILES such that these items are packaged. Alternatively if they are unneeded, avoid installing them or delete them within do_install.
iputils: 9 installed and not shipped files. [installed-vs-shipped]
WARNING: iputils-s20190709-r0 do_package: iputils: alt_link == alt_target: /usr/bin/ping == /usr/bin/ping
Signed-off-by: Alex Kiernan <alex.kiernan@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
while we control PYTHON pointing to native py3 the libs and include
directories will then point to build host version, which can result in
unexpected combination and if we are lucky we get errors if its quite
different e.g. py2 libs/includes and py3 executable
This variable can be then used to export PYTHON_LIBRARY and
PYTHON_INCLUDE_DIR so that above find_packages can work correctly
This revert restores the original code and adds a comment. The commit
that was reverted broke a number of wic templates and tools which rely
on the initramfs creation dependency and the case where the
INITRAMFS_IMAGE_BUNDLE is not set.
If an end user does not want the INITRAMFS_IMAGE generated, it should
be set to "".
[ Issue: LIN1019-1791 ]
Signed-off-by: Jason Wessel <jason.wessel@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Ross Burton [Thu, 5 Sep 2019 22:49:59 +0000 (23:49 +0100)]
systemd: ensure reproducible builds by clearly exposing the time epoch support
systemd has the ability to check the time on boot and if it's earlier than an
epoch determined at build time, set the time to that epoch. This is useful for
systems where the system time is January 1st 1970 (because the unix timestamp
was 0 at boot) as then at least the time is reset to something approximating the
right year at least.
By default systemd uses the mtime of the NEWS file, which is static for tarballs
and corresponds to the time the release was made, but for git checkouts this is
simply the time do_unpack() was executed. Thus, rebuilding systemd will cause
this embedded timestamp to change.
Remove the PACKAGECONFIG time-epoch which has the logic reversed: enabling
time-epoch will set the epoch to the unix timestamp 0). Replace with
set-time-epoch with the following semantics:
- When disabled, the time epoch is set to 0 (1st January 1970), so there is no
time manipulation on boot.
- When enabled, if reproducible builds are configured by setting
SOURCE_DATE_EPOCH then that timestamp is used for the time epoch. If
reproducible builds are not configured then the timestamp of NEWS (thus the
build time) is used.
The set-time-epoch flag is enabled by default.
[ YOCTO #13473 ]
Signed-off-by: Ross Burton <ross.burton@intel.com>
Dan Callaghan [Mon, 5 Aug 2019 01:13:53 +0000 (11:13 +1000)]
autotools: always include config_site.d files in CONFIG_SITE
Fixes: https://bugzilla.yoctoproject.org/show_bug.cgi?id=13375 Signed-off-by: Dan Callaghan <dan.callaghan@opengear.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
package.bbclass: allow shell-style wildcards in PRIVATE_LIBS
PRIVATE_LIBS is used to exclude 'private' libraries from getting added to
automatic runtime dependency resolution. This variable currently has to list
all libraries by name, which becomes a maintenance issue if the list
of such libraries frequently changes, or is very large.
This change allows using shell-style wildcards in the variable, similar
to how FILES lists what gets packaged.
Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
He Zhe [Thu, 29 Aug 2019 13:15:50 +0000 (21:15 +0800)]
qemu: Fix dependency of PACKAGECONFIG libcurl
When PACKAGECONFIG libcurl is enabled, we would get the following build failure.
ERROR: Nothing PROVIDES 'libcurl' (but perf.bb DEPENDS on or otherwise requires it). Close matches:
The dependency should be curl.
Signed-off-by: He Zhe <zhe.he@windriver.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
He Zhe [Thu, 29 Aug 2019 13:15:49 +0000 (21:15 +0800)]
qemu: Add pkg-config handling for libgcrypt
When PACKAGECONFIG libgcrypt is enabled, we would get the following error.
ERROR: /usr/bin/libgcrypt-config should not be used, use an alternative such as pkg-config
In oe-core, libgcrypt has been turned to be configured with pkg-config instead
of libgcrypt-config, but qemu configure script does not contain pkg-config
related part for libgcrypt to handle it.
Signed-off-by: He Zhe <zhe.he@windriver.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
Robert Yang [Fri, 30 Aug 2019 07:35:30 +0000 (15:35 +0800)]
build-compare: 2015.02.10 -> 2019.08.14
* Removed the following patches which are already merged by upstream:
0001-Add-support-for-deb-and-ipk-packaging.patch
Rename-rpm-check.sh-to-pkg-diff.sh.patch
functions.sh-improve-deb-and-ipk-checking.patch
functions.sh-remove-space-at-head.patch
functions.sh-run-rpm-once-to-make-it-faster.patch
pkg-diff.sh-check-for-fifo-named-pipe.patch
pkg-diff.sh-check_single_file-return-at-once-when-sa.patch
pkg-diff.sh-remove-space-in-the-end-for-ftype.patch
* Rebased Ignore-DWARF-sections.patch
This version is very outstanding when compare binary packages, e.g.:
PRSERV_HOST = "localhost:0"
INHERIT += "packagefeed-stability"
PACKAGE_CLASSES = "package_ipk
$ bitbake opkg
$ find tmp/deploy/ipk >/tmp/ipk_1
Add a "bbnote 'hello'" to autotools.bbclass' autotools_do_configure.
* BEFORE the upgrading, the result is:
$ diff /tmp/ipk_1 /tmp/ipk_2 -Nur | diffstat
ipk_2 | 1570 +++++++++++++++++++++++++++++++++---------------------------------
1 file changed, 785 insertions(+), 785 deletions(-)
* AFTER the upgrading, the result is:
$ bitbake opkg
$ find tmp/deploy/ipk >/tmp/ipk_2
$ diff /tmp/ipk_1 /tmp/ipk_2 -Nur
No output
And if we really modifed a recipe such as opkg, then it would show that it is
changed.
For a full world build AFTER the upgrading:
$ diff /tmp/ipk_6 /tmp/ipk_7 -Nur | diffstat
ipk_7 | 2090 +++++++++++++++++++++++++++++++++---------------------------------
1 file changed, 1045 insertions(+), 1045 deletions(-)
There are 10968 packages in totall, 1045 ones have been changed, so we can still
improve it in the future.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
gcc-testsuite.inc: Fix ssh.exp, ensure multilib_flags are populated
The 'process_multilib_options' function of dejagnu also populates the
'multilib_flags' content from the '--target_board=' arguments. The
'ssh.exp' generated is missing this call ('user.exp' includes it).
Signed-off-by: Nathan Rossi <nathan@nathanrossi.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Richard Purdie [Wed, 4 Sep 2019 10:29:31 +0000 (11:29 +0100)]
binutils/glibc-testsuite: inherit nopackages to fix world builds
These recipes don't need to generate packages but did contain the packaging tasks which
would be triggered by a world build. They showed warnings or errors. Simplest fix
is to remove the unneeded tasks with the nopackages class.
Also don't attempt stash_locales task
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
glibc-testsuite: Create a recipe to implement glibc test suite
A recipe needs to be created for the test suite due to the dependency
chain between libgcc -> glibc -> libgcc-initial, and the requirements of
the test suite to have libgcc for compilation and execution.
The glibc test suite does not use dejagnu like the gcc test suites do.
Instead a test wrapper script is used along with the assumed dependency
of having the same filesystem available on build host and target. For
qemu linux-user the same filesystem is inherently available, for remote
targets NFS is used. Separate test wrapper scripts are created for qemu
linux-user or ssh targets, with the same TOOLCHAIN_TEST_* variables used for
configuration.
Signed-off-by: Nathan Rossi <nathan@nathanrossi.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
gcc-runtime: Add do_check task for executing gcc test suites
Add a do_check task to implement execution of the gcc component test
suites. The component test suites require execution of compiled programs
on the target.
The implementation provided allows for execution testing against a host
via SSH or within the local build environment using qemu linux-user
execution. The selection of execution is done via the
TOOLCHAIN_TEST_TARGET variable, and configuration of the remote host is
done with the TOOLCHAIN_TEST_HOST, TOOLCHAIN_TEST_HOST_USER and
TOOLCHAIN_TEST_HOST_PORT variables.
By default the do_check task will execute all check targets, this can be
changed by setting MAKE_CHECK_TARGETS to the desired test suite target
(e.g. check-gcc or check-target-libatomic).
Signed-off-by: Nathan Rossi <nathan@nathanrossi.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
binutils-cross-testsuite: Create recipe for test suite execution
Create the do_check task in a new recipe 'binutils-cross-testsuite'.
This recipe is built within a target recipe (not -cross) to ensure
correct testing against target specific libraries/etc. The do_check task
is used to execute the binutils test suite for the cross target
binutils. By default this executes tests for binutils, gas and ld. This
can however be changed by setting CHECK_TARGETS to the desired test
suite target (e.g. 'gas').
The binutils test suites do not require any target execution, as such
the check task can be run without QEMU or a target device. However
since the binutils tests do rely on a C compiler there is dependence on
both gcc and libc in order to run the tests.
Signed-off-by: Nathan Rossi <nathan@nathanrossi.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Joshua Watt [Sat, 31 Aug 2019 01:41:24 +0000 (20:41 -0500)]
oeqa: reproducible: Preserve sstate mirror for first build
Preserves the SSTATE_MIRROR variable for the first build in the
reproducible test. This patch is intended to test the theory that using
the Yocto autobuilder sstate mirror will still be reproducible. The
autobuilder always does clean builds and never rebuilds recipes that
were built previously, thus building with the mirror but not
sharing tmpdir and sstate with previous builds should be reproducible.
There is no guarantee that all sstate caches are populated from clean
builds so this patch cannot be submitted to master, and I'm not
bothering sending it to the mailing list.
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Richard Purdie [Tue, 3 Sep 2019 21:39:37 +0000 (22:39 +0100)]
systemtap: Use latest and greatest git version for 5.2 kernel support
Systemtap has issues with the 5.2 kernel which are fixed in master, we helped
debug and submitted some of the patches. Update to a git version which includes
all the fixes.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Ross Burton [Thu, 5 Sep 2019 15:56:20 +0000 (16:56 +0100)]
core-image-sato: don't use 512MB in qemumips
qemumips doesn't appear to work well with 512MB of RAM and randomly hangs. As
the bump to 512MB was primarily motivated by the memory needs of virtualised GL
which doesn't work in qemumips currently, drop the memory back to 256MB for
qemumips.
Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Similar to bitbake, implement a 'keepalive' output to the console to
ensure CI systems do not kill the process. The default timeout for
bitbake is 5000s.
Signed-off-by: Nathan Rossi <nathan@nathanrossi.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
At some point the modules in tests/cases/loader/invalid/ were removed,
this test relied on having tests/cases/loader/invalid/oetag.py to
trigger a ImportError in the loader module. Skip this test.
Signed-off-by: Nathan Rossi <nathan@nathanrossi.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
gcc-common.inc: Process staging fixme with correct target/native sysroot
Correct the 'staging_processfixme' call so that target sysroot and
native sysroot paths are corrected when extracting the stashed build
directory. This is required for 'make check' to work correctly due paths
used in configuration and scripts which point at the native sysroot.
Signed-off-by: Nathan Rossi <nathan@nathanrossi.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Ross Burton [Mon, 2 Sep 2019 18:54:15 +0000 (19:54 +0100)]
systemd: explicitly set the path to nologin
Set the path to nologin as with sulogin etc so we don't try to execute binaries
from the build path on the target (as the build finds a nologin in the sysroot
at build time).
Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
