Ed Bartosh [Fri, 16 Jun 2017 13:19:30 +0000 (16:19 +0300)]
wic: fix restoring of original fstab
Wic updates fstab if mount points are specified in .wks.
After partition images are created the original fstab is restored.
However, if exception is raised when partition image being prepared
wic doesn't restore original fstab. This can cause duplication of
added lines in fstab when 'wic create' runs next time.
Wrapping call of 'prepare' method and restoring original fstab
even if exception occurs should fix this.
[YOCTO #11633]
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
scripts/contrib/patchtest: run patchtest on local branch
The script run patchtest on local branch commits, printing results into
stdout. This script is useful to test patches before sending to the
mailing list.
Examples:
$ git checkout master-next-1.9
Branch master-next-1.9 set up to track remote branch master-next-1.9 from origin.
Switched to a new branch 'master-next-1.9'
$ ~/scripts/contrib/patchtest.sh 166e70e: Robert Yang: Thu Apr 2 12:01:37 2015 +0100: patch: fix CVE-2015-1196: FAIL
Issue Missing or incorrectly formatted CVE tag in commit message [test_cve_presence_in_commit_message]
Suggested fix Include a "CVE-xxxx-xxxx" tag in the commit message
Issue Missing or incorrectly formatted CVE tag in included patch file [test_cve_tag_format]
Suggested fix Correct or include the CVE tag on cve patch with format: "CVE: CVE-YYYY-XXXX"
eaa4536: Robert Yang: Thu Apr 2 12:01:37 2015 +0100: wget: 1.16.1 -> 1.16.2: OK
3c29ce3: Robert Yang: Thu Apr 2 12:01:38 2015 +0100: git: 2.3.0 -> 2.3.1: OK
It's possible for tasks to stage symlinks that point to non-existent
files; an example is ncurses-native.do_populate_sysroot. There wasn't
any error checking here so this broke the build when "task" was included
in BUILDHISTORY_FEATURES. In any case we shouldn't be following symlinks
and getting the sha256sum of the link target - we need concern ourselves
only with the target path, so check if the file is a link and sha256 the
target path instead if it is. If it's neither a regular file nor a
symlink (perhaps a pipe or a device), just skip it.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Richard Purdie [Thu, 15 Jun 2017 22:43:22 +0000 (23:43 +0100)]
pciutils: Fold patches into a single patch with a description
Patches patching content from other patches is bad. This folds the three
patches into one, improves some of what its doing to be a little
cleaner and adds a proper description to the patch header.
This also moves the STRIP override to the configure commandline rather
than patching and handles passing libdir the same way.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Richard Purdie [Thu, 15 Jun 2017 17:25:54 +0000 (18:25 +0100)]
meta: Drop further remnants of uclibc support
uclibc support was removed a while ago and musl works much better. Start to
remove the various overrides and patches related to uclibc which are no longer
needed.
uclibc support in a layer would still be possible. I have strong reasons to
believe nobody is still using uclibc since patches are missing and I doubt
the metadata even parses anymore.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Richard Purdie [Thu, 15 Jun 2017 14:21:42 +0000 (15:21 +0100)]
meta: Drop remnants of uclibc support
uclibc support was removed a while ago and musl works much better. Start to
remove the various overrides and patches related to uclibc which are no longer
needed.
uclibc support in a layer would still be possible. I have strong reasons to
believe nobody is still using uclibc since patches are missing and I doubt
the metadata even parses anymore.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Richard Purdie [Tue, 25 Apr 2017 14:25:54 +0000 (15:25 +0100)]
pseudo: Handle too many files deadlock
If we have large amounts of parallelism, pseudo can end up with too
many open connections and will no longer accept further connections,
hanging. This patch works around that by closing some clients, allowing
turnover of connections and unblocking the system. The downside is a small
but theoretical window of data loss. This is likely better than locking
up entirely though. Discussions with Peter are onging about how we could
better fix this.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The recent ovmf update broke secureboot because upstream changed the
way how openssl gets compiled into ovmf. It's now integrated directly
into the ovmf build process, without having to patch it first.
In addition, more recent OpenSSL releases are supported. 1.1.0e was
explicitly mentioned in the ovmf commits and because the current
1.1.0f only has minor build enhancements, 1.1.0e is used here.
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Richard Purdie [Fri, 16 Jun 2017 08:42:30 +0000 (09:42 +0100)]
package_ipk: Clean up Source entry in ipk packages
There is the potential for sensitive information to leak through the urls
there and removing it brings this into the behavior of the other package
backends since filtering it is likely error prone.
Since ipks don't appear to be generated at all if we don't set this, set
the field to the recipe name used (basename only, no paths). This avoids
information leaking. We may want to drop the field if opkg can allow that
at a future point but the recipe name is a suitable identifier for now.
Reported-by: Andrej Valek <andrej.valek@siemens.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
package_manager: don't race on a file when installing complementary
packages
the file isn't closed before oe-pkgdata-util uses it and this
temporary file might look empty to oe-pkgdata-util, because it
wasn't flushed yet. Which resulted in almost empty debugfs tarballs
and no locale packages in regular rootfs.
* without this change:
124K May 30 07:41 core-image-full-cmdline-raspberrypi3-64-20170530054003-dbg.rootfs.tar.gz
* with this change:
173M May 30 07:29 core-image-full-cmdline-raspberrypi3-64-20170530052715-dbg.rootfs.tar.gz
Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Patrick Ohly [Wed, 14 Jun 2017 10:33:50 +0000 (12:33 +0200)]
distrooverrides.bbclass: DISTRO_FEATURES as overrides
This achieves the same goal as the same change to bitbake.conf itself,
but because the class gets added later as part expanding INHERIT, this
new approach is less likely to run into problems when DISTRO_FEATURES
contains complex code.
Another difference is that the class currently does not get inherited
by default and thus is completely absent from a build unless some
layer or include file adds it to INHERIT.
Compared to the earlier code in bitbake.conf and a similar class in
intel-iot-refkit, additional overrides now get sorted. This makes the
final OVERRIDES more deterministic.
The lessons learned about unintentionally depending on OVERRIDES are
documented in the class because such problems are more likely to show
up as unexpected signature differences when using this class.
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It turned out that the code which expands DISTRO_FEATURES early during
base config parsing can fail because some entries in DISTRO_FEATURES
might call Python functions like base_conditional() from base.bbclass
which aren't defined yet.
A different solution will be needed.
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Martin Jansa [Wed, 14 Jun 2017 10:25:38 +0000 (12:25 +0200)]
python-3.5: Move bz2.py, lzma.py and _compression.py from python3-misc to python3-compression
* the /usr/lib/python3.5/_compression.py file is possibly incorrectly included
in python3-misc. This runtime dependency is needed in order to use e.g. gzip.py in runtime:
>>> import tarfile, zlib, gzip
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python3.5/gzip.py", line 12, in <module>
import _compression
ImportError: No module named '_compression'
* at least python3-tests and lzma and bz2 still in python3-misc are using this as well:
$ grep -R import.*_compression tmp-glibc/work/aarch64-oe-linux/python3/3.5.2-r1.0/packages-split/
tmp-glibc/work/aarch64-oe-linux/python3/3.5.2-r1.0/packages-split/python3-compression/usr/lib/python3.5/gzip.py:import _compression
tmp-glibc/work/aarch64-oe-linux/python3/3.5.2-r1.0/packages-split/python3-misc/usr/lib/python3.5/lzma.py:import _compression
tmp-glibc/work/aarch64-oe-linux/python3/3.5.2-r1.0/packages-split/python3-misc/usr/lib/python3.5/bz2.py:import _compression
tmp-glibc/work/aarch64-oe-linux/python3/3.5.2-r1.0/packages-split/python3-tests/usr/lib/python3.5/test/test_bz2.py:import _compression
tmp-glibc/work/aarch64-oe-linux/python3/3.5.2-r1.0/packages-split/python3-tests/usr/lib/python3.5/test/test_lzma.py:import _compression
and python3-tests are using it as well, so add new runtime dependency
on python3-compression
Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Mark Asselstine [Tue, 13 Jun 2017 20:26:45 +0000 (16:26 -0400)]
attr: Fix build failure when image includes man-pages pkg
If you attempt to build an image with both attr(-doc) and man-pages
packages your rootfs might fail to assemble. The error will be
something like:
Error: Transaction check error:
file /usr/share/man/man2/fgetxattr.2 from install of \
attr-doc-2.4.47-r0.core2_64 conflicts with file from \
package man-pages-4.11-r0.core2_64
(the error is usually only seen on builders which don't have manpages
installed, if you have /usr/share/man/man1/man.1.gz your build will
complete but you will have duplicate manpages, just one zipped and one
not)
Backporting changes from upstream attr removes the conflicted files in
favour of those in the man-pages package.
Signed-off-by: Mark Asselstine <mark.asselstine@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
adwaita-icon-theme: add a patch to speed up the do_install() task
Goes down to 40 seconds from over 4 minutes :)
Note that there is no control over the amount of shell jobs; on my machine
this is not a problem, but if it's a problem on less capable hardware,
we can add some kind of limiter.
Signed-off-by: Alexander Kanavin <alexander.kanavin@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Mikko Ylinen [Tue, 13 Jun 2017 15:39:09 +0000 (18:39 +0300)]
runqemu: change terminal settings for valid tty's
runqemu uses stty to change terminal settings to give users
better control to qemu. However, stty does not work when
runqemu is run directly or indirectly via oe-selftest in
a Docker container (presumably some problems with Docker's
pseudo-tty implementation).
The error reported is:
stty: 'standard input': Inappropriate ioctl for device
As runqemu recently moved to subprocess.check_call() for
stty calls we now get thrown an error and all runqemu
runs fail.
sys.stdin.isatty() does proper job in detecting if the stty
calls can work so we use that check before running the stty
subprocess operations.
Signed-off-by: Mikko Ylinen <mikko.ylinen@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
uboot-config: check UBOOT_CONFIG variable, not flags, for error conditions
Sometimes there's a need to change existing UBOOT_CONFIG setting from a recipe,
distro or local config, such as an override or even switch back to UBOOT_MACHINE.
Unfortunately, there's no easy way to override or unset flags, so using them as
an error condition is rather heavy-handed. Change those conditions to check the
UBOOT_CONFIG variable itself, not its flags.
Signed-off-by: Denys Dmytriyenko <denys@ti.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Amarnath Valluri [Tue, 13 Jun 2017 06:52:38 +0000 (09:52 +0300)]
sed: changes to support merged /usr
Few of the perl scripts referring '#!/bin/sed' inside the script. But when
'usrmerge' feature is enabled this path would be /usr/bin/sed. So to satisfy
build dependency add '/bin/sed' to it's providers list.
Signed-off-by: Amarnath Valluri <amarnath.valluri@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Amarnath Valluri [Tue, 13 Jun 2017 06:52:37 +0000 (09:52 +0300)]
busybox: changes to support merged /usr
Most of the shell scripts refer to /bin/sh inside the script. When 'usrmege'
feature is enabled, this path would be /usr/bin/sh. Hence, to satisfy build
dependency add '/bin/sh' to it's providers list.
Signed-off-by: Amarnath Valluri <amarnath.valluri@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Mike Crowe [Tue, 16 May 2017 12:40:00 +0000 (13:40 +0100)]
gcc-cross-canadian: Use ${target_includedir} for compatibility with meta-micro
meta-micro puts headers in /include rather than /usr/include in the
sysroot. ${target_includedir} means that the correct path will be used
automatically.
Signed-off-by: Mike Crowe <mac@mcrowe.com> Acked-by: Phil Blundell <pb@pbcl.net> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Ed Bartosh [Tue, 13 Jun 2017 11:22:10 +0000 (14:22 +0300)]
wic: add help and usage content for 'wic rm'
Added wic_rm_help and wic_rm_usage variables to help.py.
These variables contain help content that will be used in
'wic rm help' and 'wic rm --help' output.
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Ed Bartosh [Tue, 13 Jun 2017 11:22:06 +0000 (14:22 +0300)]
wic: add Disk._put_part_image method
This method copies partition image into the wic image.
It will be used in 'wic cp' and 'wic rm' subcommands
to copy changed partition back into wic image.
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Ed Bartosh [Tue, 13 Jun 2017 11:22:05 +0000 (14:22 +0300)]
filemap: calculate dst size correctly
Fixed calculation of the dst file size using skip, seek and
length parameters. Current code does it incorrectly which
causes sparse_copy API to create unnecessary big output files.
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Ed Bartosh [Tue, 13 Jun 2017 11:22:04 +0000 (14:22 +0300)]
filemap: check if dest is written for every block
If lenght parameter is provided to sparse_copy call
it's mandatory to check if the output file is fully
written after reading unmapped block from input file.
If it's not done then sparse_copy can write more data
than specified length.
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Ed Bartosh [Tue, 13 Jun 2017 11:21:59 +0000 (14:21 +0300)]
wic: add help and usage content for 'wic cp'
Added wic_cp_help and wic_cp_usage variables to
help.py. These variables contain help content that
will be used in 'wic cp help' and 'wic cp --help'
output.
[YOCTO #11283]
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Ed Bartosh [Tue, 13 Jun 2017 11:21:54 +0000 (14:21 +0300)]
wic: add help and usage content for 'wic ls'
Added wic_ls_help and wic_ls_usage variables to
help.py. These variables contain help content that
will be used in 'wic ls help' and 'wic ls --help'
output.
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Ed Bartosh [Tue, 13 Jun 2017 11:21:52 +0000 (14:21 +0300)]
bootimg-pcbios: make boot image file unique
Plugin code uses boot.img file name for an image file. If there are
two partitions that use bootimg-pcbios wic breaks with an error
"file already exists: boot.img"
Made image file name unique by adding wks like number to it to fix
the issue.
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Ed Bartosh [Tue, 13 Jun 2017 11:21:51 +0000 (14:21 +0300)]
filemap: add parameter 'length' to sparse_copy
Added parameter 'length' to specify amount of data
to write into destination file. This is useful when only
part of source file should be written into destination file.
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Ed Bartosh [Tue, 13 Jun 2017 11:55:52 +0000 (14:55 +0300)]
mtools-native: disable reading host configs
Removed code that reads /etc/mtools.conf, /etc/default/mtools.conf,
/etc/mtools and /etc/default/mtools to ensure that mtools output
doesn't depend on the global host configs.
It's still possible to use ~/.mtoolsrc config or point MTOOLSRC
environment variable to any configuration file if user want
to configure mtools.
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Richard Purdie [Tue, 13 Jun 2017 09:22:28 +0000 (10:22 +0100)]
bitbake.conf: Don't exclude MACHINE/MACHINEOVERRIDES from hashes
A long time ago (6 years), this seemed like a good idea. The reality is
that OVERRIDES should not be being added to hashes and if it is, it likely
needs excluding in its own right. This was a nice workaround but we need
to fix the real underlying issues now. In some cases this means excluding
OVERRIDES from the variables dependency using the vardepsexclude flag however
caution is needed to ensure this is safe.
Variable values used to construct hashes are unexpanded but the values used
are computed after the application of OVERRIDES. The important detail is if
the end resulting unexpanded value changes, not the value of the OVERRIDES
used in the construction of that unexpanded value. This is why dependencies
on OVERRIDES itself shouldn't be in the hashes in general.
The recent DISTRO_FEATURES changes adding in override mappings for them
highlighted this issue. We have some good sstate tests which are effective
at highlighting where potential issues arrive with OVERRIDES contamination
(oe-selftest -r sstatetests.SStateTests).
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Richard Purdie [Tue, 13 Jun 2017 09:14:15 +0000 (10:14 +0100)]
insane: Don't depend on OVERRIDES
In common with the other package handling functions, don't depend on the
value of OVERRIDES. This means when we change MACHINE, we don't have to
repackage everything.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Aníbal Limón [Mon, 12 Jun 2017 21:41:18 +0000 (16:41 -0500)]
oeqa/core/loader: Use full and small module name on filtering
The small module name was added to support run a whole suite that
has more that 3 levels in the test case name, but this broke the
behaviour for use a full test case name.
[YOCTO #11632]
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Ming Liu [Mon, 12 Jun 2017 07:25:52 +0000 (09:25 +0200)]
rpm: add explicit dependency on bzip2-replacement-native for native builds
This fixes a following error:
| recipe-sysroot-native/usr/lib/rpm/debugedit: error while loading shared libraries: \
| libbz2.so.1: cannot open shared object file: No such file or directory
Signed-off-by: Ming Liu <peter.x.liu@external.atlascopco.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Khem Raj [Mon, 12 Jun 2017 14:59:05 +0000 (07:59 -0700)]
valgrind: Fix ptest build with musl
musl unearthed a problem when building out of tree, config.h was being
used from $(srcdir) instead of generated config.h in $(builddir)
this assumed functions e.g. mallinfo() and more which are not in musl
as a result tests broke.
Also add fixes to build remaining tests when building on musl
pass -fno-pie as it cant use PIE especially in ptests/x86_64
Signed-off-by: Khem Raj <raj.khem@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
rpm: run binary package generation via thread pools
This greatly reduces build times when there is a large amount of small
rpm packages to produce. The patches are rather invasive,
and so will be submitted upstream.
Signed-off-by: Alexander Kanavin <alexander.kanavin@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
package.bbclass: Restore functionality to detect RPM dependencies
During the transition to dnf and rpm4, the functionality to
automatically make RPM determine dependencies was lost.
Before the transition, an OE specific tool called rpmdeps-oecore had
been added to the rpm suit. It was based on the rpmdeps tool that is
part of rpm. For each file specified on its command line, it would
output the provides and requires that RPM could determine.
During the transition to rpm4, rpmdeps-oecore was replaced with the
standard rpmdeps. However, what no one noticed was that unless rpmdeps
is given options, e.g., -P or -R, to tell it what it should output, it
will not output anything. Thus, it would do all the work to determine
the requirements, but would keep silent about it. And since no output
from rpmdeps is expected unless there are requirements, there were no
warnings indicating that everything was not working as expected.
Porting the old rpmdeps-oecore to work with rpm4 is not really
possible since it relied on being able to access internals of RPM that
are no longer available. However, it turned out that rpmdeps had a
debug option, --rpmfcdebug, that would output exactly the information
that we need, albeit in a different format and to stderr. To make this
usable, rpmdeps has now received a new option, --alldeps, which sends
the information we need to stdout.
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This will send the output from rpmfcPrint() to stdout. This is an
alternative to using the --rpmfcdebug option, which will send the same
output to stderr. The two options have totally different use cases
though. While --alldeps is used when the output from rpmfcPrint() is
what is wanted, --rpmfcdebug can be used together with the other
output options, e.g., --requires, without affecting their output.
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
meta/classes/populate_sdk: Adds support for generating eSDK manifest files
Add get_extra_sdk_info to reuse code in buildhistory
The functionalities to generate SDK and eSDK manifest files are different,
the SDK comes from package information and the eSDK comes from sstate artifacts.
Only execute write_sdk_{host, target}_manifest when is on populate_sdk class.
Adds new functions write_sdk{host, target}_ext_manifest to execute on postprocess
in populate_sdk_ext because at the end we have all the sstate artifacts to
generate the manifest.
[YOCTO #9038]
Signed-off-by: Francisco Pedraza <francisco.j.pedraza.gonzalez@intel.com> Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
lib/oe/sdk: Adds get_extra_sdk_info to reuse code in buildhistory
This function is going to be used for generating the target and host
manifest files packages for eSDK. Added some fixes for buildhistory.bblclass,
and docstring for get_extra_sdkinfo at oe.sdk
[YOCTO #9038]
Signed-off-by: Francisco Pedraza <francisco.j.pedraza.gonzalez@intel.com> Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
