Ross Burton [Fri, 19 Jul 2019 20:33:19 +0000 (21:33 +0100)]
cve-update-db-native: clean up JSON fetching
Currently the code fetches the compressed JSON, writes it to a temporary file,
uncompresses that with gzip and passes the fake file object to update_db().
Instead, uncompress the gzip'd data in memory and pass the JSON directly to
update_db().
Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Ross Burton [Fri, 19 Jul 2019 20:33:18 +0000 (21:33 +0100)]
cve-update-db-native: improve metadata parsing
The metadata parser is fragile: first it coerces a bytes() to a str() (so the
string is b'LastModifiedDate:2019...'), assumes the first line is the date, and
then uses a regex to parse (which then includes the trailing quote as part of
the date).
Clean this up by parsing the bytes as UTF-8 (ASCII is probably fine, but this is
safer), iterate through the lines and split on colons to find the right
key/value pair.
Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Drop the following patches since the issues have been fixed upstream:
0001-file01.sh-Fix-in-was-not-recognized.patch
0001-lapi-Define-TST_ABI-32-64-to-detect-target-type.patch
0001-syscalls-setrlimit03.c-read-proc-sys-fs-nr_open-for-.patch
0007-fix-__WORDSIZE-undeclared-when-building-with-musl.patch
0009-fix-redefinition-of-struct-msgbuf-error-building-wit.patch
0021-Define-_GNU_SOURCE-for-MREMAP_MAYMOVE-definition.patch
0023-ptrace-Use-int-instead-of-enum-__ptrace_request.patch
0024-rt_sigaction-rt_sigprocmark-Define-_GNU_SOURCE.patch
0026-crash01-Define-_GNU_SOURCE.patch
0028-rt_sigaction.h-Use-sighandler_t-instead-of-__sighand.patch
0034-periodic_output.patch
0039-commands-ar01-Fix-for-test-in-deterministic-mode.patch
define-sigrtmin-and-sigrtmax-for-musl.patch
setregid01-security-string-formatting.patch
Refresh the following patches:
0004-build-Add-option-to-select-libc-implementation.patch
0005-kernel-controllers-Link-with-libfts-explicitly-on-mu.patch
0008-Check-if-__GLIBC_PREREQ-is-defined-before-using-it.patch
0018-guard-mallocopt-with-__GLIBC__.patch
0020-getdents-define-getdents-getdents64-only-for-glibc.patch
0035-fix-test_proc_kill-hang.patch
0036-testcases-network-nfsv4-acl-acl1.c-Security-fix-on-s.patch
0001-open_posix_testsuite-mmap24-2-Relax-condition-a-bit.patch
0001-shmctl01-don-t-use-hardcoded-index-0-for-SHM_STAT-te.patch
0001-diotest4-Let-kernel-pick-an-address-when-calling-mma.patch
0001-getrlimit03-adjust-a-bit-of-code-to-compatiable-with.patch
wic: Fix (again) partition files UIDs on multi rootfs images
Commit 450335ba5e73a375eb9932b4c4cf37979640dbfc copies the pseudo
database to the working directory in order to have ownership information
when the filesystem is generated.
Unfortunately this does not work anymore. The filenames on the database
are absolute and there is no information about the new directory.
Instead of fixing the database, we could redo a bit the way we patch the
fstab file. Now I am saving the old contents of fstab, modifying the
file and then reverting the changes on exit.
This is faster than the previous approach, although it can cause
indeterminism if the application is killed before finishing.
Signed-off-by: Ricardo Ribalda Delgado <ricardo@ribalda.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
cve-update-db-native: Remove hash column from database.
djb2 hash algorithm was found to do collisions, so the database was
sometime missing data. Remove this hash mechanism, clear and populate
elements from scratch in PRODUCTS table if the current year needs an
update.
Signed-off-by: Pierre Le Magourou <pierre.lemagourou@softbankrobotics.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The master branch's current tip commit as of this writing is [1], see the
squashfs-tool's repo at [0].
Because of commits [2]-[4] which are included in the master branch three
corresponding patches are dropped as they are not needed anymore. The single
remaining patch was rebased on top of [1] to apply cleanly.
Commits [5] & [6] introduced interesting features, namely zstd support and
reproducibility of created SquashFS images. They are reflected in two new
PACKAGECONFIG options now, but only the latter ("reproducible") is appended to
the default options as OE-core does not contain a recipe to build zstd at the
moment (a working zstd recipe can be found e.g. in meta-rauc, see [7]).
[0] https://github.com/plougher/squashfs-tools.git
[1] f95864afe883 ("unsquashfs-4: Add more sanity checks + fix CVE-2015-4645/6")
[2] 46bdc1726e5a ("mksquashfs: Make a load of functions static")
[3] b0ca8a5c98ff ("pseudo.c: add explicit <sys/stat.h> include")
[4] f95864afe883 ("unsquashfs-4: Add more sanity checks + fix CVE-2015-4645/6")
[5] 6113361316d5 ("squashfs-tools: Add zstd support")
[6] e0d74d07bb35 ("Add configuration and Mksquashfs build options for
reproducible builds")
[7] https://layers.openembedded.org/layerindex/recipe/79049/
Signed-off-by: Ulrich Ölmann <u.oelmann@pengutronix.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Kai Kang [Thu, 4 Jul 2019 13:45:19 +0000 (21:45 +0800)]
defaultsetup.conf: enable select init manager
Introduce a new variable INIT_MANAGER and create 4 init-manager-*.inc
files to configure init manager settings. Available values of
INIT_MANAGER are sysvinit, systemd, mdev-busybox and a default of none.
'none' provides backwards compatibility.
The settings of various VIRTUAL-RUNTIME variables are moved into these
files from the packagegroups.
[YOCTO #13031]
[Modifications by RP for backwards compatibility]
Signed-off-by: Kai Kang <kai.kang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Richard Purdie [Thu, 18 Jul 2019 11:46:12 +0000 (12:46 +0100)]
staging: Drop clean_recipe_sysroot
With recent changes to runqueue, this fuction is unsafe as setscene tasks can run
at the same time as normal ones and doing things before do_fetch no longer
offers any guarantees.
There is other code which cleans out things from the sysroots as tasks rerun so
we should rely upon that instead.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Alistair Francis [Wed, 17 Jul 2019 20:51:49 +0000 (13:51 -0700)]
opensbi: Fix installed-vs-shipped warning
Fix the following warning by just deleting the files:
WARNING: opensbi-0.4-r0 do_package: QA Issue: opensbi: Files/directories were installed but not shipped in any package:
/lib
/lib/libsbiutils.a
/lib/libsbi.a
Signed-off-by: Alistair Francis <alistair.francis@wdc.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Error info:
getrlimit03.c:104: FAIL: __NR_prlimit64(0) had rlim_cur = ffffffffffffffff but __NR_getrlimit(0) had rlim_cur = 7fffffff
According to kernel code: [arch/mips/include/uapi/asm/resource.h]
RLIM_INFINITY is set to 0x7fffffffUL instead of ULONG_MAX on mips32.
/*
* SuS says limits have to be unsigned.
* Which makes a ton more sense anyway,
* but we keep the old value on MIPS32,
* for compatibility:
*/
#ifndef __mips64
# define RLIM_INFINITY 0x7fffffffUL
#endif
Adding conditional statement about mips to fix this.
Signed-off-by: Hongzhi.Song <hongzhi.song@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Ross Burton [Wed, 17 Jul 2019 10:45:37 +0000 (11:45 +0100)]
flex: set CVE_PRODUCT to include vendor
There are many projects called Flex and they have CVEs, so also set the vendor
to remove these false positives.
Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Ross Burton [Wed, 17 Jul 2019 10:45:36 +0000 (11:45 +0100)]
cve-check: allow comparison of Vendor as well as Product
Some product names are too vague to be searched without also matching the
vendor, for example Flex could be the parser compiler we ship, or Adobe Flex, or
Apache Flex, or IBM Flex.
If entries in CVE_PRODUCT contain a colon then split it as vendor:product to improve the search.
Also don't use .format() to construct SQL as that can lead to security
issues. Instead, use ? placeholders and lets sqlite3 handle the escaping.
Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Andrej Valek [Wed, 17 Jul 2019 13:25:44 +0000 (15:25 +0200)]
oe/copy_buildsystem: move layer into layers directory
Layers could be located outside from poky but inside the build directory.
This case should be covered in eSDK.
meta-abc
meta-def/meta-ghi
meta-def/poky
meta-def/meta-oe/meta-oe
...
It should take all enabled layers and put them into 'layers' dir during
build-time with respecting new relative path to poky.
layers/meta-abc
layers/meta-ghi
layers/poky
layers/meta-oe/meta-oe
...
Signed-off-by: Andrej Valek <andrej.valek@siemens.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Mikko Rapeli [Wed, 17 Jul 2019 09:08:36 +0000 (12:08 +0300)]
busybox: enable unicode support
While creating and deleting files with unicode or other
encodings works, it's annoying when ls and other core utils
show questionmarks instead of the unicode characters.
In 2019, it's quite common that users of embedded devices
based on yocto need unicode support. Debugging a box with
unicode encoded file names is a bit annoying when core utils
from busybox don't support them.
The unicode config fragment has the same config as Debian in their
deb and udeb builds of version 1:1.30.1-4.
If developers do not want this or other default yocto features in busybox,
or optimize the configuration for size, then they likely run a completely
custom configuration. Thus I think it's safe to enable unicode support
by default.
Signed-off-by: Mikko Rapeli <mikko.rapeli@bmw.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
u-boot.bin with dtb & signature should be placed in ${B} so that
it can be deployed by u-boot as expected. Otherwise, the version
without signature is installed.
Signed-off-by: Jun Nie <jun.nie@linaro.org> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Chee Yang Lee [Wed, 17 Jul 2019 02:02:42 +0000 (10:02 +0800)]
wic: add support for kernel with initramfs bundled
when INITRAMFS_IMAGE_BUNDLE and INITRAMFS_IMAGE are set, wic should
look for kernel with initramfs image bundled.
Include required variable MACHINE, INITRAMFS_IMAGE_BUNDLE,
INITRAMFS_IMAGE, INITRAMFS_LINK_NAME and KERNEL_IMAGETYPE in WICVARS.
No longer require default value for variable kernel as KERNEL_IMAGETYPE
is not optional variable and included in WICVARS.
image_types_wic to inherit kernel-artifact-names to obtain default
INITRAMFS_LINK_NAME when INITRAMFS_IMAGE_BUNDLE are set.
update wic.Wic2.test_image_env test case to filter optional
variable INITRAMFS_LINK_NAME, INITRAMFS_IMAGE and INITRAMFS_IMAGE_BUNDLE.
Signed-off-by: Chee Yang Lee <chee.yang.lee@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Some distributions for various reasons (like for example mounting a
tmpfs over /var at runtime) can't use /var/lib to store the opkg
metadata, so a different path is required to have a functioning
package manager.
${localstatedir} can't be modified to something other than the
hardcoded value in bitbake.conf because other recipes depending on it
will fail to install.
So the only recourse, which is also the least invasive, is to allow
distros to overwrite the OPKGLIBDIR variable just like they are also
allowed to overwrite OPKGBUILDCMD.
Signed-off-by: Adrian Ratiu <adrian.ratiu@collabora.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Ross Burton [Tue, 16 Jul 2019 12:47:21 +0000 (13:47 +0100)]
glibc: exclude child recipes from CVE scanning
As glibc will be scanned for CVEs, we don't need to scan glibc-locale,
glibc-mtrace, and glibc-scripts which are all separate recipes for technical
reasons.
Exclude the recipes by setting CVE_PRODUCT in the recipe, instead of using the
global whitelist.
Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
CVE-2014-2524 is a readline CVE that was fixed in 6.3patch3 onwards, but the
tooling wasn't able to detect this version. As we now ship readline 8 we don't
need to manually whitelist it, and if we did then the whitelisting should be in
the readline recipe.
Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Kai Kang [Mon, 15 Jul 2019 10:20:16 +0000 (06:20 -0400)]
webkitgtk: set incomptible with tune mips
It fails to compile webkit when default tune is 'mips':
| .../tmp-glibc/work/mips-wrs-linux/webkitgtk/2.24.2-r0/webkitgtk-2.24.2
| /Source/JavaScriptCore/assembler/MacroAssemblerMIPS.h:418:23:
| error: static assertion failed: CLZ opcode is not available for this ISA
So don't build webkit when default tune is mips.
Signed-off-by: Kai Kang <kai.kang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
devtool: provide support for devtool menuconfig command
All packages that support the menuconfig task will be able to run
devtool menuconfig command. This would allow the user to modify the
current configure options and create a config fragment which can be
added to a recipe using devtool finish.
1. The patch checks if devtool menuconfig command is called for a valid
package.
2. It checks for oe-local-files dir within source and creates one if
needed, this directory is needed to store the final generated config
fragment so that devtool finish can update the recipe.
3. Menuconfig command is called for users to make necessary changes.
After saving the changes, diffconfig command is run to generate the
fragment.
Run devtool finish to update the recipe by appending the config fragment
to SRC_URI and place a copy of the fragment within the layer where the
recipe resides.
Ex: devtool finish linux-yocto meta
[YOCTO #10416]
Signed-off-by: Sai Hari Chandana Kalluri <chandana.kalluri@xilinx.com> Signed-off-by: Alejandro Enedino Hernandez Samaniego <alejandr@xilinx.com> Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
devtool/standard.py: Create a copy of kernel source within work-shared if not present
If kernel source is not already downloaded i.e staging kernel dir is
empty, place a copy of the source when the user runs devtool modify
linux-yocto. This way the kernel source is available for other packages
that use it.
[YOCTO #10416]
Signed-off-by: Sai Hari Chandana Kalluri <chandana.kalluri@xilinx.com> Signed-off-by: Alejandro Enedino Hernandez Samaniego <alejandr@xilinx.com> Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
devtool/standard.py: Update devtool modify to copy source from work-shared if its already downloaded
In the regular devtool modify flow, the kernel source is fetched by
running do_fetch task. This is an overhead in time and space.
This patch updates modify command to check if the kernel source is
already downloaded. If so, then instead of calling do_fetch, copy the
source from work-shared to devtool workspace by creating hard links
else run the usual devtool modify flow and call do_fetch task.
[YOCTO #10416]
Signed-off-by: Sai Hari Chandana Kalluri <chandana.kalluri@xilinx.com> Signed-off-by: Alejandro Enedino Hernandez Samaniego <alejandr@xilinx.com> Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Briefly:
Brazil no longer observes DST.
'zic -b slim' outputs smaller TZif files; please try it out.
Palestine's 2019 spring-forward transition was on 03-29, not 03-30.
Changes to future timestamps
Brazil has canceled DST and will stay on standard time indefinitely.
(Thanks to Steffen Thorsen, Marcus Diniz, and Daniel Soares de
Oliveira.)
Predictions for Morocco now go through 2087 instead of 2037, to
work around a problem on newlib when using TZif files output by
zic 2019a or earlier. (Problem reported by David Gauchard.)
Changes to past and future timestamps
Palestine's 2019 spring transition was 03-29 at 00:00, not 03-30
at 01:00. (Thanks to Sharef Mustafa and Even Scharning.) Guess
future transitions to be March's last Friday at 00:00.
Changes to past timestamps
Hong Kong's 1941-06-15 spring-forward transition was at 03:00, not
03:30. Its 1945 transition from JST to HKT was on 11-18 at 02:00,
not 09-15 at 00:00. In 1946 its spring-forward transition was on
04-21 at 00:00, not the previous day at 03:30. From 1946 through
1952 its fall-back transitions occurred at 04:30, not at 03:30.
In 1947 its fall-back transition was on 11-30, not 12-30.
(Thanks to P Chan.)
Changes to past time zone abbreviations
Italy's 1866 transition to Rome Mean Time was on December 12, not
September 22. This affects only the time zone abbreviation for
Europe/Rome between those dates. (Thanks to Stephen Trainor and
Luigi Rosa.)
Changes affecting metadata only
Add info about the Crimea situation in zone1970.tab and zone.tab.
(Problem reported by Serhii Demediuk.)
Changes to code
zic's new -b option supports a way to control data bloat and to
test for year-2038 bugs in software that reads TZif files.
'zic -b fat' and 'zic -b slim' generate larger and smaller output;
for example, changing from fat to slim shrinks the Europe/London
file from 3648 to 1599 bytes, saving about 56%. Fat and slim
files represent the same set of timestamps and use the same TZif
format as documented in tzfile(5) and in Internet RFC 8536.
Fat format attempts to work around bugs or incompatibilities in
older software, notably software that mishandles 64-bit TZif data
or uses obsolete TZ strings like "EET-2EEST" that lack DST rules.
Slim format is more efficient and does not work around 64-bit bugs
or obsolete TZ strings. Currently zic defaults to fat format
unless you compile with -DZIC_BLOAT_DEFAULT=\"slim\"; this
out-of-the-box default is intended to change in future releases
as the buggy software often mishandles timestamps anyway.
zic no longer treats a set of rules ending in 2037 specially.
Previously, zic assumed that such a ruleset meant that future
timestamps could not be predicted, and therefore omitted a
POSIX-like TZ string in the TZif output. The old behavior is no
longer needed for current tzdata, and caused problems with newlib
when used with older tzdata (reported by David Gauchard).
zic no longer generates some artifact transitions. For example,
Europe/London no longer has a no-op transition in January 1996.
Changes to build procedure
tzdata.zi now assumes zic 2017c or later. This shrinks tzdata.zi
by a percent or so.
Changes to documentation and commentary
The Makefile now documents the POSIXRULES macro as being obsolete,
and similarly, zic's -p POSIXRULES option is now documented as
being obsolete. Although the POSIXRULES feature still exists and
works as before, in practice it is rarely used for its intended
purpose, and it does not work either in the default reference
implementation (for timestamps after 2037) or in common
implementations such as GNU/Linux (for contemporary timestamps).
Since POSIXRULES was designed primarily as a temporary transition
facility for System V platforms that died off decades ago, it is
being decommissioned rather than institutionalized.
New info on Bonin Islands and Marcus (thanks to Wakaba and Phake
Nick).
Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The options in ${HOST_CC_ARCH}${TOOLCHAIN_OPTIONS} are already passed
via ${CC}/${CXX} and there is no reason to pass them a second time. Thus
we can remove MESON_TOOLCHAIN_ARGS. And when it is removed, the other
MESON_*_ARGS variables revert to the standard CFLAGS, CXXFLAGS and
LDFLAGS, so just use them directly instead.
Apart from the obvious improvement with not passing a lot of options
twice, this also solves a problem where -pie would be passed on the
command line in a way that it would prevent building any dynamic
libraries using meson if using a toolchain that is not built with
--enable-default-pie and if security_flags.inc is used.
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Before commit[80d17497b7 mdadm: improve the run-ptest],
the mdadm ptest just run some tests without
"--keep-going" option. After the option added in
commit 80d17497b7, all test cases have chance to
be called.
But the logic in mdadm upstream commit
[e2a8e9d tests: wait for complete rebuild in integrity checks]
will make the test enter infinite loop especially in
qemu env as commit e2a8e9d update the logic from
"check wait" to "check state 'U*'" for testcase
tests/01r5integ and tests/01raid6integ. Considering all
other cases still use "check wait" logic, so revert e2a8e9d
to make tests/01r5integ and tests/01raid6integ also use
"check wait" logic to avoid the infinite loop.
[YOCTO #13368]
Signed-off-by: Mingli Yu <Mingli.Yu@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
oeqa/runtime/rpm: ensure no user process running before deleting user
In case of systemd, `su -c 'xxx' test1' via ssh will create
several processes owned by test1, e.g. /lib/system/systemd --user.
These processes are actually managed by user@UID.service
(e.g. user@1000.service). And such service is managed
automatically by systemd. In other words, it will be cleaned
up by systemd automatically.
So we need to wait for systemd to clean it up before trying to
use `userdel' to delete the user.
Signed-off-by: Chen Qi <Qi.Chen@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
glibc-package.inc: Do not use bitbake variable syntax for shell variables
Using bitbake variable syntax (i.e., ${FOO}) for shell variables is
bad practice. First of all it is confusing, but more importantly it
can lead to weird problems if someone actually defines a bitbake
variable with the same name as the shell variable.
Also correct the indentation in stash_locale_cleanup().
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Haiqing Bai [Fri, 12 Jul 2019 12:49:26 +0000 (13:49 +0100)]
sysstat: Use sysstat.service in source for cron with systemd
The sysstat.service script in source launchs sysstat-collect and sysstat-summary
services when cron is installed with systemd. At this time, the upstream
sysstat.service must be installed.
Signed-off-by: Haiqing Bai <Haiqing.Bai@windriver.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Richard Purdie [Fri, 12 Jul 2019 12:54:19 +0000 (13:54 +0100)]
sstate: Add tweak to avoid multiple sstate stats messages
After the recent changes in bitbake to runqueue, we need to recheck sstate validity,
particularly in multiconfig builds where tasks have the same checksum.
Avoid printing summary messages in this case. Also avoid multiple events to toaster
which may not be expecting that at later points in the code.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
