valgrind: Improve non-deterministic ptest reliability

Several of the valgrind tests (particulaly helgrind) are unreliable and
can fail with a different output.

Particularly, there is a higher chance of failure on QEMU instances with
SMP enabled and on systems with more interrupts such as laptops on powersave.

The tests have been reported upstream as being unreliable dating back
over 5 years, due in part to the ordering of threads during
an "unwinding" process in helgrind.
https://bugs.kde.org/show_bug.cgi?id=345121
https://bugs.kde.org/show_bug.cgi?id=430321

A workaround to improve the reliability of such tests is to force them
to run on a single CPU core using taskset. This greatly reduces the
chance of a failure.

>From my testing, I have found it can help reduce the rate of failures
on both a laptop and QEMU by over 5x. Stress-testing in QEMU for several
hours did not result in a failure while running the test normally did.

The flaky or undeterministic thread-based tests are defined in the
taskset_nondeterministic_tests file. These test cases will be run with
taskset 0x00000001 to run on a single CPU core rather then the regular
test.

The edited run-ptest executes the flaky tests first, then ignores them
to not duplicate the results from the main tests. Everything modified is
restored when testing is complete.

The drawback is that this isn't a foolproof solution. It helps the tests
fail much less frequently, and considering how this issue has been documented
for a long time, a workaround such as this is needed.

Signed-off-by: Tony Tascioglu <tony.tascioglu@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

diff --git a/meta/recipes-devtools/valgrind/valgrind/run-ptest b/meta/recipes-devtools/valgrind/valgrind/run-ptest
index 60d243276b1f76349e04a3c829275813a500deb9..11050a831216b773ff669ee26c155eb83efec359 100755 (executable)
--- a/meta/recipes-devtools/valgrind/valgrind/run-ptest
+++ b/meta/recipes-devtools/valgrind/valgrind/run-ptest
@@ -32,6 +32,13 @@ if [ "$arch" = "aarch64" ]; then
    done
 fi
 
+echo "Run flaky tests using taskset to limit them to a single core."
+for i in `cat taskset_nondeterministic_tests`; do
+   taskset 0x00000001 perl tests/vg_regtest --valgrind=${VALGRIND_BIN} --valgrind-lib=${VALGRIND_LIBEXECDIR} --yocto-ptest $i 2>&1|tee ${LOG}
+   mv $i.vgtest $i.IGNORE
+done
+
+
 cd ${VALGRIND_LIB}/ptest && ./tests/vg_regtest \
     --valgrind=${VALGRIND_BIN} \
     --valgrind-lib=${VALGRIND_LIBEXECDIR} \
@@ -56,6 +63,11 @@ for i in `cat remove-for-all`; do
    mv $i.IGNORE $i.vgtest;
 done
 
+echo "Restore flaky and other non-deterministic tests"
+for i in `cat taskset_nondeterministic_tests`; do
+   mv $i.IGNORE $i.vgtest;
+done
+
 echo "Failed test details..."
 failed_tests=`grep FAIL: ${LOG} | awk '{print $2}'`
 for test in $failed_tests; do

diff --git a/meta/recipes-devtools/valgrind/valgrind/taskset_nondeterministic_tests b/meta/recipes-devtools/valgrind/valgrind/taskset_nondeterministic_tests
new file mode 100644 (file)
index 0000000..cf073fa
--- /dev/null
+++ b/meta/recipes-devtools/valgrind/valgrind/taskset_nondeterministic_tests
@@ -0,0 +1,4 @@
+helgrind/tests/hg05_race2
+helgrind/tests/tc09_bad_unlock
+drd/tests/bar_bad
+drd/tests/bar_bad_xml