testsdk: fix skipped testcase output "UNKNOWN" status while multiprocess execution

Usually skipped testcase output "SKIPPED"
[snip serial execution]
|RESULTS - buildgalculator.GalculatorTest.test_galculator - Testcase -1: SKIPPED (0.01s)
|RESULTS - python.PythonTest.test_python3 - Testcase -1: SKIPPED (0.01s)
[snip serial execution]

But if enable multiprocess execution, skipped testcase output "UNKNOWN" status
[snip enable multiprocess execution]
|RESULTS - buildgalculator.GalculatorTest.test_galculator - Testcase -1: UNKNOWN
|RESULTS - python.PythonTest.test_python3 - Testcase -1: UNKNOWN
[snip enable multiprocess execution]

Here is my investigation:

There is a class pairs TestProtocolClient and TestProtocolServer
provided by python3-subunit. The TestProtocolClient generates a
subunit stream of TestResult from a test run, and TestProtocolServer
parses the stream of subunit TestResult.

The class ProtocolTestCase is a unittest.TestCase adapter and it
uses TestProtocolServer to parse the stream of subunit TestResult.

In Yocto testsdk, it forks multiple processes to execute testcases
and use TestProtocolClient to generate TestResult stream; and then
it creates multiple threads to use ProtocolTestCase to parse stream
of subunit TestResult through pipe; finally it passes multiple
ProtocolTestCase as TestCase instance to main process and output
status result.

The problem point is TestProtocolServer parses `skip:' directive
after reading a `test:' directive. Without `test:' directive,
`skip:' directive will be ignored. All above requires SkipTest should
be raised inside a test method rather than setUpClass method.

Throwing SkipTest inside setUp works correctly

Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

diff --git a/meta/lib/oeqa/sdk/cases/assimp.py b/meta/lib/oeqa/sdk/cases/assimp.py
index 26c1df089cec98095b8e6520878e9ceede639020..f5c7547b8821640db5b606e77723fc98b98f9912 100644 (file)
--- a/meta/lib/oeqa/sdk/cases/assimp.py
+++ b/meta/lib/oeqa/sdk/cases/assimp.py
@@ -12,8 +12,7 @@ class BuildAssimp(OESDKTestCase):
 
     td_vars = ['DATETIME', 'TARGET_OS', 'TARGET_ARCH']
 
-    @classmethod
-    def setUpClass(self):
+    def setUp(self):
         if not (self.tc.hasHostPackage("nativesdk-cmake") or
                 self.tc.hasHostPackage("cmake-native")):
             raise unittest.SkipTest("Needs cmake")

diff --git a/meta/lib/oeqa/sdk/cases/buildcpio.py b/meta/lib/oeqa/sdk/cases/buildcpio.py
index 333dc7c226d7d4bb54a29ab9365acdc3db0dbb83..f348ac5d909f828755ad5105aa83a6f5f3b8cb64 100644 (file)
--- a/meta/lib/oeqa/sdk/cases/buildcpio.py
+++ b/meta/lib/oeqa/sdk/cases/buildcpio.py
@@ -14,6 +14,7 @@ class BuildCpioTest(OESDKTestCase):
                         self.tc.sdk_dir, self.td['DATETIME'], dl_dir=dl_dir)
         self.project.download_archive()
 
+    def setUp(self):
         machine = self.td.get("MACHINE")
         if not self.tc.hasHostPackage("packagegroup-cross-canadian-%s" % machine):
             raise unittest.SkipTest("SDK doesn't contain a cross-canadian toolchain")

diff --git a/meta/lib/oeqa/sdk/cases/buildgalculator.py b/meta/lib/oeqa/sdk/cases/buildgalculator.py
index 050d1b3b520ca680cf5f80cffa8e24adecfd90ca..9e12b3ac10deb52f120bdd4380c29e32018ccba3 100644 (file)
--- a/meta/lib/oeqa/sdk/cases/buildgalculator.py
+++ b/meta/lib/oeqa/sdk/cases/buildgalculator.py
@@ -6,8 +6,7 @@ from oeqa.sdk.utils.sdkbuildproject import SDKBuildProject
 class GalculatorTest(OESDKTestCase):
     td_vars = ['DATETIME']
 
-    @classmethod
-    def setUpClass(self):
+    def setUp(self):
         if not (self.tc.hasTargetPackage("gtk+3", multilib=True) or \
                 self.tc.hasTargetPackage("libgtk-3.0", multilib=True)):
             raise unittest.SkipTest("GalculatorTest class: SDK don't support gtk+3")

diff --git a/meta/lib/oeqa/sdk/cases/buildlzip.py b/meta/lib/oeqa/sdk/cases/buildlzip.py
index b28cc3a595ed6b978aeedbe74e7d8ae283d08daa..9d137f30ebf42281bd593631b4b3b07bfdc3a497 100644 (file)
--- a/meta/lib/oeqa/sdk/cases/buildlzip.py
+++ b/meta/lib/oeqa/sdk/cases/buildlzip.py
@@ -15,6 +15,7 @@ class BuildLzipTest(OESDKTestCase):
                         self.tc.sdk_dir, self.td['DATETIME'], dl_dir=dl_dir)
         self.project.download_archive()
 
+    def setUp(self):
         machine = self.td.get("MACHINE")
 
         if not (self.tc.hasHostPackage("packagegroup-cross-canadian-%s" % machine) or

diff --git a/meta/lib/oeqa/sdk/cases/perl.py b/meta/lib/oeqa/sdk/cases/perl.py
index ff50b468006ef1acf085a7e58e05f893b9ed02fd..e1d2bc159a07361ec9d4612ce6de4e4b169244aa 100644 (file)
--- a/meta/lib/oeqa/sdk/cases/perl.py
+++ b/meta/lib/oeqa/sdk/cases/perl.py
@@ -2,8 +2,7 @@ import unittest
 from oeqa.sdk.case import OESDKTestCase
 
 class PerlTest(OESDKTestCase):
-    @classmethod
-    def setUpClass(self):
+    def setUp(self):
         if not (self.tc.hasHostPackage("nativesdk-perl") or
                 self.tc.hasHostPackage("perl-native")):
             raise unittest.SkipTest("No perl package in the SDK")

diff --git a/meta/lib/oeqa/sdk/cases/python.py b/meta/lib/oeqa/sdk/cases/python.py
index bd5f1f67be265cfdd81411cc693b68a137197582..7c1d183e5291a60757dca5b3678898883be053c7 100644 (file)
--- a/meta/lib/oeqa/sdk/cases/python.py
+++ b/meta/lib/oeqa/sdk/cases/python.py
@@ -2,8 +2,7 @@ import subprocess, unittest
 from oeqa.sdk.case import OESDKTestCase
 
 class PythonTest(OESDKTestCase):
-    @classmethod
-    def setUpClass(self):
+    def setUp(self):
         if not (self.tc.hasHostPackage("nativesdk-python3") or
                 self.tc.hasHostPackage("python3-native")):
             raise unittest.SkipTest("No python package in the SDK")