ctest: Failing tests at a higher number of cpu threads

[Wolfgang Stöggl]

Hi,

at a higher number of cpu threads, there are failing tests using ctest.
cppcheck version: 2.18.2

E.g. using ctest -j64, 3 failing tests have been observed:

The following tests FAILED:
     12 - TestCmdlineParser (Failed)
     20 - TestFileLister (Failed)
     64 - TestSuppressions (Failed)

And using ctest -j192 the following 4 tests have been seen to fail:
The following tests FAILED:
     12 - TestCmdlineParser (Subprocess aborted)
     16 - TestCppcheck (Failed)
     20 - TestFileLister (Failed)
     52 - TestSettings (Failed)

An example build log from a current Fedora x86_64 build with -j192 is attached.

As a workaround to avoid failing tests depending on the number of cpu threads of the builder, parallel testing has been disabled so far using ctest --parallel 1 and overriding the number of -jn cpu threads.

There seem to be dependencies in the tests, where e.g. files have to be available and ideally the tests should pass at any high number of cpu threads. 

Best regards

Wolfgang

[Oliver Stöneberg]

TestCmdlineParser::invalidCppcheckCfg
terminate called after throwing an instance of 'std::runtime_error'
  what():  unconsumed stdout: ScopedFile(/builddir/build/BUILD/cppcheck-2.18.2-build/cppcheck-2.18.2/redhat-linux-build/bin/cppcheck.cfg) - could not delete file (-1)

/builddir/build/BUILD/cppcheck-2.18.2-build/cppcheck-2.18.2/test/testsettings.cpp:198(TestSettings::loadCppcheckCfg): Assertion failed. 
Expected: 
Actual: 
not a valid JSON - syntax error at line 2 near: addons\": []}\n

/builddir/build/BUILD/cppcheck-2.18.2-build/cppcheck-2.18.2/test/testfilelister.cpp:139(TestFileLister::excludeDir): Assertion failed. 
Expected: 
Actual: 
could not stat file '../.././redhat-linux-build/bin/process_46.cpp' (errno: 2)

/builddir/build/BUILD/cppcheck-2.18.2-build/cppcheck-2.18.2/test/testfilelister.cpp:64(TestFileLister::recursiveAddFiles): Assertion failed. 
Expected: 
Actual: 
could not stat file '../.././redhat-linux-build/bin/process_7.cpp' (errno: 2)

/builddir/build/BUILD/cppcheck-2.18.2-build/cppcheck-2.18.2/test/testcppcheck.cpp:68(TestCppcheck::checkWithFile): Assertion failed. Unexpected exception was thrown: ScopedFile(test.cpp) - file already exists
_____
/builddir/build/BUILD/cppcheck-2.18.2-build/cppcheck-2.18.2/test/testcppcheck.cpp:69(TestCppcheck::checkWithFS): Assertion failed. Unexpected exception was thrown: ScopedFile(test.cpp) - file already exists
_____
/builddir/build/BUILD/cppcheck-2.18.2-build/cppcheck-2.18.2/test/testcppcheck.cpp:70(TestCppcheck::suppress_error_library): Assertion failed. Unexpected exception was thrown: ScopedFile(test.cpp) - file already exists

[Oliver Stöneberg]

These are all tests which generate files (which is problematic in terms of unit tests).

TestCppcheck can be easily addressed by changing the filename.
TestFileLister needs to use different input (noted in source).
The cppcheck.cfg related tests should/could possibly be moved to the Python tests.

[Oliver Stöneberg]

I filed the follow tickets about this:
https://trac.cppcheck.net/ticket/14108
https://trac.cppcheck.net/ticket/14109
https://trac.cppcheck.net/ticket/14110