#579 top-level `make run-tests` target does not indicate pass/fail

[Ed Maste]

For CI use we need a make target that indicates whether tests pass or not. Unfortunatey make run-tests returns success in all cases and produces only a little diagnostic output listing some journal files - for example, this is the entire output from a test with failures:

make run-tests
(cd /tmp/cirrus-ci-build/test; make test)
===> libtest (test)
===> tet (test)
===> libelf (test)
TET_ROOT=../../test/tet/tet3.8 TET_EXECUTE=/tmp/cirrus-ci-build/test/libelf  TET_SUITE_ROOT=/tmp/cirrus-ci-build/test/libelf ../../test/tet/tet3.8/bin/tcc -e .
tcc: journal file is /tmp/cirrus-ci-build/test/libelf/./results/0001e/journal
===> libelftc (test)
TET_ROOT=../../test/tet/tet3.8 TET_EXECUTE=/tmp/cirrus-ci-build/test/libelftc  TET_SUITE_ROOT=/tmp/cirrus-ci-build/test/libelftc ../../test/tet/tet3.8/bin/tcc -e .
tcc: journal file is /tmp/cirrus-ci-build/test/libelftc/./results/0001e/journal
===> libdwarf (test)
TET_ROOT=../../test/tet/tet3.8 TET_EXECUTE=/tmp/cirrus-ci-build/test/libdwarf  TET_SUITE_ROOT=/tmp/cirrus-ci-build/test/libdwarf ../../test/tet/tet3.8/bin/tcc -e .
tcc: journal file is /tmp/cirrus-ci-build/test/libdwarf/./results/0001e/journal
===> ar (test)
===> ar/plugin (test)
/bin/sh run.sh
===> elfcopy (test)
===> elfcopy/plugin (test)
/bin/sh run.sh
===> elfdump (test)
TET_ROOT=../../test/tet/tet3.8 TET_EXECUTE=/tmp/cirrus-ci-build/test/elfdump  TET_SUITE_ROOT=/tmp/cirrus-ci-build/test/elfdump ../../test/tet/tet3.8/bin/tcc -e .
tcc: journal file is /tmp/cirrus-ci-build/test/elfdump/./results/0001e/journal
===> nm (test)
TET_ROOT=../../test/tet/tet3.8 TET_EXECUTE=/tmp/cirrus-ci-build/test/nm  TET_SUITE_ROOT=/tmp/cirrus-ci-build/test/nm ../../test/tet/tet3.8/bin/tcc -e .
tcc: journal file is /tmp/cirrus-ci-build/test/nm/./results/0001e/journal

[Joseph Koshy]

[r3816] should address this for the (shell script based) ar and elfcopy test suites.

For TET-based test suites we would need to look in the test run journal for test purposes that had returned a non-PASS status.

[Ed Maste]

Indeed, CI is now red due to #575
https://cirrus-ci.com/task/4873950589091840

[Joseph Koshy]

[r3817] and [r3818] implement similar checks for TET based test suites.

[Ed Maste]

Anything else still needed here?

[Joseph Koshy]

As it stands, make run-tests will fail on the first test suite failure because test-suite invocation and directory traversal happen at the same time.

It may be more informative to run all the test suites and to then report the overall pass/fail status.

[Joseph Koshy]

I have filed [#580] to track this.