|
From: Mark W. <ma...@kl...> - 2023-04-01 22:56:07
|
Hi Nick, On Sat, Mar 25, 2023 at 11:23:54AM +1100, Nicholas Nethercote wrote: > One way to do it is to divide the tests into "must pass on CI" and "the > rest". I suspect there are plenty of tests that work on all platforms, > which would give a lot of useful coverage from the start. Over time you can > hopefully move tests from the first category to the second. So for the original buildbot CI I set it up to only run the auxtests (that is build gsl and run the testsuite under valgrind). This used to work on all setups, but after a gcc (or glibc?) update some setups started failing as you can see at: https://builder.sourceware.org/buildbot/#/builders?tags=valgrind For the try builders I took out the now failing builders (debian-testing-x86_64, fedora-arm64, fedora-x86_64, ibm-power10, opensusetw-x86_64 and rawhide-x86_64). https://builder.sourceware.org/buildbot/#/builders?tags=valgrind-try Hopefully we can figure out why the auxtests fail now on some of these setups. > The other way to do it is to divide the tests into "run on CI" and "don't > run on CI", i.e. exceptions, which does require a mechanism for specifying > those exceptions. In practice I think this works out much the same as the > first approach, because a test that consistently fails on one platform > isn't much use. (In fact, it can have negative value if its presence masks > new failures in other tests.) So should we make a new check target that runs a subset of make regtests? Or maybe have a "ci mode" for regtests? So you would run CI_MODE=true make regtests and it would skip any vgtest that has prereq: test -z "$CI_MODE" And then add make regtests to the ci and try buildbots? Cheers, Mark |