|
From: Nicholas N. <n.n...@gm...> - 2023-03-25 00:24:13
|
On Fri, 24 Mar 2023 at 22:52, Mark Wielaard <ma...@kl...> wrote: > > I completely agree with this sentiment. But how do you get there? Ruthless pragmatism and an incremental approach :) > And > how do you cross the psychological barrier. I mean that it feels like > cheating to just disable failing or flaky tests. > They might fail on some, but not all setups. Or they might even be just > flaky depending on CPU model (I think I saw some failures with an AMD > Ryzen processor, which succeeded on an Intel Xeon processor). > > What should our policy be to get to zero fail? > Does that mean a test should always pass on any arch/setup? > Or do we make exceptions for tests that fail on some setups? > Do we keep an "exception list" based on...? > What do we do with the "removed" (or excepted) tests? > Do those turn into high priority bugs instead? > What about new ports, they often start with a bunch of failing tests. > One way to do it is to divide the tests into "must pass on CI" and "the rest". I suspect there are plenty of tests that work on all platforms, which would give a lot of useful coverage from the start. Over time you can hopefully move tests from the first category to the second. The other way to do it is to divide the tests into "run on CI" and "don't run on CI", i.e. exceptions, which does require a mechanism for specifying those exceptions. In practice I think this works out much the same as the first approach, because a test that consistently fails on one platform isn't much use. (In fact, it can have negative value if its presence masks new failures in other tests.) One consequence of all this is that the CI platforms become gospel. E.g. if a test passes on CI but fails locally, that's good enough. This is fine in practice, assuming the CI platforms are reasonable choices. Flaky tests can be a problem. For rare failures you can always just trigger another CI run. For regular failures you should either fix the test or disable it. Nick |