From: Brian M. <cha...@un...> - 2005-07-11 22:06:25
|
On Mon, 11 Jul 2005, Nikodemus Siivola wrote: >> If tests/ is to be adjusted to make it more fault-tolerant, I would >> suggest identifying some core platform-people combinations such that >> on those platforms the failure would still be a "hard" one: the >> rationale is to prevent the fault-tolerance from encouraging a >> (social, again) tolerance of faults. This doesn't have to be a > > I'd vote for a big "Bad news: TESTS FAILED ON A SUPPOSEDLY GOOD PLATFORM!" > message, possibly accompanied by a "Send report to fai...@sb...? > [y/n]". > > In any case I think the greatest benefit of rt-style tests would be easier > diagnosis: instead of mysterious failure in foo.pure.lisp it would be a > specific test failure(s), wasting less time on part of both the people > reporting the failures and people fixing the problems. As a secondary instead > of the "expected-to-fail-on-arch/os" reader-conditionalization scattered all > over the place we could list expected-to-fail stuff at a single location, > automatically picking up on things that get fixed by accident. For all those who are thinking about rt-style tests: what do you think the effect of ansi-tests would be on the SBCL developers if as of right now the default were to stop on the first failure on which there is no point of standards-interpretation contention? I'm wondering if that behavior would encourage people to fix the failing tests sooner and use it as a regression suite. Regarding figuring out which test failed - it's possible that certain test failures would result in crashing the lisp image. There are two ways around that. We could run each test in its own SBCL instance (which could be quite expensive), or we could keep a log of which tests we're running as we execute them. But we could do this now without rt-style tests, too. I'm not against rt-style tests, but I'm trying to point out some of the disadvantages, I guess :-) I think on the whole I would prefer to see individual groups of tests be rt-style tests where possible (something like the "build sbcl with sbcl" test will never be rt-style), and the entire suite as a whole have a verdict which is based on the failure status of all of the subtest components. > I think there are three distinct tasks here: generating the data > ("build.sh"), collating the data ("tests.sbcl.org"), and starting to move > towards rt-style tests. tests.sbcl.org is a good idea. I'll talk to kmr about getting that pointing in the right place. -- Brian Mastenbrook "God made the natural numbers; http://www.iscblog.info/ all else is the work of man." cha...@un... -- Leopold Kroneker |