[Sbcl-devel] Re: Meaning of self test failures

[Nikodemus S. <tsi...@cc...>]

[ Redirected the CC to -devel instead of -help. ]

On Tue, 7 Dec 2004, Devon Sean McCullough wrote:

> The out of the box FreeBSD SBCL scares off new users, e.g., me
> when I ran the self tests.  Here follow patches to ...
>
> * Reassure users
> * Improve tests

Thanks for the effort, but I think this is not such a good idea:

> +echo <<EOF
> +Do not be alarmed if a test fails, remove that file and re-test.
> +The regression suite is constantly growing, so a "new" failure
> +most likely is just something the older suite did not test for.
> +EOF

This makes it less likely that users will report the failures, which
is all important. Also, while there may be known problems, we IMO still
prefer to consider them problems, not just something to ignore...

> --- tests.orig/float-modes.impure.lisp	Tue Dec  7 14:24:03 2004
> +++ tests/float-modes.impure.lisp	Tue Dec  7 14:23:53 2004
> @@ -0,0 +1,22 @@
> +;;;; These tests check for known bugs which wrongly change
> +;;;; floating point arithmetic modes such as whether to
> +;;;; signal an overflow error or return an infinite value.
> +
> +;;;; Known broken in SBCL 0.8.16 under FreeBSD 5.2-RELEASE
> +;;;; so uncomment the next line to disable this test file.
> +; #+SBCL (quit :unix-status 104)	; fake success

Better display the breakage: the test suite is there to show the
problems, not hide them. The pertinent question is: does the current
suite catch the FreeBSD 5 bugs? If not, what tests do we need to add to 
catch them. (From the previous messages I was under the impression that
the bugs were caught.)

Now, I should probably elaborate what I ment with a more sophisticated 
harness: eg. something that runs all the test and afterwards just reports
wich ones failed.

> -  for (j = 0; j &lt; 10; j++) printf("a[%d] = %d.\n", j, a[j]);
> +  for (j = 0; j < 10; j++) printf("a[%d] = %d.\n", j, a[j]);

Good catch, thanks!

Cheers,

  -- Nikodemus