On Sat, Jun 17, 2006 at 05:20:51PM +0200, Lutz Euler wrote:
Content-Description: message body text
> I think the tests using RANDOM would be much more useful if they were
> using different numbers on each test run -- and there is evidence in
> some tests that their author was believing that this is the case; see
> for example this comment in arith.pure.lisp:
> ;;; give any optimizers of constant multiplication a light testing.
> ;;; 100 may seem low, but (a) it caught CSR's initial errors, and (b)
> ;;; before checking in, CSR tested with 10000. So one hundred
> ;;; checkins later, we'll have doubled the coverage.
> (dotimes (i 100)
> (let* ((x (random most-positive-fixnum)) ...)))
> So here is a patch that makes the test framework execute
> (setf *random-state* (make-random-state t))
> once in the main SBCL before the pure tests are run and once in each
> child SBCL at the beginning of an impure test run.
I am of two minds about doing this, and in fact my own personal
software does it in some cases and not in others.
To a first approximation, my preferred boundary between the two
types of tests is that
(setf *random-state* (make-random-state t))
is appropriate when it is easy to turn the failed test into a
repeatable description of the failed test case, so that it can be
installed in another test file where it runs deterministically after
In practice, when SBCL tests are running on an SBCL developer's
machine, very often he will be able to create a repeatable test case
from the backtrace even there's no care taken (when writing the test)
to support this.
However, I think that for tests that we recommend for first-time users
everywhere, the deterministic tests might be preferred. I worry that
without determinism, some noticeable fraction of error reports could
become aggravating "how do we repeat this? is it even repeatable?"
An alternative would be to leave the default deterministic, but make
an optional way of running the tests with a fresh random number seed,
perhaps combined with an optional way to execute >1 passes through the
tests, so that one's computer can be a better space heater on frosty
winter nights, and encourage developers and other masochists to do the
random tests as well as the deterministic tests.
William Harold Newman <william.newman@...>
PGP key fingerprint 85 CE 1C BA 79 8D 51 8C B9 25 FB EE E0 C3 E5 7C
Ubi saeva indignatio ulterius cor lacerare nequit. -- Jonathan Swift's epitaph