[Sbcl-devel] Tests using RANDOM (with patch)

[<lut...@fr...>]

Hi,

I am currently working on improving the LOGCOUNT VOP for x86-64 and in
the course of this wrote a more thorough test for LOGCOUNT that besides
with some regular patterns also tests with random patterns. When testing
this test by building an SBCL with an intentionally slightly broken
LOGCOUNT and running "sh run-tests.sh --break-on-failure" I noticed that
the test always failed for the same random pattern while I expected it
to use a different sequence of random numbers on each run.

It turns out that SBCL always initialises *RANDOM-STATE* from the same
seed and that the test framework does nothing to change that. The
reasons for the former (with which I fully agree, btw.) are given in
NEWS, under "changes in sbcl-0.6.3":

  *RANDOM-STATE* is no longer automatically initialized from 
  (GET-UNIVERSAL-TIME), but instead from a constant seed. Thus, the
  default behavior of the system is to repeat its behavior every time
  it's run. If you'd like to change this behavior, you can always
  explicitly set the seed from (GET-UNIVERSAL-TIME); whereas under the 
  old convention there was no comparably easy way to get the system to 
  repeat its behavior every time it was run.

I think the tests using RANDOM would be much more useful if they were
using different numbers on each test run -- and there is evidence in
some tests that their author was believing that this is the case; see
for example this comment in arith.pure.lisp:

  ;;; give any optimizers of constant multiplication a light testing.
  ;;; 100 may seem low, but (a) it caught CSR's initial errors, and (b)
  ;;; before checking in, CSR tested with 10000.  So one hundred
  ;;; checkins later, we'll have doubled the coverage.
  (dotimes (i 100)
    (let* ((x (random most-positive-fixnum)) ...)))

So here is a patch that makes the test framework execute

  (setf *random-state* (make-random-state t))

once in the main SBCL before the pure tests are run and once in each
child SBCL at the beginning of an impure test run.

With kind regards

Lutz Euler