test suite driver enhancements

[Bruno H. <br...@cl...>]

Hi,

When looking at the "make check" results on the various platforms, I'm seeing
room for improvement:

1) A couple of tests should be skipped (or with an acceptable failure) on
specific platforms. Example:

------------------------------------------------------------------------------------------------------
Form: (LET ((*REOPEN-OPEN-FILE* NIL)) (WITH-OPEN-FILE (COPY S :DIRECTION :OUTPUT) (STREAMP COPY)))
CORRECT: T
CLISP  : ERROR
OPEN: Filename for #1=#<OUTPUT UNBUFFERED FILE-STREAM CHARACTER> is unknown

OUT:
"[SIMPLE-FILE-ERROR]: OPEN: Filename for #1=#<OUTPUT UNBUFFERED FILE-STREAM CHARACTER> is unknown

"
------------------------------------------------------------------------------------------------------
On AIX and HP-UX. Caused by the lack of support for /dev/fd/1.

2) When a group of tests belongs together, and a test fails, skip the remaining tests
   of the same group.

3) When a test is skipped, we need to write the condition twice:
   #+condition form #+condition result

However, keep in mind the important design goals of the test suite driver:

  ============================================================================
  * It must be easy to add a new test.
    This means, minimal clutter. Ideally, no need to write 'deftest' forms,
    'assert' statements etc.

  * When a test fails, it must be easy to execute it in single-step mode.
    Where "single-step mode" means to copy entire lines of test into the Lisp
    REPL.
    This means, use SETQ/DEFVAR, not LET/LET*, to create temporary objects
    that the rest of the test references.
  ============================================================================

What I do not want is a test suite style like these:

[From ansi-tests:]
(deftest rename-file.1
  (let ((pn1 #p"file-to-be-renamed.txt")
        (pn2 #p"file-that-was-renamed.txt"))
    (delete-all-versions pn1)
    (delete-all-versions pn2)
    (with-open-file (s pn1 :direction :output) (format s "Whatever~%"))
    (let ((results (multiple-value-list (rename-file pn1 pn2))))
      (destructuring-bind (defaulted-new-name old-truename new-truename)
          results
          (values
           (=t (length results) 3)
           (probe-file pn1)
           (notnot (probe-file pn2))
           (list (notnot (pathnamep defaulted-new-name))
                 (notnot (pathnamep old-truename))
                 (notnot (pathnamep new-truename))
                 (typep old-truename 'logical-pathname)
                 (typep new-truename 'logical-pathname))
           (notnot (probe-file defaulted-new-name))
           (probe-file old-truename)
           (notnot (probe-file new-truename))))))
  t nil t (t t t nil nil) t nil t)

[From sacla-tests]
(let ((ba (make-array '(1 2 3 4 5) :element-type 'bit)))
  (dotimes (i (* 1 2 3 4 5))
    (setf (sbit ba
                (floor i (* 1 2 3 4 5))
                (floor (mod i (* 2 3 4 5)) (* 3 4 5))
                (floor (mod i (* 3 4 5)) (* 4 5))
                (floor (mod i (* 4 5)) 5)
                (mod i 5))
          (if (evenp i) 0 1)))
  (dotimes (i (* 1 2 3 4 5) t)
    (unless (eql (row-major-aref ba i) (if (evenp i) 0 1))
      (return nil))))

[From sbcl tests:]
(let ((fun (compile nil '(lambda () (make-instance 'class-with-special-ssvuc-2
                                     :some-slot 1)))))
  (assert (= *special-ssvuc-counter-2* 0))
  (funcall fun)
  (assert (= *special-ssvuc-counter-2* 0))
  (defmethod (setf slot-value-using-class) :before
      (new-value class (instance class-with-special-ssvuc-2) slotd)
    (incf *special-ssvuc-counter-2*))
  (funcall fun)
  (assert (= *special-ssvuc-counter-2* 1)))

In other words, I see the test suite files as *data* (to be interpreted by the
test suite driver), not as a Lisp program.


To solve problem 1), I propose to add - in tests.lisp - OS names to the
*features* list. So that it becomes possible to write #+(or AIX HP-UX).
Outside of the test suite, i.e. for user programs, there should be no
distinction between Linux, AIX, HP-UX, etc.

To solve problem 2), I propose a syntax with braces:
{
FORM1
EXPECTED-RESULT1
FORM2
EXPECTED-RESULT2
...
}

To solve problem 3), I propose a syntax like this:
$+CONDITION
FORM
EXPECTED-RESULT

Comments?

Bruno