I'm writing to report the current progress of my automated SBCL testing
projects and ask for feedback regarding some open questions.
The first project  consists of automatically (triggered by git
commits) building SBCL with different feature sets on several
architecture/OS combinations, running the test suite and analysing the
results. The feature sets currently are
* without sb-unicode
* without sb-thread
* with sb-safepoint, sb-thruption, sb-wtimer
The architecture/OS combinations are
* x86/Ubuntu Quantal
* x86_64/Ubuntu Quntal
* x86_64/MacOS Lion
Console logs of the build and test process and test results are publicly
Build no. 65 https://ci.cor-lab.org/job/sbcl-master/65/
Test report https://ci.cor-lab.org/job/sbcl-master/65/testReport/
... for one slave https://ci.cor-lab.org/job/sbcl-master/65/featureset=2,label=ubuntu_quantal_32bit/testReport
Console output https://ci.cor-lab.org/job/sbcl-master/65/featureset=2,label=ubuntu_quantal_32bit/consoleFull
Of course, all of these things are directly or indirectly linked from
the summary page.
Many builds complete without errors. The most common occasionally
occurring errors are:
* Violations of expected timing during tests, e.g. test cases take
longer than permitted and thus fail or tests aimed at detecting
performance regressions fail. I suspect these errors to be
caused by our test setup in which the virtual machines running
the jobs can get loaded heavily.
* gc.impure.lisp / BUG-936304 on Ubuntu Quantual 32bit
* Hanging tests (e.g. )
My questions regarding this project are:
* The build server can email "responsible" committers on
build/test failures. However, the current rate of false
positives and already known errors seems to diminish the
usefulness of this feature. What do you think?
* Are instructions for building on Windows available online? If
so, I could try to add Windows slaves.
* Analysis of test results is done by patching SBCL's test harness
to optionally output results using the JUnit XML format. Should
I submit this patch for inclusion in SBCL?
The second project  is automated regression testing based on
cl-test-grid . The basic idea is taking the output of each of the
--fancy builds mentioned above and performing a cl-test-grid run with
that SBCL. The respective outputs of such test runs are then compared
against the outputs of a previous SBCL version to find regressions, e.g.
current master vs. 1.1.7. This is done for the same architecture/OS
combinations as above.
The biggest problem in this project currently are failing uploads of
cl-test-grid results to online storage. The Ubuntu slaves fail to upload
occasionally while the MacOS slave never uploaded successfully. I'm
working with Anton Vodonosov, the author of cl-test-grid, to solve this
Should we be able to fix these problems, results could like this:
My questions for this project are
* The whole process of downloading quicklisp, checking out
cl-test-grid, setting everything up, running cl-test-grid and
generating a regression report is implemented as one shell
script which should only require bash, git and an internet
connection. Should I submit this script for inclusion in SBCL?
* Regression reports are currently built for the respective
working copy version against 1.1.7. Which comparisons would make
sense in general (reports comparing arbitrary versions are cheap
once the corresponding tests have been run and results
Thanks in advance for your feedback and kind regards,