From: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX - 2009-05-26 10:42:00
|
As I indicated in the mail I sent this morning, ABCL has a test suite of its own. This test suite can be found in the source tree under the path: test/lisp/abcl/ In that directory, there are a number of files which contain (in total) (from memory) 224 test cases. Quite a few of them fail at the moment. I'd say that's an interesting opportunity to improve ABCL's quality before the 0.15 release in roughly 2 weeks. In order to further improve (and maintain) ABCL's quality, I'd like to propose that - as much as possible - fixes to our code base (like r11915 from last night - just as an example) will be accompanied by at least 1 test to ensure correctness in the future. Having talked to others, outside of this mailing list, I found that more people believe this will be a good way to work on the trust of our users. BTW: I have planned to extend the ANSI test suite with a number of tests to ensure our closures fixes from the last weeks stay fixed too. Surely I have to put my contributions where my mouth is too. Bye, Erik. |
From: Philip H. <phi...@in...> - 2009-05-26 15:09:06
|
On 26 May, 2009, at 11:41 am, Erik Huelsmann wrote: > In order to further improve (and maintain) ABCL's quality, I'd like to > propose that - as much as possible - fixes to our code base (like > r11915 from last night - just as an example) will be accompanied by at > least 1 test to ensure correctness in the future. > > Having talked to others, outside of this mailing list, I found that > more people believe this will be a good way to work on the trust of > our users. For my part, and accepting that I'm not a major contributor so what I want doesn't necessarily carry that much weight, I only feel comfortable contributing Java code that I know will continuously be tested. That is, I want the JUnit tests I *always* write for my Java code to be part of the same commit as that code. I want those tests integrated into a build which contains a JUnit step. That step runs all JUnit tests available. -- Phil Hudson PGP/GnuPG ID: 0x887DCA63 |
From: Ville V. <vil...@gm...> - 2009-05-26 15:48:43
|
On Tue, May 26, 2009 at 4:45 PM, Philip Hudson <phi...@in...> wrote: > For my part, and accepting that I'm not a major contributor so what I > want doesn't necessarily carry that much weight, I only feel > comfortable contributing Java code that I know will continuously be > tested. That is, I want the JUnit tests I *always* write for my Java > code to be part of the same commit as that code. I want those tests > integrated into a build which contains a JUnit step. That step runs > all JUnit tests available. I like this idea, and it's apparently trivial to integrate JUnit tests into our ANT build. Contributions to the build for this would be highly appreciated. |
From: Mark E. <ev...@pa...> - 2009-05-26 17:04:36
|
Philip Hudson wrote: […] > For my part, and accepting that I'm not a major contributor so what I > want doesn't necessarily carry that much weight, I only feel > comfortable contributing Java code that I know will continuously be > tested. That is, I want the JUnit tests I *always* write for my Java > code to be part of the same commit as that code. I want those tests > integrated into a build which contains a JUnit step. That step runs > all JUnit tests available. I think we're all set up for those who wish to follow this sort of discipline. Running the current test suite as assembled in 'abcl.asd' takes about twenty minutes on a recent Intel Duo Core 2 processor not constrained by RAM, so perhaps this is a reason that some people avoid the suite. I might as well given an informal explanation of the nature of the tests in the Ant-based build system, as perhaps it is not widely understood what my intentions were. As a design principle, I have attempted to implement the invocation of the Lisp based tests via the ASDF definition in 'abcl.asd' so the test suite would be accessible to the Lisp-hosted build as well. Due to either my lack of understanding of ASDF, or the early version of the ASDF facility included with ABCL (ABCL has ASDF 1.3, where SBCL ships with ASDF 1.130. ABCL cannot be upgraded to a later ADSF version without at least implementing the long form of DEFINE-METHOD-COMBINATION), the use of ASDF is currently very hackish, requiring a :FORCE t passed for the invocations, and in reality relying on "stub code" for each test suite under 'test/lisp'. As Erik mentioned earlier, the target 'abcl.test' will do what you ask: build ABCL, and run tests. The tests are subdivided into the Lisp based tests invoked via the 'abcl.test.lisp' target and the JUnit based tests invoked by the 'abcl.test.java' target. 'abcl.test.lisp' will first invoke the compiled version of the GCL ANSI test suite if a sibling directory to the ABCL root named '#p../ansi-tests exists. Then 'abcl.test.lisp' will invoke the internal ABCL tests located in 'test/lisp/abcl'. 'abcl.test.java' will invoke the JUnit tests located under 'test/src'. As each test suite is run, a log-file is created on the filesystem containing the output of the test-suite run. The ANSI tests currently create this file in the top-level ABCL, whereas the ABCL tests create the log file under 'test/lisp/abcl'. This should be rationalized to one or the other locations. The log files are named with the invocation time including HHMM, so as long as you don't invoke the tests in the same minute, you shouldn't clobber previous test results. The removal of the log files should probably be added to the 'abcl.clean' target. What is not working: 'test/lisp/cl-bench' is an incomplete wrapping of the CL-BENCH suite. Unfortunately, I don't remember now what the problem was (something about the invocation depending on a Bourne shell script?) To submit tests with your patches: 1. Lisp based tests ABCL uses its own version of the venerable MIT Regression Test (RT) package which in order to not conflict with the RT tests in the GCL ANSI suite is defined in theABCL-REGRESSION-TEST package. Other than that, I think it is pretty identical to the version shipped with the GCL ANSI test suite. To define a test, one uses the DEFTEST form which has the basic template of: (in-package #:abcl.test.lisp) (deftest foo.1 TEST-FORM RESULT-FORM) When invoked, the FOO.1 test evaluates the TEST-FORM, and compares it to the RESULT-FORM. If [the two are EQL ][1], the test passes. [1]: Not sure about the EQL comparison: might be something slightly different. By convention tests are named 'FOO.N' where N is a positive integer, but this is just a grouping mechanism for human parsing as far as I know. To add a Lisp test, either add it to an existing file in 'test/lisp/abcl', like 'misc-tests.lisp', or create a new file and add it to the RUN method in 'test/lisp/abcl/package.lisp'. One can use ABCL-RT:DO-TEST to invoke individual tests, although sometimes RT gets weird about which package one invokes the tests from. Changing the invoking REPL to ABCL-TEST-LISP package seems to help from what I remember. 2. Java based tests ABCL currently uses JUnit 4.5, for the reason that the sole tests ABCL ever received (from Phillip Hudson) used this package. As far as I can tell, JUnit with the use of Java 5 annotations is about everything you really need in a lightweight Java test suite, but if someone had reasons for using, say, 'Test-NG' we would definitely entertain employing that to get contributed tests. Adding a JUnit test consists of first including the code under 'test/src', probably in the 'test/src/org/armedbear/lisp' directory (and associated package). Since this code doesn't get included in the 'abcl.jar', it is apparently pretty safe to use the same package as the code you are testing. But I don't think this issue has been given much thought, so perhaps tests should go in 'org.armedbear.lisp.test', to avoid static initialization problems. After the code is in place, one has to explicitly mention the source file name in the 'abcl.test.java' target in 'build.xml'. It should be pretty easy to automatically pick up everything under 'test/src', but since we currently have only one test, I didn't bother doing this quite yet. --- Hopefully, that information can help someone out who wishes to contribute tests with ABCL patches. I haven't really had the time to keep up with my planned ABCL work over the past half year, so if someone wanted to come and re-think how we add tests to ABCL, I wouldn't be offended. But at least I have now sort of documented the current state, and some of the background thinking, to give someone a head start. yers in the Bear, Mark <ev...@pa...> -- "A screaming comes across the sky. It has happened before, but there is nothing to compare to it now." |