CLUnit - A Unit Test Tool for Common Lisp

F. A. Adrian
ancar technology

Insuring that code runs and continues to run after changes is extremely important in developing systems in a timely and effective manner. Unit testing is well known way to ensure that code works properly. Unit testing provides functions that are run to verify proper operation of the system under test. The CLUnit package provides a unit test environment for Common Lisp Implementations. It provides facilities for defining, organizing, and running tests.

CLUnit is provided under the terms of the LGPL (see http://www.opensource.org/licenses/lgpl-license.html). As such, it may be loaded and compiled into a product for general release. However, the code is distributed without any warranty (see the text of LGPL for all applicable licensing conditions) and the user assumes all risk of usage.

To start using CLUnit, load the file CLUnit.lisp:

(load "CLUnit.lisp") Dummy error occurred in test "Error test" 1 test run; 0 tests passed; 1 test failed. Dummy error occurred in test "Error test" 1 test run; 0 tests passed; 1 test failed. 2 tests run; 2 tests passed; 0 tests failed. 11 tests run; 11 tests passed; 0 tests failed. CLUnit self-test passed. The first thing that CLUnit does is to run a self-test to ensure that it is running properly.

Basic Testing

Tests are defined using the deftest macro. In its simplest form, the macro takes a test description and a test function that is invoked when the test is run. If the test function returns a non-nil value, the test is assumed to have succeeded. If it throws an error that escapes from the function or returns nil, the test is assumed to have failed:

(deftest "Test car 1" :test-fn #'(lambda () (eq (car '(a b)) 'a))) #<ORG.ANCAR.CLUNIT::TEST Test car 1/*UNCATEGORIZED* #x1011B20> (run-named-test "Test car 1") T (run-all-tests) 1 test run; 1 test passed; 0 tests failed. T 0 1 As we can see, a test can be defined and run by name and all tests thus far defined can be run. The run-all-tests function returns three values – a boolean, telling us if all tests have succeeded or not, the number of failed tests and the number of tests that passed. If we define a test that fails and then run-all-tests again: (deftest "Test car 2" :test-fn #'(lambda () (error "A test error"))) #<ORG.ANCAR.CLUNIT::TEST Test car 2/*UNCATEGORIZED* #x10AD190> (run-all-tests) A test error occurred in test "Test car 2" 2 tests run; 1 test passed; 1 test failed. NIL 1 1 In this case, the error is trapped and the run-all-tests function returns the appropriate values.

Let’s redefine the second test for a more graceful failure. By defining the test with the same name, the original test is replaced:

(deftest "Test car 2" :test-fn #'(lambda () nil)) #<ORG.ANCAR.CLUNIT::TEST Test car 2/*UNCATEGORIZED* #x10C85B0> (run-all-tests) Output did not match expected output in test "Test car 2" 2 tests run; 1 test passed; 1 test failed. NIL 1 1 Again, the error is detected. Note that, when a test fails, a message is printed giving the description of the failed test and the reason for failure. In addition, the failed-tests function will return a list of all tests that failed: (failed-tests) (#<ORG.ANCAR.CLUNIT::TEST Test car 2/*UNCATEGORIZED* # x10C85B0>) Now let’s get rid of the failing test: (remove-test "Test car 2") (#<ORG.ANCAR.CLUNIT::TEST Test car 1/*UNCATEGORIZED* #x1097258>) (run-all-tests) 1 test run; 1 test passed; 0 tests failed. T 0 1 And we see that the removed test is no longer in the system.

Finally, we can get rid of all of the tests:

(clear-tests) NIL (run-all-tests) 0 tests run; 0 tests passed; 0 tests failed. T 0 0

Categorizing Tests

Categories are a useful tool to organize sets of tests. A category for a test is specified by using the :category keyword in the deftest macro and tests from a specific category are run using the run-category function:

(deftest "Test car of nil" :category "Test car" :test-fn #'(lambda () (eq (car nil) nil))) #<ORG.ANCAR.CLUNIT::TEST Test car of nil/Test car #x10F8630> (deftest "Test car of dotted cons" :category "Test car" :test-fn #'(lambda () (eq (car '(a . b)) 'a))) #<ORG.ANCAR.CLUNIT::TEST Test car of dotted cons/Test car #x110F688> (deftest "Another test" :category "General" :test-fn #'(lambda () t)) #<ORG.ANCAR.CLUNIT::TEST Another test/General #x11255C0> (run-category "Test car") 2 tests run; 2 tests passed; 0 tests failed. T 0 2 (run-category "General") 1 test run; 1 test passed; 0 tests failed. T 0 1 (run-all-tests) 3 tests run; 3 tests passed; 0 tests failed. T 0 3 By default, tests are categorized under the category named *UNCATEGORIZED*. Let’s clear the tests again: (clear-tests) NIL

Better Test Specifications

All of the work of the tester can be done using only this functionality, but sometimes the testing becomes a bit clunky. Suppose we had a function returning multiple values: (defun my-func () (values 1 2 3)) MY-FUNC (my-func) 1 2 3 To test this function, a test would have to be written as follows: (deftest "Test my-func" :test-fn #'(lambda () (equal (multiple-value-list (my-func)) '(1 2 3)))) #<ORG.ANCAR.CLUNIT::TEST Test my-func/*UNCATEGORIZED* #x103EA68> (run-all-tests) 1 test run; 1 test passed; 0 tests failed. T 0 1 In order to handle multiple values more gracefully, this test can be written using the :output-fn keyword: (deftest "Test my-func" :test-fn #'my-func :output-fn #'(lambda () (values 1 2 3))) #<ORG.ANCAR.CLUNIT::TEST Test my-func/*UNCATEGORIZED* #x1096640> (run-all-tests) 1 test run; 1 test passed; 0 tests failed. T 0 1 (clear-tests) NIL This feature can also be used for more mundane uses: (deftest "Test car of list" :category "Test car" :test-fn (lambda () (car '(a b))) :output-fn #'(lambda () 'a)) #<ORG.ANCAR.CLUNIT::TEST Test car of list/Test car #x10CA018> (run-all-tests) 1 test run; 1 test passed; 0 tests failed. T 0 1 The expected output function is always a function of zero arguments that returns the values that the output of the test function is compared against. The comparison is done by turning each set of values into a list using multiple-value-list and comparing the list with a comparison function. This comparison function defaults to #’equal, but the user can specify another comparison function using the :compare-fn keyword: (clear-tests) NIL (deftest "Test compare-fn" :test-fn #'(lambda () "abc") :output-form "abc" :compare-fn #'(lambda (rlist1 rlist2) (reduce #'and (mapcar #'string-equal rlist1 rlist2)))) #<ORG.ANCAR.CLUNIT::TEST Test compare-fn/*UNCATEGORIZED* #xFB5500> (run-all-tests) 1 test run; 1 test passed; 0 tests failed. T 0 1 Note that the comparison function is comparing two lists holding the results of the output and test functions. As such, the comparison functions can often be quite hairy.

Rather than specifying the :output-fn keyword, one can use the :output-form keyword. With respect to the test above, the test would change as follows:

(clear-tests) NIL (deftest "Test car of list" :category "Test car" :test-fn (lambda () (car '(a b))) :output-form 'a) #<ORG.ANCAR.CLUNIT::TEST Test car of list/Test car #x10E6370> (run-all-tests) 1 test run; 1 test passed; 0 tests failed. T 0 1 If both :output-fn and :output-form keywords are specified, the :output-fn keyword takes precedence.

Analogously, :input-fn and/or :input-form functions can be specified:

(clear-tests) NIL (deftest "Test +" :input-fn #'(lambda () (values 1 2 3)) :test-fn #'+ :output-form 6) #<ORG.ANCAR.CLUNIT::TEST Test +/*UNCATEGORIZED* #x1105888> (run-all-tests) 1 test run; 1 test passed; 0 tests failed. T 0 1 The test function is applied to the multiple values returned by the input function. The :input-form keyword returns only one value so the test function, in this case, must be unary: (clear-tests) NIL (deftest "Test car" :input-form '(a b) :test-fn #'car :output-form 'a) #<ORG.ANCAR.CLUNIT::TEST Test car/*UNCATEGORIZED* #x1121CD0> (run-all-tests) 1 test run; 1 test passed; 0 tests failed. T 0 1 Specifying an incorrect arity in the test function will result in an error: (clear-tests) NIL

(deftest "Test cons" :input-form '(a b) :test-fn #'cons :output-form '(a . b)) #<ORG.ANCAR.CLUNIT::TEST Test cons/*UNCATEGORIZED* #xFC0EB8> (run-all-tests) Wrong number of arguments occurred in test "Test cons" 1 test run; 0 tests passed; 1 test failed. NIL 1 0 (deftest "Test cons" :input-form (values 'a 'b) :test-fn #'cons :output-form '(a . b)) #<ORG.ANCAR.CLUNIT::TEST Test cons/*UNCATEGORIZED* #xFDA318> (run-all-tests) 1 test run; 1 test passed; 0 tests failed. T 0 1 (clear-tests) NIL

After a test failure, one may want to run a test outside the error trapping code of run-all-tests or run-category to debug a test failure. We use the function run-named-test to do this: (run-named-test "Test cons") ;;; An error occurred in function _WRONG-NUMBER-OF-ARGS-ERROR: ;;; Error: Wrong number of arguments ;;; Entering Corman Lisp debug loop. ;;; Use :C followed by an option to exit. Type :HELP for help. ;;; Restart options: ;;; 1 Abort to top level. ;;; Debugging stuff…

The function run-named-test also takes an optional "protected mode" flag to allow the user to try an individual test: (run-named-test "Test cons" t) Wrong number of arguments occurred in test "Test cons" (NIL #<Simple-error #x101EDB8>) This form of the run-named-test function returns two values, the first telling whether the function succeeded, the second giving the error in case of failure.

Miscellaneous Features

Other miscellaneous functions include list-all-tests and list-categories:

(clear-tests) NIL (deftest "Test 1" :category "Category 1") #<ORG.ANCAR.CLUNIT::TEST Test 1/Category 1 #x1037060> (deftest "Test 1.2" :category "Category 1") #<ORG.ANCAR.CLUNIT::TEST Test 1.2/Category 1 #x1048DF0> (deftest "Test 2" :category "Category 2") #<ORG.ANCAR.CLUNIT::TEST Test 2/Category 2 #x105BB90> (list-tests) ("Test 2/Category 2" "Test 1.2/Category 1" "Test 1/Category 1") (list-tests "Category 1") ("Test 1.2/Category 1" "Test 1/Category 1") (list-categories) ("Category 1" "Category 2") (run-all-tests) 3 tests run; 3 tests passed; 0 tests failed. T 0 3 Note the other (somewhat useless) feature demonstrated above – a test without a test function specified always succeeds.

Tips and Usage Hints

There are a few general usage hints that will help you use CLUnit to its best advantage:

Final Word

I hope you enjoy CLUnit and that it helps you to deliver error-free code in a timely manner. If you have any suggestions for improvements or changes, please contact me at fadrian@qwest.net.