[Pyunit-interest] Data-driven tests

[Gareth R. <gd...@ra...>]

Here's some discussion of PyUnit's handling of data-driven tests.  I 
hope to persuade you that improvement is needed.

By "data-driven tests" I mean groups of tests handled by the same 
code by driven by tables or files of data.  For example, a test case 
that reads records from two databases and checks that they are 
consistent.  Or a test case that repeatedly calls a function under 
test with arguments taken from a table.


Requirements for data-driven tests:

   1. You mustn't have to know in advance how many tests will be made 
(you can't when checking files or databases).

   2. You must be able to find as many errors as possible each time 
the test case is run: this is particularly important when the test 
case is expensive to set up.


Here are some suggestions for implementing data-driven tests, with analysis:

   1. Write an ordinary unittest test case.  This will stop when it 
finds the first error, thus failing requirement 2.

   2. Declare a method for each test case, perhaps using tricks with 
'eval', as in the manytests.py example distributed with PyUnit 1.3.1. 
This is only possible if you know in advance how many test cases 
there will be, failing requirement 1.  Also the volume of output from 
the test runner is likely to be unhelpful and annoying (when you have 
1000 test cases you don't want to see a report for each case, merely 
a report for each failure and a summary of the cases executed).

   3. Collect errors as they are found yourself and then report them 
all at once at the end of the test.  This loses much of the 
convenience of PyUnit.

   4. Extend PyUnit by providing a mechanism for specifying that a 
test case be called multiple times.  This might be done by having a 
subclass of unittest.TestCase which repeatedly calls its test method, 
with a numeric argument giving the number of times it's been called 
so far, and stops when the method returns false.

For table-driven tests this might work, but for tests driven by files 
or databases I think it will make the tests very hard to write, as a 
test that is naturally written as a loop will have to be broken up 
into separate calls to the test method in order to fit this approach. 
Also this design means that it will be very easy to make mistakes 
that result in infinite loops.

   5. Extend PyUnit by providing methods for reporting errors without 
aborting the test case.  This could be done quite simply by providing 
a way for a test case to call the addFailure() method of the 
TestResult object.  For example, you could add the line

         self.__result = result

near the start of the TestCase.__call__ method, and add a method like 
this to the TestCase class:

     def addFailure(self, msg):
         # Raise an exception and catch it immediately so that we can get a
         # traceback.
         try:
             raise self.failureException, msg
         except self.failureException:
             self.__result.addFailure(self,self.__exc_info())

A disadvantage of this approach is that it breaks the one-to-one 
correspondence between tests and failures.  If you have 10 test cases 
and 5 failures, it doesn't mean that you've had 5 successes: the 5 
failures may have come from a single test case.  I don't think that 
this correspondence is particularly important, but if you think there 
are users who will want to write data-driven tests that preserve the 
correspondence you could have methods like addFailureIf(expr, msg) 
and so on that increment the number of test cases in the TestResult 
object when they are called, regardless of whether they succeed or 
fail.


I think that solution 5 has the most to recommend it.  I've 
successfully used this approach in my own unit tests.  I would like 
to see it or something like it included in future releases of PyUnit.

Solution 4 could be offered as an another approach.  However, I would 
need to develop the solution and gain some experience using it before 
I was able to recommend it.