Re: [Ocaml-lib-devel] Future of ExtLib

[Janne H. <ja...@hy...>]

> How do you (a) generate the input and (b) predict the
> expected result?

It is wholly dependent on the test, of course.  For example, random 
strings with random length and contents, possibly having zero elements 
in it.  Then feeding it into a function and verifying with asserts that 
the output is what it's supposed to be.

The randomness depends heavily on the problem domain.  With programs 
such as compilers, you want a lot of randomness, but for simple things 
like string functions you're fine with just a few carefully selected inputs.

In one compiler project I used an AST and a code generator that 
generated code based on that AST.  When generating code, you have a lot 
of opportunities for randomly creating different variations so that the 
generated output still does the same thing.

I fed the randomly generated programs through the compiler, ran them and 
compared the results against a direct interpretation of the AST (the AST 
was conveniently interpretable, as it was created for this purpose only).

To predict the output, you of course need to create cases where you in 
some way (programmatically) know the result.  For example, if you know 
that a function should raise an exception with some input, you feed it 
with that input and then assert that it actually raises it.  I'm sure 
everyone is familiar with this sort of testing.

I don't have my tester for one of my libraries at hand, but perhaps 
tomorrow I can post some examples if you're interested.

> I *rely* heavily on my tests in Felix.
> In fact they're part of the distro and the build always runs the
> basic ones.
> 
> I do lots of 'refactoring' and enhancements and I'd be screwed
> without the testing harness.

Yep, same here.

>>I guess for Extlib, you're always assumed to pass *all* tests before 
>>making a release or committing changes.  
> 
> 
> Yup, but when a test fails you need to know which one :)

Because I use assertions.  I can do just:

export OCAMLRUNPARAM=b=1
make all
make test

and then examine the callstack to see where something went wrong.

Or use a debugger.  I don't see any reason why it should be any more 
complicated than that.  As Extlib, at least in its current, form is 
relatively small, running the full test suite probably is a matter of 
few seconds.

> So would I. The main obstacle is just to specify how
> to structure the tests. I'm sure a couple of us could write
> the test harness once that is done, and someone can volunteer
> to collect tests from this mailing list and glue them into CVS.
> 
> How about this:
> 
> (a) Each test is called 'test_*.ml'.
> (b) Each is a program linked against ExtLib.
> (c) The test passes if 
> 
>    i) it exits with 0
>   ii) stderr is empty
>   iii) stdout agrees with 'test_*.expected'
> 
> (d) Each test optionally has a file 'test_*.input'
> which is passed via stdin.
> 
> An alternative is: the first line of the test
> is the checker: eg:
> 
> (* env FRED=something test_298 <somein >tmp.tmp && diff tmp.tmp
> test_298.expected *)
> 
> which allows command line args etc.

What I have done for my O'Caml libraries, is just to write one .ml file 
per module and then have separate tests as functions inside this file. 
Then I create one executable that links all these different test modules 
together to form the full tester for the library.  I just call all the 
modules from my "main".

It is somewhat unclear to me what we would gain by having all tests as 
separate executables.  I believe this kind of a framework is well suited 
for a program like GCC, but I believe it is a little overkill for Extlib.

But then again, having this machinery might make it more easy and 
convenient to add new tests into the suite.

Best regards,
Janne Hellsten