Re: [Ocaml-lib-devel] Future of ExtLib

[skaller <sk...@us...>]

On Sat, 2004-12-18 at 09:42, Janne Hellsten wrote:

> I find these types of tests extremely handy for developing any 
> software.  In fact we use this kinds of sanity/regression tests in all 
> our software at our company.  What I normally do (for C and O'Caml) is 
> that I use a lot of asserts to do sanity checks in library code and then 
> write a tester (sometimes called smoketester) that calls the library 
> functions with heavily biased random input.  The input is biased towards 
> corner cases, such as array length zero etc.  Every test case will 
> naturally test that the function(s) it is testing is working in the 
> expected manner.

How do you (a) generate the input and (b) predict the
expected result?

> I have personally found this technique to really effective in 
> development.  Modifying large parts of code is much easier if you have a 
> good tester that actually tests everything.

I *rely* heavily on my tests in Felix.
In fact they're part of the distro and the build always runs the
basic ones.

I do lots of 'refactoring' and enhancements and I'd be screwed
without the testing harness.

> I guess for Extlib, you're always assumed to pass *all* tests before 
> making a release or committing changes.  

Yup, but when a test fails you need to know which one :)

> I don't think it would be so hard to find volunteers for this kind of 
> testing.  In fact I'd be willing to write a few tests for it myself.

So would I. The main obstacle is just to specify how
to structure the tests. I'm sure a couple of us could write
the test harness once that is done, and someone can volunteer
to collect tests from this mailing list and glue them into CVS.

How about this:

(a) Each test is called 'test_*.ml'.
(b) Each is a program linked against ExtLib.
(c) The test passes if 

   i) it exits with 0
  ii) stderr is empty
  iii) stdout agrees with 'test_*.expected'

(d) Each test optionally has a file 'test_*.input'
which is passed via stdin.

An alternative is: the first line of the test
is the checker: eg:

(* env FRED=something test_298 <somein >tmp.tmp && diff tmp.tmp
test_298.expected *)

which allows command line args etc.

-- 
John Skaller, mailto:sk...@us...
voice: 061-2-9660-0850, 
snail: PO BOX 401 Glebe NSW 2037 Australia
Checkout the Felix programming language http://felix.sf.net