Re: [Ocaml-lib-devel] Future of ExtLib

[skaller <sk...@us...>]

On Tue, 2004-12-21 at 07:24, Janne Hellsten wrote:
[Makefile .. test structure]

Janne currently has a three level test structure:

1. The mainline calls the test for each module.
2. The module test calls individual tests for each module
3. Individual tests call subtests 

In all 3 cases the call looks like this:

let test () = 
  Util.run_test ~test_name:"jh_ExtList.iteri" test_iteri;
  Util.run_test ~test_name:"jh_ExtList.mapi" test_mapi;
  Util.run_test ~test_name:"jh_ExtList.exceptions" test_exceptions;
  Util.run_test ~test_name:"jh_ExtList.find_exc" test_find_exc

In addition each file has to be listed in the file

test.mak

which is 'triple handling', in fact worse, since the test name
has to be written as well.. :)

Ideally, the test.mak, mainline, and module level
test function should be generated, not hand coded,
that is the Makefile should generate the these things
using a refined equivalent of 'ls test_*.ml'.

I.e. the to create a new test you just commit the
test file -- you don't change any existing files,
instead the files are constructed.

I suppose I can write an Ocaml program which will create these
files. This would work as follows:

(1) Look in extlib-dev for all mli files, to find the
names of top level modules. [Alternative -- hard code
the list of module names]

(2) Generate the mainline 'test_mail.ml' using these module names.

(3) For each module, generate a test file test_<module>.ml,
which calls each test for that module in the extlib-test directory
found with name

test_*_<module>_*.ml

Note this implies names like

test_jh_ExtList.ml

would not be allowed, a suffix would be required.
That could be fixed by also allowing names of that form.
I personally think we should mandate the suffix:
we'd better decide fast, before there are too many tests:
CVS isn't good at file renaming .. :)

Capitalisation rules would apply (the module names must
be exactly as used in Ocaml code). I'm not sure how this
will work on systems that aren't strict about filenames..

An alternative to an Ocaml program to do this would
be a bash script.. probably easier to write but could
be less portable, considering native Win32 environment
for example.

A BIG advantage of automatically generating the
test call protocol, apart from ease of adding new
tests, is that the actual protocol can be changed
to do things like count success/failure, generate
error reports blah blah .. without manual recoding,
except possibly at the individual test level.

It may actually be possible to build the individual
test() functions too, if the test functions within
the individual test files are annotated with
some special kind of mark eg:

TEST inner_test_name
 ......

which is replaced by

let inner_test_name () =

but which also gets added to the harness at the bottom of the
file like this:

let test () =
  Util.run_test ~test_name:"js_DynArray.inner_test_name" inner_test_name
  ...


Thus fully automating the whole suite construction, requiring
only the special form of declaration. This would imply changing
the CVS filenames to something like

test_jh_ExtLib_123.tst

[to avoid overwriting them when generating the *.ml file]

Comments?

-- 
John Skaller, mailto:sk...@us...
voice: 061-2-9660-0850, 
snail: PO BOX 401 Glebe NSW 2037 Australia
Checkout the Felix programming language http://felix.sf.net