From: Nicolas C. <war...@fr...> - 2004-03-19 08:30:02
|
Hi list, I'm pretty sure all of you have been reading the caml-list recently, I had some private talks with Benjamin Geer concerning ExtLib. His point was that if ExtLib was not just an extension to the ocaml stdlib but a full replacement - including more modules and more functionality - then it will fasten its spread and ease its usage. I kindof agree with him : currently there is several ways of working with ExtLib, and you can have some part of your project using stdlib while others parts are using ExtLib. That's convenient, but in fact since we're keeping 99% compability with stdlib, having the user to port the small parts of his code and then he can use all extensions provided as if he was using stdlib seems to be a fair trade. Of course this means some changes to current project, and that's why I would like to get your comments and advices about this - I'm especialy interested in current ExtLib's contributors opinions. Concerning additionnals modules here's what's on my wish-list (which is somehow also on my current todo-list) : - base64 encode/decode - abstract high level I/O with support for C basic types ( read_i16 , write_f16 ..... ) - zlib deflate/inflate written in pure OCaml. Regards, Nicolas Cannasse |
From: Nicolas C. <war...@fr...> - 2004-12-16 09:24:33
|
Hi list, It's been long time without activity on this list, I have been quite busy lately but I keep on fixing bugs when reports are coming sometimes ;). Right now the ExtLib acheivements are pretty good : I am a happy ExtLib user and I don't feel anymore when I write Ocaml that I miss all theses functions from Stdlib. That's good, and I hope you feel same. As for the future, I am not certain which path we should follow. I tend to appreciate the current shape of ExtLib and I'm not wishing to include a lot more modules, but other people might think different. What I would like to be included in ExtLib would be small Xml and Date modules, but I'm not sure we can reach agreement on what should be put inside theses ;). Any insights or wishes are welcome ! Regards, Nicolas Cannasse |
From: Bardur A. <oca...@sc...> - 2004-12-16 09:47:50
|
On Thu, Dec 16, 2004 at 10:22:05AM +0100, Nicolas Cannasse wrote: > Hi list, > > It's been long time without activity on this list, I have been quite busy > lately but I keep on fixing bugs when reports are coming sometimes ;). Right > now the ExtLib acheivements are pretty good : I am a happy ExtLib user and I > don't feel anymore when I write Ocaml that I miss all theses functions from > Stdlib. That's good, and I hope you feel same. > > As for the future, I am not certain which path we should follow. I tend to > appreciate the current shape of ExtLib and I'm not wishing to include a lot > more modules, but other people might think different. What I would like to > be included in ExtLib would be small Xml and Date modules, but I'm not sure > we can reach agreement on what should be put inside theses ;). Any insights > or wishes are welcome ! > I am also quite happy with it as is, but there are a couple of things I miss: 1) More flexible String functions, e.g.: - lchop_if: Only removes char if matching some set of chars (and *never* throws an exception). Actually, I'd probably rename lchop to lchop_any and add a "which-chars" argument to lchop. Alternatively, one could just use an optional argument for one lchop function. - nsplit should take an optional maximum-number-of-splits argument - rstrip/lstrip instead of just strip - A few more less important, but nonetheless occasionally useful additions Currently, I'm very busy with Real Life(tm), but I hope to submit patches for comment/inclusion in the not too distant future. 2) Path manipulation: An extended and "regularized" (think 'principle of least surprise') version of Filename. In fact, I have a "Path" module up at http://www.imada.sdu.dk/~bardur/personal/45-programs/index.html (search for "ocaml-path") with various commonly needed functions which I'd love feedback on. It's currently UNIX-only, but that can be remedied (however, probably not by me since I don't have any other platforms to test on, and probably don't have sufficient knowledge of the idiosyncrasies of those platforms anyway). Clearly, this needs to be as cross-platform as possible before inclusion is even considered. Cheers, -- Bardur Arantsson <ba...@im...> <ba...@sc...> - Who do you think actually compresses and decompresses an MP3 or a DivX file? Magical forest smurfs? bugmaster | http://kuro5hin.org |
From: skaller <sk...@us...> - 2004-12-17 07:39:05
|
On Thu, 2004-12-16 at 20:47, Bardur Arantsson wrote: > 2) Path manipulation: An extended and "regularized" (think > 'principle of least surprise') version of Filename. > > In fact, I have a "Path" module up at > > http://www.imada.sdu.dk/~bardur/personal/45-programs/index.html > (search for "ocaml-path") I guess this is the best around: http://webperso.easyconnect.fr/gildor/ocaml-fileutils.html It supports MacOS, Windows, Cygwin and Unix using functors, one of which is the host system. Documentation here: http://webperso.easyconnect.fr/gildor/ocaml-fileutils/index.html AUTHOR: Sylvain LE GALL <syl...@po...> ---------------- ocaml-fileutils. ---------------- This library is intended to provide a basic interface to the most common file and filename operation. It provides different filename function : reduce, make_absolute, make_relative... It also enables to manipulate real file : cp, mv, rm, touch... It is separated in two modules : SysUtil and SysPath. The first one manipulate files ( real one ), the second one is made for manipulating abstract filename. -- John Skaller, mailto:sk...@us... voice: 061-2-9660-0850, snail: PO BOX 401 Glebe NSW 2037 Australia Checkout the Felix programming language http://felix.sf.net |
From: Bardur A. <oca...@sc...> - 2004-12-17 09:09:22
|
On Fri, Dec 17, 2004 at 06:38:58PM +1100, skaller wrote: > On Thu, 2004-12-16 at 20:47, Bardur Arantsson wrote: > > > 2) Path manipulation: An extended and "regularized" (think > > 'principle of least surprise') version of Filename. > > > > In fact, I have a "Path" module up at > > > > http://www.imada.sdu.dk/~bardur/personal/45-programs/index.html > > (search for "ocaml-path") > > http://webperso.easyconnect.fr/gildor/ocaml-fileutils.html This looks *very* interesting and the license seems to be ExtLib-compatible. :) However, for glancing at the code there are a couple of minor issues: - It uses lexers and parsers for reducing path names to an internal representation which seems like overkill. (Or is it just me?) - The FileUtil module seems a bit rough around the edges, for instance FileUtil.cmp uses Stream.of_channel to compare files character-by-character -- hopefully the channel itself does readahead, but even so it is probably still hideously slow. The size handling also seems strange (why not just always convert input sizes to number of bytes?), but again, maybe that's just me. Overall, I must say I like it a lot. It has very little in the way of OS-specific code and it already has a reasonable set of test cases. ... which brings up another thing which ExtLib is missing: A test suite (and some mechanism for executing it). We have already seen examples of bugs which unit testing could easily have caught (read_i32, DynArray.append). It also provides some reasonable degree of regression testing if implementations are changed. I think we should definitely add one, especially since there are quite a few modules already which could benefit from it. -- Bardur Arantsson <ba...@im...> <ba...@sc...> Anyone who is capable of getting themselves made President should on no account be allowed to do the job. Douglas Adams | Hitchiker's Guide to the Galaxy |
From: skaller <sk...@us...> - 2004-12-17 10:13:21
|
On Fri, 2004-12-17 at 20:09, Bardur Arantsson wrote: > > http://webperso.easyconnect.fr/gildor/ocaml-fileutils.html > > This looks *very* interesting and the license seems to be > ExtLib-compatible. :) > > However, for glancing at the code there are a couple of > minor issues: > > - It uses lexers and parsers for reducing path names to an > internal representation which seems like overkill. (Or > is it just me?) Doing things 'the right way' can't be overkill can it? I would expect this to be lightning fast and it should make it easy to generalise/extend ..? > - The FileUtil module seems a bit rough around the edges, Perhaps -- but I think the interface is the primary concern .. the implementation can always be improved later. > ... which brings up another thing which ExtLib is missing: > A test suite (and some mechanism for executing it). I raise that issue before -- as I recall, Nicolas correctly said that's it is a *big* job. For Felix, I merge the tests and tutorial.. an added incentive. Learn how to use Extlib by studying the test cases.. But it is a big job .. more is involved that just writing the test case (a test harness is needed too ..) -- John Skaller, mailto:sk...@us... voice: 061-2-9660-0850, snail: PO BOX 401 Glebe NSW 2037 Australia Checkout the Felix programming language http://felix.sf.net |
From: Bardur A. <oca...@sc...> - 2004-12-17 11:37:04
|
On Fri, Dec 17, 2004 at 09:13:04PM +1100, skaller wrote: > On Fri, 2004-12-17 at 20:09, Bardur Arantsson wrote: > > > http://webperso.easyconnect.fr/gildor/ocaml-fileutils.html > > > > This looks *very* interesting and the license seems to be > > ExtLib-compatible. :) > > > > However, for glancing at the code there are a couple of > > minor issues: > > > > - It uses lexers and parsers for reducing path names to an > > internal representation which seems like overkill. (Or > > is it just me?) > > Doing things 'the right way' can't be overkill can it? > I would expect this to be lightning fast and it should > make it easy to generalise/extend ..? I'm not sure there will ever be any need to generalise/extend, but anyway... To my mind, using full-blown parsers is overkill for splitting UNIX paths into their constituent parts, but I guess that's just an implementation detail, so let's ignore that for the moment. Another concern, which is more related to the interface is that the module seems to raise exceptions in situations one wouldn't normally expect. An an example: FilePath.check_extension p ext raises an exception if the filename doesn't have an extension. I can't tell whether this is a part of the interface of check_extension since no ocamldoc comment is present, but it certainly is *not* following the principle of least surprise. That's the kind of thing that can be *really* annoying to a library user. Apart from that, I feel that simple 'shortcut' path queries like is_dir, is_link, etc. should be added to the FilePath module. I realize that this removes the separation of purely abtract paths and concrete files, but it's just too convenient to pass up IMO. Constrast if Path.is_dir p then with if FileUtil.test FileUtil.Is_dir p then Not a huge difference, but it is significant enough, IMO. > > > - The FileUtil module seems a bit rough around the edges, > > Perhaps -- but I think the interface is the primary > concern .. the implementation can always be improved later. You're right, of course. All of the above is stuff that's fixable, so I guess the best idea would be to just decide on an answer to the question Do we want a path/file query/manipulation module in ExtLib? My answer would definitely be 'yes' (regardless of which particular module it would end up being). This is an area where the standard library is very weak and such a module would be useful to most people using ExtLib. I would be quite happy to use FilePath/FileUtil as the basis for such a module. > > > ... which brings up another thing which ExtLib is missing: > > A test suite (and some mechanism for executing it). > > I raise that issue before -- as I recall, Nicolas correctly > said that's it is a *big* job. Agreed, but IMHO it is becoming increasingly necessary as the amount of code in ExtLib grows. Just an example from real life: When I was writing OptParse I would have a test program invoke the parser with various test cases just to make sure that the various modifications I was doing didn't create new bugs. When it was included in ExtLib, those test cases couldn't be included and any future modifications would run the risk of causing regressions. This may also cause people who don't know the code intimately to be relucant to try to improve/fix it. (In some cases this even extends to the original authors if they haven't looked at the code in question for a long time). > For Felix, I merge the tests and tutorial.. an added incentive. > Learn how to use Extlib by studying the test cases.. I'm not sure how necessary an ExtLib tutorial is, but that sound like a good idea. :) > But it is a big job .. more is involved that just writing > the test case (a test harness is needed too ..) Maybe I'm just stupid, but I don't see why a test harness would require lots of work...? My suggestion would be something like this: Use OUnit for running individual test cases/sets, add a lazy global 'test_cases' value to all the relevant modules. Finally, add a Test module containing something along the lines of let all_test_cases = ... [ Lazy.force ExtString.test_cases; Lazy.force OptParse.test_cases; ... ] in let counters = OUnit.run_test_tt_main all_test_cases in let rc = if counters.errors > 0 then 1 else if counters.failures > 0 then 2 else 0 in printf "statistics: ...."; exit rc; This would enable all the installation methods to run the test cases with very little effort. Of course, writing individual test cases for all the modules would be a lot of work, but this is work that can be done incrementally. Am I missing something? -- Bardur Arantsson <ba...@im...> <ba...@sc...> |
From: Nicolas C. <war...@fr...> - 2004-12-17 12:07:51
|
> Of course, writing individual test cases for all the > modules would be a lot of work, but this is work that can > be done incrementally. > > Am I missing something? Yes : who is volunteering for such a work ? I'll skip ;) Nicolas |
From: Bardur A. <oca...@sc...> - 2004-12-17 12:43:30
|
On Fri, Dec 17, 2004 at 01:05:56PM +0100, Nicolas Cannasse wrote: > > Of course, writing individual test cases for all the > > modules would be a lot of work, but this is work that can > > be done incrementally. > > > > Am I missing something? > > Yes : who is volunteering for such a work ? > I'll skip ;) > Actually, I would, but I'm currently too busy :(. Maybe in a month or two... -- Bardur Arantsson <ba...@im...> <ba...@sc...> - This is the worst kind of discrimination. The kind against me! Bender | Futurama |
From: skaller <sk...@us...> - 2004-12-17 13:07:45
|
On Fri, 2004-12-17 at 22:36, Bardur Arantsson wrote: > > Doing things 'the right way' can't be overkill can it? > > I would expect this to be lightning fast and it should > > make it easy to generalise/extend ..? > > I'm not sure there will ever be any need to > generalise/extend, but anyway... Consider: (a) C has certain rules for resolving #include filenames. Recall eveb "kjkjh" and <kjhgkjg> are distinct .. (b) Felix (and Interscript) also have rules for resolving filenames. Interscript rules are in fact quite interesting: Names of include files in interscript are *required* to be Unix relative filenames. From the command line, a native filename prefix is taken. The two are spliced together, after converting the interscript name to a native one. [This ensures everything 'in document' is as OS independent as possible] (c) There are several conventions for PATH names. The unix one (separator ':') is one, but TeX uses kpathsea .. (d) I have used an archaic system which is much better than any of the above .. a TI os which has no subdirectories at all. Instead it has something much better: an environment plus structured filename convention. Filename components are replaced from the environment, subsuming the current directory idea completely. (e) Hmm what about URL/URI things .. :) In any case the idea of using a parser for filenames isn't overkill IMHO .. on the contrary the problem is more likely to be that mere LALR1 parsing and Ocamllex lexing simply isn't good enough (some OS use UCS-2 filenames .. solaris and Win32 for example ..) Ocamllex, for example, can't even translate UTF-8 (I tried once, it blows the lexer generators brains out). The real advantage of a lexer/parser combination seems to be that the specification is heavily declarative. > To my mind, using full-blown parsers is overkill for > splitting UNIX paths into their constituent parts, But we're not restricted to Unix.. > Another concern, which is more related to the interface is > that the module seems to raise exceptions in situations > one wouldn't normally expect. An an example: > > FilePath.check_extension p ext > > raises an exception if the filename doesn't have an > extension. I can't tell whether this is a part of the > interface Agree. I think exceptions should be reserved for unrecoverable errors if possible, but in any case it should be documented. > Apart from that, I feel that simple 'shortcut' path > queries like is_dir, is_link, etc. should be added to the > FilePath module. I realize that this removes the > separation of purely abtract paths and concrete files, but > it's just too convenient to pass up IMO. I think the separation is an *essential* feature. It's there by design and for a good reason: you can have Unix, MacOS, and Win32 modules all available at once. These modules must not be permitted to touch the file system. However short version in the FileUtil module, OR, a third module, (eg FileQuery) may make sense. (Especially as 'fstat' and friends are fairly OS specific animals) > All of the above is stuff that's fixable, so I guess the > best idea would be to just decide on an answer to the > question > > Do we want a path/file query/manipulation module in ExtLib? > > My answer would definitely be 'yes' Mine too. Filenames are needed even in Pervasives .. > Maybe I'm just stupid, but I don't see why a test harness > would require lots of work...? Because it has to (a) Collate all the tests (b) Run the tests -- terminating rogues (c) Collate the results (d) Standardise a way to actually report results Point (d) is extremely difficult. > Of course, writing individual test cases for all the > modules would be a lot of work, but this is work that can > be done incrementally. Yes, I don't think writing the tests is the issue. We can start with just half a dozen, and make sure for every bug there is a regression test generated. > Am I missing something? Yeah -- designing a test system is probably harder than designing a library or an average application. This is especially the case for Ocaml I suspect, since it doesn't support dynamic loading and unloading. That seems to mean each test must be a separate process. In addition some tests are dangerous, especially ones that mess with the filesystem or network -- usually you'd run your tests in a suitably restricted environment as an low privilege user.. which makes the test harnes OS specific .. In addition, we will need community support: something like a web page for submitting tests, so that they're easy to install, and meet requirements, such as having a description, expected output, etc .. I personally think 'unit' testing is a bit silly. It rarely finds bugs because it can't cover enough cases, can't handle integration, etc.. I prefer a more sloppy concept of just collecting whatever test code you can and running it, just to get some confidence you didn't completely mess something up whilst committing a minor change to CVS. -- John Skaller, mailto:sk...@us... voice: 061-2-9660-0850, snail: PO BOX 401 Glebe NSW 2037 Australia Checkout the Felix programming language http://felix.sf.net |
From: Bardur A. <oca...@sc...> - 2004-12-17 13:41:10
|
On Sat, Dec 18, 2004 at 12:07:30AM +1100, skaller wrote: > > To my mind, using full-blown parsers is overkill for > > splitting UNIX paths into their constituent parts, > > But we're not restricted to Unix.. I was thinking specifically of the UNIX case... [--snip--] > > Maybe I'm just stupid, but I don't see why a test harness > > would require lots of work...? > > Because it has to > > (a) Collate all the tests > (b) Run the tests -- terminating rogues > (c) Collate the results > (d) Standardise a way to actually report results ... all of which my small code snippet does (well, would do if it were actually syntactically valid...) when combined with OUnit...?!? (a) Tests are combined in all_tests. (b) OUnit takes care of that (there is no parallel execution, but that's hardly a huge problem for testing ExtLib code). (c) OUnit (d) ./test returns 0 if all tests were succesful, non-zero otherwise. This enables the "make check" target to fail, install.ml can run ./test and check its return code. I think we may be discussing different things... I'm only talking about something which ExtLib developers can use exclusively for testing ExtLib itself, not about making a test harness available to ExtLib users. > > Am I missing something? > > Yeah -- designing a test system is probably harder > than designing a library or an average application. > This is especially the case for Ocaml I suspect, > since it doesn't support dynamic loading and unloading. > That seems to mean each test must be a separate process. In general, yes, but it doesn't mean that we can't agree on an *internal* method for testing ExtLib itself. > > In addition some tests are dangerous, especially ones > that mess with the filesystem or network -- usually you'd run > your tests in a suitably restricted environment > as an low privilege user.. which makes the test > harnes OS specific .. Sure, but what I'm talking about is much simpler: *We* would be the ones writing the tests, so we simply avoid writing test cases for potentially dangerous things (ie. tests with side effects outide of the "virtual machine"). Thus, there is no need to protect from such secondary effects. Of course it means less coverage, but I can certainly live with that, especially since the alternative is *no* coverage. > > I personally think 'unit' testing is a bit silly. > It rarely finds bugs because it can't cover enough cases, > can't handle integration, etc.. I prefer a more sloppy > concept of just collecting whatever test code you can > and running it, just to get some confidence you didn't > completely mess something up whilst committing a minor > change to CVS. Yes, perhaps I shouldn't have used the term 'unit test', but I was thinking along the same lines you are, ie. basic sanity/logic tests and regression tests. Trusting unit tests to find all bugs is silly, but they can provide a bit more confidence that you haven't introduced really obvious bugs... -- Bardur Arantsson <ba...@im...> <ba...@sc...> - "Eternal happiness for one dollar? I'd be happier with the dollar." Montgomery Burns | The Simpsons |
From: Janne H. <ja...@hy...> - 2004-12-17 22:42:27
|
Bardur Arantsson wrote: > >>I personally think 'unit' testing is a bit silly. >>It rarely finds bugs because it can't cover enough cases, >>can't handle integration, etc.. I prefer a more sloppy >>concept of just collecting whatever test code you can >>and running it, just to get some confidence you didn't >>completely mess something up whilst committing a minor >>change to CVS. >> >> > >Yes, perhaps I shouldn't have used the term 'unit test', >but I was thinking along the same lines you are, ie. basic >sanity/logic tests and regression tests. Trusting unit >tests to find all bugs is silly, but they can provide a >bit more confidence that you haven't introduced really >obvious bugs... > > I find these types of tests extremely handy for developing any software. In fact we use this kinds of sanity/regression tests in all our software at our company. What I normally do (for C and O'Caml) is that I use a lot of asserts to do sanity checks in library code and then write a tester (sometimes called smoketester) that calls the library functions with heavily biased random input. The input is biased towards corner cases, such as array length zero etc. Every test case will naturally test that the function(s) it is testing is working in the expected manner. The tester's "coverage" can be easily metered with gcov. This way you can get sort of profiling of your test code.. E.g., only spend time writing test cases that actually reach yet unentered parts of the library. I have personally found this technique to really effective in development. Modifying large parts of code is much easier if you have a good tester that actually tests everything. I guess for Extlib, you're always assumed to pass *all* tests before making a release or committing changes. If this is the case, I don't see much need for fancy logging machinery, since the tests would pretty much just report PASSED. I don't think it would be so hard to find volunteers for this kind of testing. In fact I'd be willing to write a few tests for it myself. Best regards, Janne Hellsten |
From: skaller <sk...@us...> - 2004-12-17 23:52:08
|
On Sat, 2004-12-18 at 09:42, Janne Hellsten wrote: > I find these types of tests extremely handy for developing any > software. In fact we use this kinds of sanity/regression tests in all > our software at our company. What I normally do (for C and O'Caml) is > that I use a lot of asserts to do sanity checks in library code and then > write a tester (sometimes called smoketester) that calls the library > functions with heavily biased random input. The input is biased towards > corner cases, such as array length zero etc. Every test case will > naturally test that the function(s) it is testing is working in the > expected manner. How do you (a) generate the input and (b) predict the expected result? > I have personally found this technique to really effective in > development. Modifying large parts of code is much easier if you have a > good tester that actually tests everything. I *rely* heavily on my tests in Felix. In fact they're part of the distro and the build always runs the basic ones. I do lots of 'refactoring' and enhancements and I'd be screwed without the testing harness. > I guess for Extlib, you're always assumed to pass *all* tests before > making a release or committing changes. Yup, but when a test fails you need to know which one :) > I don't think it would be so hard to find volunteers for this kind of > testing. In fact I'd be willing to write a few tests for it myself. So would I. The main obstacle is just to specify how to structure the tests. I'm sure a couple of us could write the test harness once that is done, and someone can volunteer to collect tests from this mailing list and glue them into CVS. How about this: (a) Each test is called 'test_*.ml'. (b) Each is a program linked against ExtLib. (c) The test passes if i) it exits with 0 ii) stderr is empty iii) stdout agrees with 'test_*.expected' (d) Each test optionally has a file 'test_*.input' which is passed via stdin. An alternative is: the first line of the test is the checker: eg: (* env FRED=something test_298 <somein >tmp.tmp && diff tmp.tmp test_298.expected *) which allows command line args etc. -- John Skaller, mailto:sk...@us... voice: 061-2-9660-0850, snail: PO BOX 401 Glebe NSW 2037 Australia Checkout the Felix programming language http://felix.sf.net |
From: Bardur A. <oca...@sc...> - 2004-12-18 00:30:57
|
On Sat, Dec 18, 2004 at 10:51:54AM +1100, skaller wrote: > > I guess for Extlib, you're always assumed to pass *all* tests before > > making a release or committing changes. > > Yup, but when a test fails you need to know which one :) What about just having a -verbose switch for the ./test_*.ml program which prints the number of each test before running it...? Seems simple enough to me... Of course, stack traces would really be the optimal way to do this, but OCaml doesn't support stack traces for exceptions (although the debugger does... go figure). > > > I don't think it would be so hard to find volunteers for this kind of > > testing. In fact I'd be willing to write a few tests for it myself. > > So would I. The main obstacle is just to specify how > to structure the tests. I'm sure a couple of us could write > the test harness once that is done, and someone can volunteer > to collect tests from this mailing list and glue them into CVS. > > How about this: > > (a) Each test is called 'test_*.ml'. It might be a bit less cluttered to have a directory (or multiple directories?) exclusively for tests, but I guess any convention which clearly separates test from non-test would be fine. > (b) Each is a program linked against ExtLib. Yup. > (c) The test passes if > > i) it exits with 0 > ii) stderr is empty > iii) stdout agrees with 'test_*.expected' > > (d) Each test optionally has a file 'test_*.input' > which is passed via stdin. Wouldn't it be easier to just hardcode input/expected values into the test_*.ml files themselves instead of relying on "external" files? Certainly, all the things I've thought about writing test cases for would be just as easy to hardcode input/output values for. > > An alternative is: the first line of the test > is the checker: eg: > > (* env FRED=something test_298 <somein >tmp.tmp && diff tmp.tmp > test_298.expected *) > > which allows command line args etc. Wouldn't this necessitate a UNIX-ish environment? -- Bardur Arantsson <ba...@im...> <ba...@sc...> Friends are like plants. They need attention and they need to drink. SPYvSPY | http://kuro5hin.org |
From: Janne H. <ja...@hy...> - 2004-12-18 00:34:51
|
> How do you (a) generate the input and (b) predict the > expected result? It is wholly dependent on the test, of course. For example, random strings with random length and contents, possibly having zero elements in it. Then feeding it into a function and verifying with asserts that the output is what it's supposed to be. The randomness depends heavily on the problem domain. With programs such as compilers, you want a lot of randomness, but for simple things like string functions you're fine with just a few carefully selected inputs. In one compiler project I used an AST and a code generator that generated code based on that AST. When generating code, you have a lot of opportunities for randomly creating different variations so that the generated output still does the same thing. I fed the randomly generated programs through the compiler, ran them and compared the results against a direct interpretation of the AST (the AST was conveniently interpretable, as it was created for this purpose only). To predict the output, you of course need to create cases where you in some way (programmatically) know the result. For example, if you know that a function should raise an exception with some input, you feed it with that input and then assert that it actually raises it. I'm sure everyone is familiar with this sort of testing. I don't have my tester for one of my libraries at hand, but perhaps tomorrow I can post some examples if you're interested. > I *rely* heavily on my tests in Felix. > In fact they're part of the distro and the build always runs the > basic ones. > > I do lots of 'refactoring' and enhancements and I'd be screwed > without the testing harness. Yep, same here. >>I guess for Extlib, you're always assumed to pass *all* tests before >>making a release or committing changes. > > > Yup, but when a test fails you need to know which one :) Because I use assertions. I can do just: export OCAMLRUNPARAM=b=1 make all make test and then examine the callstack to see where something went wrong. Or use a debugger. I don't see any reason why it should be any more complicated than that. As Extlib, at least in its current, form is relatively small, running the full test suite probably is a matter of few seconds. > So would I. The main obstacle is just to specify how > to structure the tests. I'm sure a couple of us could write > the test harness once that is done, and someone can volunteer > to collect tests from this mailing list and glue them into CVS. > > How about this: > > (a) Each test is called 'test_*.ml'. > (b) Each is a program linked against ExtLib. > (c) The test passes if > > i) it exits with 0 > ii) stderr is empty > iii) stdout agrees with 'test_*.expected' > > (d) Each test optionally has a file 'test_*.input' > which is passed via stdin. > > An alternative is: the first line of the test > is the checker: eg: > > (* env FRED=something test_298 <somein >tmp.tmp && diff tmp.tmp > test_298.expected *) > > which allows command line args etc. What I have done for my O'Caml libraries, is just to write one .ml file per module and then have separate tests as functions inside this file. Then I create one executable that links all these different test modules together to form the full tester for the library. I just call all the modules from my "main". It is somewhat unclear to me what we would gain by having all tests as separate executables. I believe this kind of a framework is well suited for a program like GCC, but I believe it is a little overkill for Extlib. But then again, having this machinery might make it more easy and convenient to add new tests into the suite. Best regards, Janne Hellsten |
From: Janne H. <ja...@hy...> - 2004-12-18 00:48:15
|
Sorry, forgot to say that.. > Because I use assertions. I can do just: > > export OCAMLRUNPARAM=b=1 > make all > make test > > and then examine the callstack to see where something went wrong. The above works only for bytecode (callstack). But asserts print out the file:line information where the assertion failed. So if something goes wrong, you always get notified and can see easily what went wrong and where. Best regards, Janne Hellsten |
From: skaller <sk...@us...> - 2004-12-18 01:41:56
|
On Sat, 2004-12-18 at 11:34, Janne Hellsten wrote: > It is somewhat unclear to me what we would gain by having all tests as > separate executables. I believe this kind of a framework is well suited > for a program like GCC, but I believe it is a little overkill for Extlib. > But then again, having this machinery might make it more easy and > convenient to add new tests into the suite. That was my thought -- we want to encourage community contribution, which I think implies being able to write lots of little self contained tests, in particular to actually check them separately. However it is slower to run the tests this way and needs more external machinery. -- John Skaller, mailto:sk...@us... voice: 061-2-9660-0850, snail: PO BOX 401 Glebe NSW 2037 Australia Checkout the Felix programming language http://felix.sf.net |
From: Janne H. <ja...@hy...> - 2004-12-18 07:33:28
|
skaller wrote: > That was my thought -- we want to encourage community > contribution, which I think implies being able to > write lots of little self contained tests, in particular > to actually check them separately. After thinking about this a little, I think I'd still rather go for the the simple assertion-based system I proposed. This is because the framework is almost non-existent and thus doesn't take much time to implement (~none). The assertion-based tests work well with other systems as well, and you could integrate these kinds of tests easily into a system that you were proposing -- just make sure you compile all these executables with asserts turned on. > However it is slower to run the tests this way > and needs more external machinery. Yes, much slower, depending on the tests. Especially Cygwin where process invokation is dead slow for some reason. The testing suite also needs to build and run easily on Windows. Best regards, Janne Hellsten |
From: Richard J. <ri...@an...> - 2004-12-18 11:39:01
|
On Sat, Dec 18, 2004 at 09:33:13AM +0200, Janne Hellsten wrote: > Yes, much slower, depending on the tests. Especially Cygwin where=20 > process invokation is dead slow for some reason. >=20 > The testing suite also needs to build and run easily on Windows. How about lots of tiny test programs (test1.ml, test2.ml, test_string_exists.ml, ...) which are linked together into a main program. The main program (test.ml) would be generated by the Makefile and would basically consist of something like: let () =3D Test1.test (); Test2.test (); Test_string_exists.test (); (* ... *) print_endline "Tests OK" Rich. --=20 Richard Jones. http://www.annexia.org/ http://www.j-london.com/ >>> http://www.team-notepad.com/ - collaboration tools for teams <<< Merjis Ltd. http://www.merjis.com/ - improving website return on investment Learn Objective CAML - http://www.merjis.com/developers |
From: Janne H. <ja...@hy...> - 2004-12-18 11:47:36
|
> How about lots of tiny test programs (test1.ml, test2.ml, > test_string_exists.ml, ...) which are linked together into a main > program. The main program (test.ml) would be generated by the > Makefile and would basically consist of something like: > > let () = > Test1.test (); > Test2.test (); > Test_string_exists.test (); > (* ... *) > print_endline "Tests OK" This is pretty much what I was proposing, although I'd probably put more tests into one .ml file. I will make an initial test today and post a link to the results here. Perhaps interested people could take a look and we could continue after the initial review. I'm considering the use of OMake for the tester. I wonder if people are much against that? At least it works on all platforms and is much much nicer than GNU Make. ciao, janne |
From: Bardur A. <oca...@sc...> - 2004-12-18 11:55:21
|
On Sat, Dec 18, 2004 at 01:47:15PM +0200, Janne Hellsten wrote: > I'm considering the use of OMake for the tester. I wonder if people are > much against that? At least it works on all platforms and is much much > nicer than GNU Make. I've been playing around with OMake and it *is* much nicer than GNU Make (and SCons, jam, and cook, IMO), but there are still some relatively serious issues with the current (pre-1.0) releases, so I don't think it would be a good idea to use it for ExtLib just yet. Most of the issues are being addressed and it looks like OMake may become a very nice build system (not only for OCaml-based projects, but also in general). Cheers, -- Bardur Arantsson <ba...@im...> <ba...@sc...> - Witty remarks are like... uh, the things with... um. Uh, I think they went over there. Emily Dickinson |
From: Janne H. <ja...@hy...> - 2004-12-18 12:01:42
|
Bardur Arantsson wrote: > I've been playing around with OMake and it *is* much nicer > than GNU Make (and SCons, jam, and cook, IMO), but there > are still some relatively serious issues with the current > (pre-1.0) releases, so I don't think it would be a good > idea to use it for ExtLib just yet. > > Most of the issues are being addressed and it looks like > OMake may become a very nice build system (not only for > OCaml-based projects, but also in general). > > Cheers, Yes, I guess you're right. And I maybe it's not a good idea to require OMake installation in order to run the testing suite. ciao, janne |
From: skaller <sk...@us...> - 2004-12-18 12:39:03
|
On Sat, 2004-12-18 at 22:38, Richard Jones wrote: > The main program (test.ml) would be generated by the > Makefile and would basically consist of something like: > > let () = > Test1.test (); > Test2.test (); > Test_string_exists.test (); > (* ... *) > print_endline "Tests OK" OK, but the naming 'test999.ml' sux :) Random contributions require a more consistent naming scheme. So how about: test_<author>_<module>_<name>.ml <author> is the authors initials eg rj, js, nc, jh. <module> is the Extlib module being tested, or 'multi' if more than one. <name> is a suggestion what the test is about, possibly with some digits, abbreviated, or just a random code of some kind. All the letters shall be lower case. Example: test_js_enum_iter01.ml The tests all get checked into a CVS repository module distinct from the library. Nicolas will name and create it? Within that module, a directory 'utests' will contain the actual tests .. and nothing else. The Makefile, mainline, etc should be kept separate from the tests. The idea is basically anyone can add tests, but one person should be responsible for the harness. Comments, improvements, violent disagreements? -- John Skaller, mailto:sk...@us... voice: 061-2-9660-0850, snail: PO BOX 401 Glebe NSW 2037 Australia Checkout the Felix programming language http://felix.sf.net |
From: Richard W.M. J. <ri...@me...> - 2004-12-18 12:48:49
|
On Sat, Dec 18, 2004 at 11:38:31PM +1100, skaller wrote: > OK, but the naming 'test999.ml' sux :) Yup. > Random contributions require a more consistent naming scheme. > So how about: >=20 > test_<author>_<module>_<name>.ml Seems sensible. > <author> is the authors initials eg rj, js, nc, jh. > <module> is the Extlib module being tested, or 'multi' if more > than one. Should we have tests which test more than one module? Rich. --=20 Richard Jones. http://www.annexia.org/ http://www.j-london.com/ >>> http://www.team-notepad.com/ - collaboration tools for teams <<< Merjis Ltd. http://www.merjis.com/ - improving website return on investment http://subjectlink.com/ - Lesson plans and source material for teachers |
From: Janne H. <ja...@hy...> - 2004-12-18 15:42:33
|
>> Random contributions require a more consistent naming scheme. >> So how about: >> >> test_<author>_<module>_<name>.ml > > Seems sensible. > >> <author> is the authors initials eg rj, js, nc, jh. >> <module> is the Extlib module being tested, or 'multi' if more >> than one. > > Should we have tests which test more than one module? We should have tests which test more than one module. But I think we should run single module tester first to ease debugging. I already started working on the tester, hope to post an initial tester for review in a few hours. ciao, janne |