From: Peter K. <ps...@cs...> - 2002-12-12 05:42:40
|
Hello, My name is Peter Keller, and I was directed here by Noel Welsh to talk about Scheme Unit and a testing infrastructure of similar design that I had written for the Chicken Scheme compiler(before I knew about Scheme Unit). I think that if the ideas in Scheme Unit and what I have are combined, it will be a very good thing. Here is an API description with some theory of operations description of the testing infrastructure: http://www.call-with-current-continuation.org/chicken-manual.pdf Look in the table of contents under "Additional Files : test-infrastructure.scm". It is pretty comprehensive, but this manual is missing a couple things that are in the cvs repository and not released yet. (Minor things, all told). If you download the 1082 build of chicken, the files that start with test-infrastructure* are the ones you should look at. I read the paper just today about Scheme Unit, and discovered that Scheme Unit and the stuff I have do about 90% of the same things, but I seem to have implemented a lot more "future plan" features that you guys were thinking about and I have some more elaborate control structures than Scheme Unit. Also, there seems to be some information that Scheme Unit is PLT specific. The stuff I have is pure R5RS code(with hygienic macros) and loads with the full feature set into a few popular scheme implementations without modification(I have a seperate implementation specific loader file, and then the real code base, and then an implementation specific code base for certain test structures that only make sense in the local implementation). One of the couple real differences between our code bases is that you chose to handle things like divide by zero errors and I had not(technically you chose option three in the paper about what kinds of things to return from an assertion and I chose option two). This was because I noticed different scheme implementations either did or didn't throw any sort of an exception when things like divide by zero happened and I couldn't find a pure R5RS way of handling it. But, that just might be stupidity on my part. If so, then I can still stay with option 2, but catch the error and handle it intelligently and then return a result object at the assertion level. So the appearance of option two holds. Another major difference is I have elaborate(but not confusingly complex) control structures that allow you to terminate a test case by effectively calling a lexically continuation of a test package that you are contained in. The scope of the termination is preserved and is analyzable later. Also, I don't have such a tighty coupled prologue/epilogue structure like you do for your expectations, instead I implemented a atexit() like feature where you can dynamically build a queue of operations to do when you leave the test case or test package. This, like the termination functions are also lexically scoped. The future things I implemented was an optional warning syntax for test packages so you could supply warning objects(usually strings), test cases, and expectations, a let-like binding form structure for test packages and test cases, and things like a "todo", "side-effect", and "skip" system which allows you to keep track of todo stuff you need to do, code that you only want run for the side effects, and code that you wish to skip all together. Of course, the evaluation of the test suites you generate with my system is decoupled from the analysis and output generation of the results of the tets. Basically a large in memory tree of the results is created that you can analyze any which way you please. Can you guys check out my work and see what you think? An overriding goal of mine is to keep the code R5RS conformant so it works in as many implementations as possible. Thanks. -pete |
From: MJ R. <ma...@cl...> - 2002-12-12 11:10:35
|
Peter Keller <ps...@cs...> wrote: > http://www.call-with-current-continuation.org/chicken-manual.pdf Aside: Is it possible to generate xhtml from that document? > [...] Also, there seems to be some information that Scheme Unit is > PLT specific. [...] I hope Noel doesn't mind me mentioning that a R5RS port exists. I can't remember if that's public knowledge yet. I've not tested it yet, sadly. > because I noticed different scheme implementations either did or didn't > throw any sort of an exception when things like divide by zero happened > and I couldn't find a pure R5RS way of handling it. [...] Yes, you would probably need some SRFI support in the implementation for this one, I think. I hope that initial feedback useful to you. -- MJR http://mjr.towers.org.uk/ IM: sl...@ja... This is my home web site. This for Jabber Messaging. How's my writing? Let me know via any of my contact details. |
From: Peter K. <ps...@cs...> - 2002-12-12 17:12:28
|
On Thu, Dec 12, 2002 at 11:10:25AM +0000, MJ Ray wrote: > Peter Keller <ps...@cs...> wrote: > > http://www.call-with-current-continuation.org/chicken-manual.pdf > > Aside: Is it possible to generate xhtml from that document? Yes. When all of the tests are evaluated, they produce an in memory tree of all of the result types. You may then analyze this tree(how many warnings, terminations, skips, todos, etc) and generate any kind of output from human readable to html to script readable from the tree. I supply a human readable output function in the current API, and when I get around to it (during Christmas holidays) I'll add a nice html output generation of a result tree. > > because I noticed different scheme implementations either did or didn't > > throw any sort of an exception when things like divide by zero happened > > and I couldn't find a pure R5RS way of handling it. [...] > > Yes, you would probably need some SRFI support in the implementation for > this one, I think. Yeah, I figure as long as it is clear cut in the abstraction mechanisms I chose to add this later, I can wait until it is standardized. Thank you. -pete |
From: Noel W. <noe...@ya...> - 2002-12-12 16:03:09
|
Hi Peter, I'll take a look at your testing framework and get back to you. Work is very busy right now but I'll certainly get to it over Xmas. I think the two good things in SchemeUnit are: - define-assertion - the optimised UI Everything else is fairly standard. Noel __________________________________________________ Do you Yahoo!? Yahoo! Mail Plus - Powerful. Affordable. Sign up now. http://mailplus.yahoo.com |
From: Peter K. <ps...@cs...> - 2002-12-12 17:33:12
|
Hello, On Thu, Dec 12, 2002 at 08:02:57AM -0800, Noel Welsh wrote: > Hi Peter, > > I'll take a look at your testing framework and get > back to you. Work is very busy right now but I'll > certainly get to it over Xmas. Cool. I appreciate the time you spend looking into it. I also have most of an R5RS test suite (for the language itself) that you can look at if you want to see an example of the use of my system. I don't use things like terminations or the destructor system very much because most of that code doesn't have those kinds of side effects. > I think the two good things in SchemeUnit are: > - define-assertion I go back and forth on if I think there is a need for something like the above. I suspect yes, but already the API I made was so rich, that I'm not sure it was worth it. I'm still thinking about it. I need to determine on what level do I want this to happen: new control forms specific to an implementation, or composite expectations built by the test suite programmer? > - the optimised UI The UI is completely separate from the testing infrastructure I wrote. When it is all said and done, you get an in memory tree that you have a well-known API to manage the tree and each result type object. I abstracted the internal representation of the results and the tree for flexibility and backwards compatibility reasons. Thanks. -pete |
From: Noel W. <noe...@ya...> - 2003-01-06 14:08:22
|
> On Thu, Dec 12, 2002 at 08:02:57AM -0800, Noel Welsh > wrote: > > I'll take a look at your testing framework and get > > back to you. Work is very busy right now but I'll > > certainly get to it over Xmas. Well my printer died as I attempting to print it out so I only got a few very smudged sheets. Never buy an Epson inkjet; the ink is forever getting clogged. The upshot is that I haven't read the API yet. I'll read it soon. Honest. ;-) Noel __________________________________________________ Do you Yahoo!? Yahoo! Mail Plus - Powerful. Affordable. Sign up now. http://mailplus.yahoo.com |
From: Peter K. <ps...@cs...> - 2003-01-06 17:16:49
|
On Mon, Jan 06, 2003 at 06:08:15AM -0800, Noel Welsh wrote: > Well my printer died as I attempting to print it out > so I only got a few very smudged sheets. Never buy an > Epson inkjet; the ink is forever getting clogged. The > upshot is that I haven't read the API yet. I'll read > it soon. Honest. ;-) Ok. I had found a design problem with my API I'm not happy about. Basically, when you use the terminate function passing it a continuation from the scope you wish to terminate out of, any successful test results are lost and no destructor object calls occur from your current position up to the scope you escaped out of(because the continuation you passed to the terminate function really _was_ a true call/cc continuation). Effectively, it would be like invoking an exception in C++, but not calling any of the destructors in the frames you are escaping or passing over. I think I know how to fix this(meaning keeping the successful results and calling the destructors on the way out), but it needs some thought and some tinkering. I want to keep the semantic continuation, but not the logical one. :) Also, I've thought about your define-assertion. I think I will add one to my API as well, though with R5RS thrown in, it makes for some interesting problems since I cannot construct my own identifiers... -pete |
From: Noel W. <noe...@ya...> - 2003-01-13 13:28:59
|
--- Peter Keller <ps...@cs...> wrote: > Also, I've thought about your define-assertion. I > think I will add one to my > API as well, though with R5RS thrown in, it makes > for some interesting > problems since I cannot construct my own > identifiers... I'm sure if you sacrifice a copy of SICP to the demon on macros, Alternating Names Petrosky, there will be a way. I just do this, because location information is not available in R5RS and so the macro version is not useful: (define-syntax define-assertion (syntax-rules () ((define-assertion (name param ...) expr ...) (define name (lambda (param ...) (if (begin expr ...) #t (fail-assertion name param ...))))))) Noel __________________________________________________ Do you Yahoo!? Yahoo! Mail Plus - Powerful. Affordable. Sign up now. http://mailplus.yahoo.com |
From: Peter K. <ps...@cs...> - 2003-01-13 17:39:24
|
On Mon, Jan 13, 2003 at 05:28:53AM -0800, Noel Welsh wrote: > I just do this, because location information is not > available in R5RS and so the macro version is not > useful: > > (define-syntax define-assertion > (syntax-rules () > ((define-assertion (name param ...) expr ...) > (define name > (lambda (param ...) > (if (begin expr ...) > #t > (fail-assertion name > param ...))))))) Ah, see I can't do that this easily since the expectations are already macros. :) I need to generate a macro generating macro. And the reason why I would need constructed identifiers is because I evaluate the arguments to my expectation macros only once, but I might reference the result a few times. Since I don't want any unruly side effects, I bind the evaluations to internal variables which R5RS conveniently makes hygienic for me. However, I'm sure if I put more burden on the author of the new expectation, I wouldn't need to construct new identifiers. Here is an example of one of my expectations: ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; ;; expect-false: Expect a value to be #f ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; (define test:_expect-false (lambda (msg val) (equal? #f val))) (define-syntax expect-false (syntax-rules (warn) ;; with optional warning syntax ((_ msg (warn warning) val) (let ((warnobj warning) (message msg) (value val)) (let ((result (test:_expect-false message value))) (test:make-expect-result result "false" message 'val value warnobj)))) ;; without optional warning syntax ((_ msg val) (let ((message msg) (value val)) (let ((result (test:_expect-false message value))) (test:make-expect-result result "false" message 'val value)))))) Now, if I just wanted to design a new "single style expectation" (one that takes a single argument to evaluate), then I can most likely construct a new macro which parameterizes the important bits above. Sure it might be ugly, but it'll be doable. However, suppose I wanted to design a new expectation that took 5 parameters and returns a completely new result type? Now I'm in trouble as it becomes increasingly complex for the test suite author to construct a new type along with the expectation macro that goes with the new type--and don't forget the numerous set!/ref functions that abstract the type result. Since I had not chosen a data driven type model in my code(as I should have and one would see in SICP), a lot of nasty changes must happen in the base source code to accept the new type/result manipulation that the test suite author had created(if such a way to create them existed). However, I really, really wanted this API I had designed to exist. :) So I said "screw it" to the data driven type model because I couldn't design it that far forward without seeing concrete uses of the code to make sure that what I had made was useable and flexible(but, without the ability to create new expectations). Currently, I'm pretty satisfied with my code, and I realize that I will have to rewrite the internals to use a data driven type model since that is the ONLY way I can have a define-expectation kind of system be a legitimate and flexible thing. It was my intention to have the API exist, and being used for what it can be used for(which is a lot in my opinion) while I work on the data driven code. Then, when it is ready and if I was clever, nothing changes on the API side when the new code goes online. :) -pete |
From: Peter K. <ps...@cs...> - 2003-01-23 22:54:12
|
On Mon, Jan 06, 2003 at 11:13:31AM -0600, Peter Keller wrote: > On Mon, Jan 06, 2003 at 06:08:15AM -0800, Noel Welsh wrote: > > Well my printer died as I attempting to print it out > > so I only got a few very smudged sheets. Never buy an > > Epson inkjet; the ink is forever getting clogged. The > > upshot is that I haven't read the API yet. I'll read > > it soon. Honest. ;-) So, have you had a chance to look at it yet? I'm planning on adding the few new things I need and calling it done (until use of the system shows me more thing I must implement). I just need to find a free weekend.... grrr... -pete |
From: MJ R. <ma...@cl...> - 2002-12-12 23:06:04
|
Peter Keller <ps...@cs...> wrote: >> > because I noticed different scheme implementations either did or didn't >> > throw any sort of an exception when things like divide by zero happened >> > and I couldn't find a pure R5RS way of handling it. [...] >> Yes, you would probably need some SRFI support in the implementation for >> this one, I think. > Yeah, I figure as long as it is clear cut in the abstraction mechanisms I > chose to add this later, I can wait until it is standardized. Sorry for being unclear. I was meaning that you would need one of the exception-capable SRFIs to be supported by the implementation. I think SRFI-34 might be useful to you? -- MJR http://mjr.towers.org.uk/ IM: sl...@ja... This is my home web site. This for Jabber Messaging. How's my writing? Let me know via any of my contact details. |
From: Peter K. <ps...@cs...> - 2002-12-12 23:18:17
|
On Thu, Dec 12, 2002 at 11:05:59PM +0000, MJ Ray wrote: > Sorry for being unclear. I was meaning that you would need one of the > exception-capable SRFIs to be supported by the implementation. I think > SRFI-34 might be useful to you? Well, I just looked at the list of scheme implementations supporting SRFIs and 34 isn't in any of them. When it gets more adopted, I can add it. -pete |
From: MJ R. <ma...@cl...> - 2002-12-13 00:10:14
|
On 2002-12-12 17:16:34 -0600 Peter Keller <ps...@cs...> wrote: > Well, I just looked at the list of scheme implementations supporting > SRFIs and > 34 isn't in any of them. When it gets more adopted, I can add it. The list on srfi.schemers.org predates SRFI-34, from comparing the dates on them. Of course, I doubt anyone supports it yet, but this is an excuse to ask you to advocate supporting this SRFI. -- MJR http://mjr.towers.org.uk/ IM: sl...@ja... This is my home web site. This for Jabber Messaging. How's my writing? Let me know via any of my contact details. |
From: <sp...@in...> - 2002-12-15 10:18:48
|
>>>>> "MJ" == MJ Ray <ma...@cl...> writes: MJ> Peter Keller <ps...@cs...> wrote: >>> > because I noticed different scheme implementations either did or didn't >>> > throw any sort of an exception when things like divide by zero happened >>> > and I couldn't find a pure R5RS way of handling it. [...] >>> Yes, you would probably need some SRFI support in the implementation for >>> this one, I think. >> Yeah, I figure as long as it is clear cut in the abstraction mechanisms I >> chose to add this later, I can wait until it is standardized. MJ> Sorry for being unclear. I was meaning that you would need one of the MJ> exception-capable SRFIs to be supported by the implementation. I think MJ> SRFI-34 might be useful to you? I actually don't think so. (There's a micro thread at http://srfi.schemers.org/srfi-35/mail-archive/msg00002.html about the possibility of using SRFIs 34-36 for writing test suites.) Note that SRFI 34 is explicitly a SRFI for "Exception Handling for Writing Programs." It's not really intended for things like catching divide-by-zero which the program could prevent by itself (i.e., bugs in the program). A SRFI with an exception handling system for catching bugs would be a very worthwhile thing (for what you're trying to do, among other things), but it would have different properties from SRFI 34. This, btw., has been one of the central problems with standardizing "exception systems" in Scheme (and it actually mirrors the "module system problem"): people have been trying to do different things with exception systems but insisted on doing them all with the *same* exception systems. For a prime example, look at the article and the thread off http://zurich.ai.mit.edu/pipermail/rrrs-authors/1996-April/001837.html However, there is space for at least three separate ones with very different goals and, hence, optimal designs: - exception systems for communicating between programs - exception systems for catching bugs in programs - exception systems for handling asynchronous events I'll be glad to help with input on doing #2. (I cannot currently write it myself, because I've got 2 other SRFIs in the queue which are taking major work.) -- Cheers =8-} Mike Friede, Völkerverständigung und überhaupt blabla |