[Hostess-info] Re: test suite finals

[<re...@in...>]

Dear Fyodor,

> Bugs found by the test suite the day before were killed. So now my
> compiler passes all tests for O1 that do not contain FOR.

Good. I'm glad the test suite was useful for you.


> The general impression of test suite interface. I think it was not a
> good idea to output nothing in the case of success. I even don't know
> if the test reached its end or not. Furthermore even if I sure that
> procedure Test was completed successfully I cannot be sure that any of
> other procedure was called.

I know. The problem are the libraries. I guess we could assume that
something like Out.String must exists. I remember we didn't agree
on Out.Open....

One consideration I had, is that the suite should also be useful when
building a new compiler, by testing the single features one at the
time. And importing is a very complex one which usually get implemented
quite late.


> I think it would be better to output something on start and end of each
> procedure. Also it may be better to output intermediative results
> explicitly than to use ASSERT. For example
> ASSERT(i=5)
> change into
> Out.Int(i,16)
> and then compare output generated by the program with the standard one.

The test-suite is defined in a way that it is possible to specify a
file containing the expected results. We just didn't use this feature
very often.


> Do you use any regular technics for creating tests?

I can speak only for myself here:
1. when building the compiler I tried to test the single language
features and the various code patterns (e.g. adding two constants
using local variables, parameters, globals, constants), but there
are still a features that are not tested yet.

2. when debugging I first write a test case for the erroneous
feature to be sure the problem gets corrected.

I've read about some techniques (G. Myers' book on testing is 
really enlightning), but the test suite is more a set
of tests developed ad-hoc while building the compiler.


> I have invented a way of creation of test suite on regular basis. Two
> men (or groups) _independently_ (that is the most important word)
> develop compilers of the language. Then each create test set that
> executes every statement and expression in their compiler and yields
> every possible error message of the compiler. After that they exchange
> their tests and discuss differences. What do you think about this
> method?

This is similar to what we are trying to do with the test suite:
we would like the oberon compilers developers to contribute
the tests they wrote for their compilers to the suite, so that we
get a good test diversity and coverage.

From the experience with hostess (this suite): your strategy
will probably point out all the spots in the language that are
not clearly defined and where the groups made different assumptions.


-Patrik
--
Patrik Reali, re...@in...
http://www.inf.ethz.ch/personal/reali/
http://www.oberon.ethz.ch/jaos/
http://www.oberon.ethz.ch/native/