[glean-dev] Test construction [Was: Basic tests on ARB_Occlusion_Query]

[Allen A. <ak...@ar...>]

On Tue, Jul 10, 2007 at 11:27:48AM +0800, Wang, Wei Z wrote:
| The attached files are for some basic conformance tests on extension 
| "ARB_Occlusion_Query".

Thanks for submitting the tests!

I see Brian's already checked them in.  (Thanks, Brian.)

I have a few thoughts about how an ideal test should be constructed.
I'll mention them here so everyone has a chance to consider them.  I
won't have reliable net access for a couple of weeks, so I can't
guarantee that I can participate in a discussion right away,
unfortunately.

One good goal is to make sure each test has a meaningful compareOne()
method.  As the number of tests grows larger, and as individual tests
get more complex, it gets harder to see how results change from run to
run.  For any OpenGL test suite that's even close to complete, a test
log is enormous; it isn't practical to look for meaningful differences
in warnings that are written to stderr, or determine whether numerical
changes are statistically significant.  Good compareOne() methods make
it possible to automate checking for changes, and allow the author of
the test to determine whether a change is significant (something that
might be hard for another glean user to determine if he doesn't know the
test well).

A consequence of wanting meaningful compareOne() methods is that a glean
test should record *everything* that's needed to interpret the result in
its Result structure.  Not just pass/fail, but numerical results,
diagnostic information, consistency problems (like GL errors that
shouldn't occur), etc.

One thing I didn't do originally, but I wish I had, was to generate log
messages only from Result structures, not from program logic during the
run of a test.  That would make it possible to regenerate a log from a
results database.  Then you wouldn't need to keep the log *and* the
database to compare two runs; you'd only need the database.  Keeping
this idea in mind is also a good way to decide whether you've put
everything into the Result structure that a given test really needs.

That's it for the moment.  I hope it's good food for thought!

Allen