Re: [glean-dev] Downloads broken?

[Allen A. <ak...@po...>]

On Tue, Dec 20, 2005 at 10:28:51AM +0200, Kalle Raita wrote:
| I now succeeded in using the CVS. Maybe some transient problem or bad 
| copy-pasting by me. Anyways, thanks for the help.

I'm glad it finally worked!

| I'm actually writing my master's thesis about using image comparison in 
| graphics API testing. ...

You've chosen a challenging topic, but an important one.  Verification
often requires more effort than chip design.

I managed groups at SGI that worked on the OpenGL conformance tests and
on SGI's internal test suite (ogtst).  I guess I could sum up my
experience with image comparison this way:

	Image comparison works particularly well for regression testing
	on a single graphics device (or a family of devices with the
	same circuit design).

	Creating and maintaining the image database is a lot of work.

	Image comparison is less successful when you have to test a new
	device or compare two unrelated devices.  Even subtle
	implementation differences can cause image differences that may
	be hard to validate.

	Image analysis can produce more meaningful results (especially
	on unrelated devices) than simple comparison, if the graphics
	semantics are specified with sufficient care.

	Analysis code is *much* more difficult to write than comparison
	code.

Glean and the conformance tests are intended to run on many unrelated
devices, so both of those suites rely more on analysis than comparison.
ogtst relied more heavily on comparison (and I would guess the same is
true of most vendors' test suites).

Please let us know when your thesis is done.  I'd like to read your
conclusions!

By the way, if you find books or papers on testing that you can
recommend, I'd like to hear about those, too.  One that I used as an
introduction for the ogtst project group at SGI was "The Art of Software
Testing" by Glenford Myers.

|                   ... I thought that Glean might be good reference 
| material on how conformance testing can be done. ...

It's a start, anyway.  Glean doesn't attempt to test the API
comprehensively, because that's far too large a project to attempt with
volunteers.  But it illustrates the basic techniques.

|                                              ... Referring to actual API 
| conformance tests might be a bit problematic as they are not publicly 
| available and discussing the techniques used might be a violation of the 
| licensing terms.

I would hope not, but it's hard to say.  You'd need to talk to the legal
folks at SGI.

| By the way, have you thought about porting your work to OpenGL ES?

A little.  I haven't put much effort into glean lately because the
hardware vendors have their own test suites that are much more complete.
I'd expect the same to be true for OpenGL ES.  But Brian Paul tells me
that glean is still useful, so maybe some more effort would be
worthwhile.

Allen