glean-dev — Glean development discussions

Re: [glean-dev] test locations and rigor

[Allen A. <ak...@po...>]

If you're convinced that image comparison is the best way to structure
a particular test, here's the way I'd do it.  Have the "run()" method
just generate the image in the results database and return a NOTE
rather than a PASS or FAIL.  Then check the image manually and verify
that it's correct.  Put the results database into a collection of
"golden" databases.  Finally, make the "compare()" method compare
images pixel-by-pixel.

To use this, the tester will have to run "glean -r" and then "glean
-c" (for one or more golden databases) to see if there are any
differences.  If differences crop up, he/she'll have to verify the new
results manually and add them to the golden databases if they're
correct.

Another approach would be to use small images and gradually add the
correct ones (as data declarations with initializers) to the source
code for the test.  That allows "glean -r" to return PASS or FAIL, but
the user still has to look at the image after a FAIL to determine
whether it's really wrong.

Personally, I've always found maintaining the golden databases to be a
lot more trouble in the long run than writing a test that's smart
about analyzing result images for correctness.

Currently glean is designed so that you don't need anything other than
the executable to run "glean -r".  I'd be reluctant to change that,
since it would mean dragging another database around every time you
wanted to run tests.  But if there were a truly compelling reason to
do otherwise, I'd consider it.

| An additional unrelated issue is that as we get more and more glean tests 
| (if I write 20 this week, say) it seems like it might be worth moving the 
| tests away from the glean base source in the directory tree.

Yes, good point.  I haven't done it so far because it involves
changing the Makefiles and source-code directory structure in a way
that (I suspect) would make them much less portable.  (I'm
particularly wary of depending on incremental linking.)

I'm open to suggestions on this one, so long as they preserve the
current nice feature of the Makefiles:  You don't have to edit the
Makefile for a directory when new source files are added to that
directory.

| Given those two concerns, I'm wondering if it might make sense to partition 
| tests into "canonical" and "non-canonical", and move them each to separate 
| subdirectories under src/glean.  ...

Beware the power of the dark side; if once you turn to it, forever
will it dominate your destiny.

If it's perceived as being a lot easier to write non-canonical tests,
then that's what people will do.  There'll be more duplication of
functionality (because non-canonical tests will tend to split along
vendor lines) and less confidence in the test results.

We tried separating tests into categories at SGI, and while it wasn't
what I'd call a failure, it did result in more work for less gain.

Allen

[glean-dev] More stuff

[Brian S. <br...@gl...>]

Yargh!  I don't know why this is taking me so long to get building.

It apparently doesn't build with the MSVC6 STL, regardless.  I think I was 
using the MSVC5 STL last time.  Maybe that works.  Or maybe something 
changed since then.

But I just downloaded STLPort 4.0 and it's bombing with #errors in the 
STLPort: "Only multi-threaded runtime library may be linked with STLPort!"

Is this a version issue?  Should I be using an earlier version of the STLPort?

-Brian
==========
GLSetup, Inc.

Re: [glean-dev] other comments

[Brian S. <br...@gl...>]

At 03:30 PM 10/25/00 -0400, you wrote:
>-- The build expects GLEAN_INC_DIR, GLEAN_LIB_DIR, and GLEAN_BIN_DIR to be 
>defined.

Oops, never mind this one.  I don't know why it wasn't working, but these 
are already in make/common.win to begin with, so they weren't the problem.

-B

[glean-dev] other comments

[Brian S. <br...@gl...>]

A few more comments.

On the glean.sourceforge.net "how do I build glean for win32?" page, it 
fails to mention a few things:

-- The build expects GLEAN_INC_DIR, GLEAN_LIB_DIR, and GLEAN_BIN_DIR to be 
defined.

-- You need to have glext.h installed

More as I find them...

-Brian
==========
GLSetup, Inc.

[glean-dev] Win32 glean oddities

[Brian S. <br...@gl...>]

I think I had these problems last time I built glean for win32, and looking 
through the glean-dev archives, I think I even mailed out about it.  I 
can't remember why I didn't fix it.

Anyway, am I the only one with a problem with the makefiles on Win32?  I'm 
using MSVC6.0 (or 6.3, can't remember) to build, and nmake doesn't like the 
".SILENT :" directive at the top of all the makefile.win files.  Complains 
about how it can't have any dependents (fatal error U1077, if that's of any 
use).

Furthermore, every makefile.win comes through with two 0x0d characters on 
the end of every line (Ctrl-Ms, emacs says).  I have to remove those for it 
to build -- nmake comes back with a "fatal error U1023: syntax error in 
expression" otherwise.

I think last time someone said they didn't see the same thing, but I just 
did a fresh grab from the repository.  Is there any translation setting for 
CVS that I ought to know about?  This is rather confusing.  I can get it to 
build if I remove all that cruft, but I don't know if I should check all 
the makefiles back in with the changes.

-Brian
==========
GLSetup, Inc.

[glean-dev] test locations and rigor

[Brian S. <br...@gl...>]

Heya,

One of my goals with glean is to convince IHVs' driver writers to write 
their own little internal tests as glean tests (I know at 3dfx I wrote 
little toy programs to test the driver all the time).  But it occurs to me 
that this might be hard the way glean is currently set up; I don't think 
I'd usually write something at 3dfx that would qualify as a rigorous glean 
test.  For example, I get the feeling that in writing glean tests this week 
I'm going to become awfully familiar with the nitpicky details of the 
OpenGL invariance rules, whereas I rarely considered them while working on 
a single driver, usually because the driver was more than conformant when 
working correctly.  So I could check for pixel-accuracy in compared images, 
for instance, where OpenGL might allow a pixel deviation or something.

An additional unrelated issue is that as we get more and more glean tests 
(if I write 20 this week, say) it seems like it might be worth moving the 
tests away from the glean base source in the directory tree.

Given those two concerns, I'm wondering if it might make sense to partition 
tests into "canonical" and "non-canonical", and move them each to separate 
subdirectories under src/glean.  So have src/glean/canonical and 
src/glean/noncanonical.  From there, encourage anybody who has to write a 
little test to do it as a subclass of Test (or BasicTest or whatever) and 
put it in noncanonical, and have people interested in advancing the 
canonical set of glean tests either write tests and put them in canonical 
to begin with, or start with a non-canonical test, clean it up, maybe make 
it more general if it's a really special-case little thing, and ensure that 
it never passes for nonconformant behavior and never fails for conformant 
behavior, and then move it over to src/glean/canonical.

Just wondering if people think this sounds like a reasonable idea.

-Brian
==========
GLSetup, Inc.

Re: [glean-dev] One more thing...

[Allen A. <ak...@po...>]

On Wed, Oct 25, 2000 at 03:33:14AM -0400, Brian Sharp wrote:
| 
| Better yet, is there a super-easy way to convert SGML into HTML (on Win32 
| systems)?

davecentral doesn't list anything, surprisingly.

A Google search turned up this tutorial:

	http://ourworld.compuserve.com/homepages/hoenicka_markus/ntsgml.html

For the meantime I've attached glean.html.

Allen

Re: [glean-dev] libtiff and stlport

[Allen A. <ak...@po...>]

On Wed, Oct 25, 2000 at 03:22:38AM -0400, Brian Sharp wrote:
| 
| And also, is the STLPort requirement a new one?  I don't recall having to 
| get STLPort last time (albeit a number of months ago).

It might not be necessary anymore.  When Johan did the original
Windows port Microsoft's implementation of STL wasn't sufficient, but
that may have changed by now.

| Furthermore, does glean on Win32 build only with nmake?  Is there any 
| chance that it'll (i.e. has anyone had any success getting it to) build 
| with the Cygwin make?

I haven't tried it (don't have a Win32 development machine these days).
My guess is that it would require some tweaking in common.mak to set
"make" variables appropriately, but otherwise should work.  (Assuming
you're not using a filesystem with an 8.3 filename limitation.)

Allen

Re: [glean-dev] One more thing...

[Brian P. <br...@va...>]

Brian Sharp wrote:
> 
> It seems that the glean.sgml document doesn't have a mirrored HTML document
> checked into the repository.
> 
> What does one use to read SGML docs these days?  (And how ironic is it that
> I don't know of a program to view a document with a format the first word
> of whose name is "Standard"?)
> 
> Better yet, is there a super-easy way to convert SGML into HTML (on Win32
> systems)?

On Linux, I use sgml2html to make HTML.  I don't know about Win32, sorry.

-Brian

Re: [glean-dev] libtiff and stlport

[Neal T. <ne...@ps...>]

From: Brian Sharp <br...@gl...>
> Hey, all, some questions:
>
> First of all, where can I get prebuilt win32 libtiff .libs?  I remember
> finding them on some obscure site last time I was working with glean, and
I
> really don't want to assume the task of getting yet another package to
> build, so I'd like to find them (prebuilt) again.  Any pointers would be
> much appreciated.

You can get a prebuilt Win32 libtiff from the downloads area of
http://www.pseudonymz.demon.co.uk

Neal Tringham (Sick Puppies / Empire Interactive)

ne...@ps...
ne...@em...

[glean-dev] One more thing...

[Brian S. <br...@gl...>]

It seems that the glean.sgml document doesn't have a mirrored HTML document 
checked into the repository.

What does one use to read SGML docs these days?  (And how ironic is it that 
I don't know of a program to view a document with a format the first word 
of whose name is "Standard"?)

Better yet, is there a super-easy way to convert SGML into HTML (on Win32 
systems)?

Thanks.
-Brian
==========
GLSetup, Inc.

[glean-dev] libtiff and stlport

[Brian S. <br...@gl...>]

Hey, all, some questions:

First of all, where can I get prebuilt win32 libtiff .libs?  I remember 
finding them on some obscure site last time I was working with glean, and I 
really don't want to assume the task of getting yet another package to 
build, so I'd like to find them (prebuilt) again.  Any pointers would be 
much appreciated.

And also, is the STLPort requirement a new one?  I don't recall having to 
get STLPort last time (albeit a number of months ago).

Furthermore, does glean on Win32 build only with nmake?  Is there any 
chance that it'll (i.e. has anyone had any success getting it to) build 
with the Cygwin make?

Thanks.
-Brian
==========
GLSetup, Inc.

Re: [glean-dev] work this week

[Allen A. <ak...@po...>]

On Mon, Oct 23, 2000 at 06:27:52PM -0400, Brian Sharp wrote:
| 
| Just to let you know, this week I'm planning on doing a first-pass 
| implementation of all the tests in the suggested test list doc in the 
| repository.  ...

Excellent!  But be aware that even a first-pass on *all* those tests
will be a lot of work.

| A question: should I remove or otherwise notate the tests in the test list 
| HTML doc as I write them?

I think a notation would be good -- that way everyone can see at a
glance whether a test is open, underway, complete, or whatever.

Allen

[glean-dev] work this week

[Brian S. <br...@gl...>]

Hey, all,

Just to let you know, this week I'm planning on doing a first-pass 
implementation of all the tests in the suggested test list doc in the 
repository.  I'm sure there'll be issues with them, but I figure it'll be 
good to fill out the test base and then work on fine-tuning each test from 
there.

A question: should I remove or otherwise notate the tests in the test list 
HTML doc as I write them?

-Brian
==========
GLSetup, Inc.

Re: [glean-dev] bug in command line parsing?

[Allen A. <ak...@po...>]

On Thu, Oct 19, 2000 at 05:42:31PM -0600, Brian Paul wrote:
| 
| Allen, can you give an example of the syntax used to indicate
| that two or more specific tests are to be run by clean?
| 
| I've tried several variations such as:
| 
| glean -r results -t blendFunc -t getString
| 
| glean -r results -t + blendFunc + getString
| 
| etc, but haven't been successful.


If the test names are all in one argument the second form should
work:

	glean -r results -t +blendFunc+getString

or equivalently

	glean -r results -t blendFunc+getString

The first form (multiple -t options) doesn't work because each -t
option replaces the results of the ones before it.

Sorry about the lack of clarity in the usage message...

Allen

[glean-dev] bug in command line parsing?

[Brian P. <br...@va...>]

Allen, can you give an example of the syntax used to indicate
that two or more specific tests are to be run by clean?

I've tried several variations such as:

glean -r results -t blendFunc -t getString

glean -r results -t + blendFunc + getString

etc, but haven't been successful.

-Brian

Re: [glean-dev] question about blendFunc error tolerance

[Allen A. <ak...@po...>]

On Tue, Oct 17, 2000 at 02:15:20PM -0500, Jeffrey Mehlhorn wrote:
| 
|    I am curious to know how the error tolerance is determined
|    for the Glean blendFunc test.  ...

The test assumes that apps expect blending arithmetic to be about as
accurate as is implied by the depths of the color channels. 
(Otherwise, why not have cheaper, faster, shallower color buffers?)
The sort of apps that depend on this include volume renderers,
multipass shaders, and special-effects compositors for film/video.

So the tolerance is one least-significant bit, as determined by the
depths of the color channels.  (Each channel is tested independently.)
This is twice the theoretical minimum tolerance (0.5 LSB, which you
have to allow in order to accommodate rounding), but it's about as
good as the "cheap" hardware implementations I know of can manage.

As an example, if a color channel is 8 bits deep, then its LSB has a
magnitude of (roughly) 1/256, or 0.00390625.  If a blend computation
is shown to have an error of 0.01, that would be expressed as
0.01/0.00390625, or about 2.6, LSB.  Such an implementation wouldn't
be providing as much accuracy as its color channel depths imply, so it
fails the test.

If a color channel was 6 bits deep, however, the same 0.01 error would
be only 0.01/0.015625, or 0.6, LSB.  Such an implementation would be
doing pretty well, in fact very close to the best possible.  It
passes.

There are some subtleties in how implementations might represent
source alpha, but the test assumes it's never represented with lower
precision than the color channels in the framebuffer, which is safe as
far as I know.

|                              ...  I have sifted through the
|    glean documentation and source code, but have been unable
|    to glean (pardon the pun) an understanding of how there
|    error between actual and expected pixel color values
|    is determined.

No need to apologize for puns; such things are to be encouraged. :-)

As far as the error computation goes, if "delta" is the absolute value
of the difference between actual and expected color values each
expressed in the range [0,1] (according to the usual OpenGL
conventions), and "channelDepth" is the number of bits in the relevant
color channel, then the computed error (in bits) is

	log2(delta) + channelDepth

The function that computes this is ErrorBits() in misc.cpp.  If you're
curious about how the formula is derived, let me know and I'll
summarize it.  I suspect the easiest way to understand it is to start
with a version using the integer representation of color values and
then take the log-base-two.

Allen

[glean-dev] question about blendFunc error tolerance

[Jeffrey M. <Jef...@In...>]

   I am curious to know how the error tolerance is determined
   for the Glean blendFunc test.  I have sifted through the
   glean documentation and source code, but have been unable
   to glean (pardon the pun) an understanding of how there
   error between actual and expected pixel color values
   is determined.

   Jeff

Re: [glean-dev] FYI - GLUT-related compilation problem in RedHat 7.0

[Allen A. <ak...@po...>]

It appears RedHat already knows about this one -- Bugzilla bug number
18952.  bero says if you need an "official" fix in a hurry, you can
download it from http://www.linux-easy.com/rh-updates/ .

Allen

[glean-dev] FYI - GLUT-related compilation problem in RedHat 7.0

[Allen A. <ak...@po...>]

It appears that /usr/include/GL/glut.h declares the stdlib function
exit(), rather than leaving that job up to stdlib.h, and unfortunately
the declaration in glut.h is incompatible with the declaration in
stdlib.h.  This prevents the GLUT-based tools (showtiff and difftiff)
from compiling.  Since it's not clear Mark Kilgard is supporting GLUT
any longer, I'll report the problem to RedHat.  In the meantime, a
reasonable workaround is to comment out line 202 (which declares
exit() for non-Win32 systems) in glut.h.

Allen

Re: [glean-dev] Build problems on debian potato?

[Rik F. <fa...@va...>]

On Thu  5 Oct 2000 13:54:12 -0400,
   Bill White <bil...@gr...> wrote:
> Does anybody build glean on Debian potato?  I have been getting an illegal
> instruction error during initialization of the tests.  It doesn't seem to
> matter which tests I enable, I get the same error in the same place for
> them all.

Is it easy to provide a stack backtrace?  In particular, are you in
code that was generated for glean, or in a library call?

> My configuration is:
> o The tip of the CVS tree, as obtained from anoncvs.
> o Debian potato, with libc6 version 2.1.3.
> o libstdc++2.10 version 2.95.2
> o g++ version 2.95.2.  I have tried egcs 1.1.2 as well, with no
>   success.

Did you try with and without -O0, -O, and -O2?  The gcc -O0 code
generator is significantly more straight-forward and stupid than the
other levels of optimization, so it's often a good base case for
determining code generation problems.  The compiler should never
generate an illegal instruction.

Re: [glean-dev] Ignore my previous plaintive bleatings.

[Bill W. <bil...@gr...>]

> I'll apply the patch.  For future convenience, though, I'll just add
> you to the list of developers so you can make changes directly via
> CVS.  (I'd like to keep access relatively open unless/until we run
> into problems.)
Tnx.  Please be sure to get the second patch.  I messed the first
one up.  You can tell if it's the second version of the patch if the
line for "clean_dir:" has backslashes at the end of all the appropriate
lines.  I left a backslash out of the "fi" line in the first one.

Re: [glean-dev] Ignore my previous plaintive bleatings.

[Allen A. <ak...@po...>]

On Thu, Oct 05, 2000 at 02:42:44PM -0400, Bill White wrote:
| The build problem I had was with the optimization strings.  My computer
| is an AMD K6-2, and the makefile is set up to compile with -march=pentiumpro.
| 
| Things seem to be working better now.

Excellent!  I'm glad it wasn't something intractable.

BTW, just checked SourceForge and saw your patch has reached the
database.  I still haven't seen any notification mail for it, but mail
from SourceForge has been slow in the past few days.

I'll apply the patch.  For future convenience, though, I'll just add
you to the list of developers so you can make changes directly via
CVS.  (I'd like to keep access relatively open unless/until we run
into problems.)

Allen

[glean-dev] Ignore my previous plaintive bleatings.

[Bill W. <bil...@gr...>]

Here is a foolproof method for finding a the solution to a problem.
Let everyone know you are stumped, and the answer will appear as if
by magic.

The build problem I had was with the optimization strings.  My computer
is an AMD K6-2, and the makefile is set up to compile with -march=pentiumpro.

Things seem to be working better now.

[glean-dev] Build problems on debian potato?

[Bill W. <bil...@gr...>]

Does anybody build glean on Debian potato?  I have been getting an illegal
instruction error during initialization of the tests.  It doesn't seem to
matter which tests I enable, I get the same error in the same place for
them all.

My configuration is:
o The tip of the CVS tree, as obtained from anoncvs.
o Debian potato, with libc6 version 2.1.3.
o libstdc++2.10 version 2.95.2
o g++ version 2.95.2.  I have tried egcs 1.1.2 as well, with no success.

The error seems to occur initializing an STL-type string from a char *. 

I have this odd feeling that it is a compiler configuration problem, but
I thought I would ask.  Usually I find the solution to really obvious 
problems immediately after I let people know I have overlooked the obvious.
So, I might as well get it started and take my punishment earlier rather
than later.

N.B. I had a compilation error caused by an empty definition of $(DIRS),
but I fixed that, and have submitted a patch.