Re: [FreeWRL-develop] Testing update

[doug s. <hig...@ho...>]

Paulo,
>
> I can confirm that the fixtures generated seem different per machine....
> Even when testing in two similar architectures "Linux" the fixtures
> seemed to differ.
> I think this might be down to a combination of library versions, graphic
> cards etc ...

I have bit differences in 20% of comparisons on a single win32 platform,
 but they are scattered across the image -not header related- and larger
 color spreads on textured objects.
That's why I use the graphicsMagick mean square error measure.

> Further, I commented the code that was writing the header file of the
> bmp file and that seemed to generate more stable results across runs.
> At least I don't have any bit differences between runs.

The header issue is puzzling to me. I should have zeroed the header, and put only useful stable numbers in it. There are a few gotchas that I didn't look into: struct alignment -if a compiler pads a struct to an even 8 bytes then I write out the struct as a binary blob I'll get padding, which is bad. And intel vs motorola byte ordering - I'm not sure how to fix that. 

64 BIT VS 32 BIT, INTEL VS MOTOROLA

The padding can be fixed by writing each element of the struct out separately. And do I have the right data types to avoid 64bit ints? One way to tell is to try the graphicsmagick or gimp readers on each platform and see if they can read my header generated on various platforms, assuming gimp and graphicsmagick have the right idea.
(Right now I have only 32bit hardware and compilers, and opengl 2 only on one machine)

>
> I used a different approach though ...
>
> First I commented that time reduction hack so that I could record things
> smoothly.
> https://github.com/pecastro/freewrl/commit/4926a80062179dd1223296537eb45bf04b9a17eb
>
> Using recording mode, I manually created one master .fwplay whilst
> moving in the scenegraph back and forth rotating etc and in between each
> move taking some manual snapshots.
> This master file was created using the freewrl/tests/2.wrl
> https://github.com/pecastro/freewrl/commit/044850835cc7d56e4f6c2b52360485134620e4d3
>
> Then I used this .fwplay file to test all of the .wrl/x3d files.
> I iterate over the list of .wrl files to be tested and for each one I
> amend the .fwplay file, and substitute the scenefile for the current
> file being tested.
> I then run freewrl in playback mode for that specific .wrl file.
> I kept all the headerless .bmp files generated and committed them to a
> local branch.
>
> Then I built freewrl from scratch and ran exactly the same procedure of
> running the script in playback mode and in the end it was a matter of
> asking my source control system if any of those .bmp files had changed.

Great idea. 
So summary: this method is good for checking geometry rendering changes in detail, but doesn't allow for scene-specific mouse or keyboard input -except see below^- such as clicking something.
What it's good for:
a) inspecting scene rendering in finer detail
b) using only one fwplay, with high-fidelity avatar movements, to view each scene using the same avatar movements for each scene

^It might also be very helpful when you have one giant test scene with everything in it -or everything that's of interest for your development changes- and you use just that one scene for testing. In that case you can do clicking and keyboarding for that scene. For example we have 52 tests, and you could put them all side-by-side in a single scene file, then navigate the avatar between them during a test run.

Q. if you have 100 test scenes, how long does it take to run the test on all of them?
(the degraded-frame-rate, one-fwplay-per-small-scene takes 3 minutes to playback and compare 104 scenes on a 32 bit pentium)

>
> I never really used the Playback option.

-P --playback is just -F --fixture, with a different output folder name. Comparing current fixture to source-control fixture is equivalent to comparing playback to fixture.
If someone improves the code -and verifies manually that the code is correct- they should re-generate the fixtures or in your case check-in the current fixtures to source control.

>
> The code hacky has it is can be seen at
> https://github.com/pecastro/freewrl/tree/testing-freewrl

Thanks for the link. And the ideas. I wonder if the two methods can be combined somehow, perhaps as options / parameters, so developers can conveniently choose.

-Doug