Re: [FreeWRL-develop] Testing > file compare

[doug s. <hig...@ho...>]

>>
>>>> 3. Compare - perl script that compares Record vs Playback snapshots for a given test file
>>>> -- there might be several snapshots for a given test file run, named by the frame#
>>>> -- calls the platform file compare function
>>>> --- in ms windows its fc file1 file2
>>> .
>>> The perl I have on my windows XP has compare(f1,f2)
>>> So this works with text and binary files:
>>> #!/usr/bin/env perl
>>> use File::Compare;
>>> if (compare("file1.rgb","file2.rgb") == 0) {
>>> print "They're equal\n";
>>> }else{
>>> print "They're different\n";
>>> }
>>> -Doug
>>> more...
>>> -IIRC I downloaded the Activestate perl from http://www.perl.org/get.html
>> .
>> 2 screen snapshots of the same scene 1.wrl that _should_ be equal are different, and fc shows the differences spread out. (I hid the statusbar with its frames-per-second readout, by mousing over it, which brings up the menu bar). Hypotheses for the diff:
>> H0: something about my video board - a few bad pixels and I have the window in a different place
>> H1: floating point computations round differently depending how many frames have been rendered
>> H2: something in the .bmp header
>> If it's a small difference like 1/256 (1 bit out of a pixel's color value) perhaps a special compare function will be needed.
> .
> I downloaded GraphicsMagick from http://www.graphicsmagick.org (a forked & advanced project of ImageMagick) and ran the image compare function http://www.graphicsmagick.org/compare.html as follows on my 2 screen shots:
> .
> gm compare -metric mse freewrl_snapshot_A.bmp freewrl_snapshot_C.bmp
> pause
> gm compare freewrl_snapshot_A.bmp freewrl_snapshot_C.bmp -file diff.bmp -highlight-color purple -highlight-style Assign
> gm convert diff.bmp win:
> pause
> .
> The first compare shows a statistical summary:
> Image Difference (MeanSquaredError):
>            Normalized    Absolute
>           ============  ==========
>      Red: 0.0000000000        0.0
>    Green: 0.0000000000        0.0
>     Blue: 0.0000010175        0.0
>    Total: 0.0000003392        0.0
> .
> The second compare generates an image with any different pixels colored purple. Here's the diff image converted to png and compressed - you'll see several purple pixels around the ! and checkmark icons on the menu bar:
> http://dug9.users.sourceforge.net/web3d/diff.png
> .
> I had played with those two menu options during one of the runs. Hypothesis: it renders a bit different between original and post-activation on the menu bar.
> .
> Summary: looks like the main graphics window renders the same -no random floating point rounding differences in opengl- and differences are due to different user activity during one run, as seen by GraphicsMagic compare on uncompressed snapshots.
> .
> Conclusion: we should be able to do software testing based on snapshot differences.
.
Paulo,
I retract that conclusion, based on further testing:
I put in -record and -playback commandline options (not checked in) that record the frame# (starting with 1 after scene loaded), dtime, keystrokes, mouse to a text playback file:
http://dug9.users.sourceforge.net/web3d/testing/playback.play 
keystroke 'x' (command for saving a snapshot image) done at the same frame and dtime, and after the same mouse actions, was supposed to produce the same snapshot. Comparing with GraphicsMagic, I got this (compresssed png):
http://dug9.users.sourceforge.net/web3d/testing/diff_record_playback.png
showing the cone geometry speckled with dozens of different pixels.
.
Hypotheses:
H0: a small numerical difference of 1/256 due to floating point rounding in opengl or video board
Test: do a pixel compare channel by channel, to see the maximum difference
http://dug9.users.sourceforge.net/web3d/testing/freewrl_snapshot_record.bmp
http://dug9.users.sourceforge.net/web3d/testing/freewrl_snapshot_playback.bmp
If it's a 1/256 rounding error, then a special compare function could be used/developed that would forgive that small difference
H1: I didn't apply the mouse actions at the right point in the playback frame
H2: the floating point rounding is on the mouse coordinates (when converted to screen) and I don't have enough decimal places in the fprintf 
H3: ???
.
I wonder if I should check it in, or forget it as hopeless, or do more tests.
-Doug