Re: [FreeWRL-develop] Testing > file compare

[doug s. <hig...@ho...>]

>>>>
>>>>>> 3. Compare - perl script that compares Record vs Playback snapshots for a given test file
>>>>>> -- there might be several snapshots for a given test file run, named by the frame#
>>>>>> -- calls the platform file compare function
>>>>>> --- in ms windows its fc file1 file2
>>>>> .
>>>>> The perl I have on my windows XP has compare(f1,f2)
>>>>> So this works with text and binary files:
>>>>> #!/usr/bin/env perl
>>>>> use File::Compare;
>>>>> if (compare("file1.rgb","file2.rgb") == 0) {
>>>>> print "They're equal\n";
>>>>> }else{
>>>>> print "They're different\n";
>>>>> }
>>>>> -Doug
>>>>> more...
>>>>> -IIRC I downloaded the Activestate perl from http://www.perl.org/get.html
>>>> .
>>>> 2 screen snapshots of the same scene 1.wrl that _should_ be equal are different, and fc shows the differences spread out. (I hid the statusbar with its frames-per-second readout, by mousing over it, which brings up the menu bar). Hypotheses for the diff:
>>>> H0: something about my video board - a few bad pixels and I have the window in a different place
>>>> H1: floating point computations round differently depending how many frames have been rendered
>>>> H2: something in the .bmp header
>>>> If it's a small difference like 1/256 (1 bit out of a pixel's color value) perhaps a special compare function will be needed.
>>> .
>>> I downloaded GraphicsMagick from http://www.graphicsmagick.org (a forked & advanced project of ImageMagick) and ran the image compare function http://www.graphicsmagick.org/compare.html as follows on my 2 screen shots:
>>> .
>>> gm compare -metric mse freewrl_snapshot_A.bmp freewrl_snapshot_C.bmp
>>> pause
>>> gm compare freewrl_snapshot_A.bmp freewrl_snapshot_C.bmp -file diff.bmp -highlight-color purple -highlight-style Assign
>>> gm convert diff.bmp win:
>>> pause
>>> .
>>> The first compare shows a statistical summary:
>>> Image Difference (MeanSquaredError):
>>> Normalized Absolute
>>> ============ ==========
>>> Red: 0.0000000000 0.0
>>> Green: 0.0000000000 0.0
>>> Blue: 0.0000010175 0.0
>>> Total: 0.0000003392 0.0
>>> .
>>> The second compare generates an image with any different pixels colored purple. Here's the diff image converted to png and compressed - you'll see several purple pixels around the ! and checkmark icons on the menu bar:
>>> http://dug9.users.sourceforge.net/web3d/diff.png
>>> .
>>> I had played with those two menu options during one of the runs. Hypothesis: it renders a bit different between original and post-activation on the menu bar.
>>> .
>>> Summary: looks like the main graphics window renders the same -no random floating point rounding differences in opengl- and differences are due to different user activity during one run, as seen by GraphicsMagic compare on uncompressed snapshots.
>>> .
>>> Conclusion: we should be able to do software testing based on snapshot differences.
>> .
>> Paulo,
>> I retract that conclusion, based on further testing:
>> I put in -record and -playback commandline options (not checked in) that record the frame# (starting with 1 after scene loaded), dtime, keystrokes, mouse to a text playback file:
>> http://dug9.users.sourceforge.net/web3d/testing/playback.play
>> keystroke 'x' (command for saving a snapshot image) done at the same frame and dtime, and after the same mouse actions, was supposed to produce the same snapshot. Comparing with GraphicsMagic, I got this (compresssed png):
>> http://dug9.users.sourceforge.net/web3d/testing/diff_record_playback.png
>> showing the cone geometry speckled with dozens of different pixels.
>> .
>> Hypotheses:
>> H0: a small numerical difference of 1/256 due to floating point rounding in opengl or video board
>> Test: do a pixel compare channel by channel, to see the maximum difference
>> http://dug9.users.sourceforge.net/web3d/testing/freewrl_snapshot_record.bmp
>> http://dug9.users.sourceforge.net/web3d/testing/freewrl_snapshot_playback.bmp
>> If it's a 1/256 rounding error, then a special compare function could be used/developed that would forgive that small difference
>> H1: I didn't apply the mouse actions at the right point in the playback frame
>> H2: the floating point rounding is on the mouse coordinates (when converted to screen) and I don't have enough decimal places in the fprintf
>> H3: ???
>> .
>> I wonder if I should check it in, or forget it as hopeless, or do more tests.
> .
> If I run the -playback a second time, and compare the 2 playback images, there are zero differences. If I had implemented -record mode as a recording of mouse and keyboard, followed by a -playback to get the snapshot images, then I would see no differences between record and playback snapshots.
> .
> Summary: the way I implemented record and playback are not the same - a slightly different process or rounding, gives slightly different images. (for example the mouse coordinates are normalized to -1.0 to 1.0 floating point range, and in the record mode I use them directly, in playback they go through fprintf and sscanf which round them)
> .
> Conclusion: comparing record and playback images will work for functional testing, once implemented properly.
> 
.
Confirmed. Using the same rounded mouse x,y during record as playback, there are no differences now between the record snapshot and the playback snapshot. 
.
I'll do a little more testing, and work on naming of the playback and (uncompressed) snapshot files before checking it in.
-Doug