Re: [FreeWRL-develop] Testing > file compare

[doug s. <hig...@ho...>]

>>>
>>>>> 3. Compare - perl script that compares Record vs Playback snapshots for a given test file
>>>>> -- there might be several snapshots for a given test file run, named by the frame#
>>>>> -- calls the platform file compare function
>>>>> --- in ms windows its fc file1 file2
>>>> .
>>>> The perl I have on my windows XP has compare(f1,f2)
>>>> So this works with text and binary files:
>>>> #!/usr/bin/env perl
>>>> use File::Compare;
>>>> if (compare("file1.rgb","file2.rgb") == 0) {
>>>> print "They're equal\n";
>>>> }else{
>>>> print "They're different\n";
>>>> }
>>>> -Doug
>>>> more...
>>>> -IIRC I downloaded the Activestate perl from http://www.perl.org/get.html
>>> .
>>> 2 screen snapshots of the same scene 1.wrl that _should_ be equal are different, and fc shows the differences spread out. (I hid the statusbar with its frames-per-second readout, by mousing over it, which brings up the menu bar). Hypotheses for the diff:
>>> H0: something about my video board - a few bad pixels and I have the window in a different place
>>> H1: floating point computations round differently depending how many frames have been rendered
>>> H2: something in the .bmp header
>>> If it's a small difference like 1/256 (1 bit out of a pixel's color value) perhaps a special compare function will be needed.
>> .
>> I downloaded GraphicsMagick from http://www.graphicsmagick.org (a forked & advanced project of ImageMagick) and ran the image compare function http://www.graphicsmagick.org/compare.html as follows on my 2 screen shots:
>> .
>> gm compare -metric mse freewrl_snapshot_A.bmp freewrl_snapshot_C.bmp
>> pause
>> gm compare freewrl_snapshot_A.bmp freewrl_snapshot_C.bmp -file diff.bmp -highlight-color purple -highlight-style Assign
>> gm convert diff.bmp win:
>> pause
>> .
>> The first compare shows a statistical summary:
>> Image Difference (MeanSquaredError):
>> Normalized Absolute
>> ============ ==========
>> Red: 0.0000000000 0.0
>> Green: 0.0000000000 0.0
>> Blue: 0.0000010175 0.0
>> Total: 0.0000003392 0.0
>> .
>> The second compare generates an image with any different pixels colored purple. Here's the diff image converted to png and compressed - you'll see several purple pixels around the ! and checkmark icons on the menu bar:
>> http://dug9.users.sourceforge.net/web3d/diff.png
>> .
>> I had played with those two menu options during one of the runs. Hypothesis: it renders a bit different between original and post-activation on the menu bar.
>> .
>> Summary: looks like the main graphics window renders the same -no random floating point rounding differences in opengl- and differences are due to different user activity during one run, as seen by GraphicsMagic compare on uncompressed snapshots.
>> .
>> Conclusion: we should be able to do software testing based on snapshot differences.
> .
> Paulo,
> I retract that conclusion, based on further testing:
> I put in -record and -playback commandline options (not checked in) that record the frame# (starting with 1 after scene loaded), dtime, keystrokes, mouse to a text playback file:
> http://dug9.users.sourceforge.net/web3d/testing/playback.play
> keystroke 'x' (command for saving a snapshot image) done at the same frame and dtime, and after the same mouse actions, was supposed to produce the same snapshot. Comparing with GraphicsMagic, I got this (compresssed png):
> http://dug9.users.sourceforge.net/web3d/testing/diff_record_playback.png
> showing the cone geometry speckled with dozens of different pixels.
> .
> Hypotheses:
> H0: a small numerical difference of 1/256 due to floating point rounding in opengl or video board
> Test: do a pixel compare channel by channel, to see the maximum difference
> http://dug9.users.sourceforge.net/web3d/testing/freewrl_snapshot_record.bmp
> http://dug9.users.sourceforge.net/web3d/testing/freewrl_snapshot_playback.bmp
> If it's a 1/256 rounding error, then a special compare function could be used/developed that would forgive that small difference
> H1: I didn't apply the mouse actions at the right point in the playback frame
> H2: the floating point rounding is on the mouse coordinates (when converted to screen) and I don't have enough decimal places in the fprintf
> H3: ???
> .
> I wonder if I should check it in, or forget it as hopeless, or do more tests.
.
If I run the -playback a second time, and compare the 2 playback images, there are zero differences. If I had implemented -record mode as a recording of mouse and keyboard, followed by a -playback to get the snapshot images, then I would see no differences between record and playback snapshots.
.
Summary: the way I implemented record and playback are not the same - a slightly different process or rounding, gives slightly different images. (for example the mouse coordinates are normalized to -1.0 to 1.0 floating point range, and in the record mode I use them directly, in playback they go through fprintf and sscanf which round them)
.
Conclusion: comparing record and playback images will work for functional testing, once implemented properly.
-Doug