Re: [FreeWRL-develop] Testing was Codebase port to c++?

[doug s. <hig...@ho...>]

Paulo,
>
>>> Recently we were wondering if it's possible/how to do more automated
>>> testing with graphical programs. We have a way to 'dump' a scene (as
>>> text, showing scenegraph) to a file. And we have a way to do a screen
>>> snapshot (to an image file). One more idea: have some way to specify
>>> the number of loops to advance (perhaps also time?) But nothing
>>> working today as automated test suite.
>> Paulo,
>> There was also an idea of a record mode in freewrl: specifically:
>> - number of frames
>> - input: keystroke, mouse action
>> - snapshot
>> - dumpscene
>> And perhaps put it in a special file format, for playback during testing.
>> To make the number of frames smaller -so the tests run faster- the
>> record mode could purposely run slow - pausing on each frame.
>> -Doug
> I've written a small perl test script that's using a library that I'm
> currently maintaining that allows keyboard and mouse interactions when
> running on a X11 env, it currently does the following:
> - Opens freewrl with a given test wrl file.
> - Takes a snapshot
> - Goes to the next file
>
> I have the following problems:
>
> - Sometimes freewrl takes some time to load and gives no indication of
> when it has finished rendering. We could use such hint in order to
> trigger a snapshot after rendering rather than "sleeping" and hope it's
> enough....
.
Perhaps for both fixture creation and later comparision testing the main program could be different or go into a different loop, holding off rendering until parsing is finished, or not counting frames until it's finished. A commandline parameter might put it into fixture mode or testing mode.
.
> - Currently the snapshot only seems to work with PNG which has a file is
> not suitable for comparison with for instance a previous master version.
.
Because of compression? Is there a way to turn off compression? If so then it switch on a global testing mode parameter. Or is there more saved, such as the date and timestamp in the .png, causing a generic binary file compare to fail?
(win32 saves .bmp which has no compression -great for testing- but it's a pain then emailing a screenshot because I have to use another tool to convert it to .png for cross-platform consumption)
Q. instead of saving to any standard file format, don't we first get a binary blob from opengl somehow, and could we just save the blob and use it for comparison? Perhaps store widthxheight in another file (playback file?) in case someone wants to visually check a failed comparison? Or would we always know the widthxheight by setting the opengl window to a standard size for testing?
.
> - It would be interesting to know how to move to a specific viewpoint
> say 45 degrees to the left,right, up, down... without having to manually
> do so by means of keyboard/mouse
>
> I'm not entirely sure that comparing image snapshots with a previous
> known good image is the right way to do it but at least it exercises
> some of the points you where making above.
>
> My idea based on your comments is somehow similar to this; If we imagine
> a cylinder around the main item of a given test file perhaps we could do
> the following....
> - Open a test file
.
Good. Right now there are separate threads for reading/parsing the file, and the rendering loop. Perhaps in test mode there would need to be some counting of frames after load is finished.
For example if just parsing is being tested, then the 1st frame after parsing is complete.
> - Wait for rendering to finish.
Some tests are needed for things like animation and routing, which happens over multiple frames. Some animation things are simple interpolators with a time or delta-time input.
One idea is to save both frames-after-parsing and the double dtime value to go with it, during the test fixture generation:
{int frame, double dtime, ?other?}
then during test run, instead of using clock time substitute the saved dtime, to match the fixture run.
.
> - Take a snapshot
.
During fixture generation save the {snapshot, iframe#, dtime} (or else perhaps save the snapshot command keystroke?) and during testing save the snapshot on iframe#, then do a binary file compare
.
> - Move to a different viewpoint coordinate on the cylinder and take a
> snapshot. and so on
> - North, South, East, West, Above, Below
> - Compare those snapshots with previously good known ones.
.
Some scenes and things you want to test are not single objects but rather spread out scenes, and sometimes the things being tested are not even graphical - such as routing and script tests that show up in the text console. So not every test could benefit from this exact sequence. 
The graphics window is a proxy for the scenegraph state. So one way to test the non-graphical stuff is by saving the scenegraph state. Except a problem: it's full of memory pointers, which aren't the same between runs (malloc creates them different each time), and we may be trying to change the scenegraph storage - the subject of our tests. And not all things are stored in the scenegraph. But for some tests some kind of pointer-neutralized dump of the scenegraph might make a good test fixture.
.
So I think we would need general / flexible approach to testing to cover all cases. I kind of like what Michel had been attempting, saving the mouse coordinates for playback.
Perhaps what's needed is a playback file that records a lot of things, per post-parse frame#
playback[i] = {iframe, dtime, keystrokes or NULL, mouse (xy,button sequence) or NULL, snapshot URL or NULL, scenegraph_dump URL or NULL, ?other?}
.
>
> *This would obviously not cope well with moving objects in the scene but
> that's a different ball game.
.
Perhaps the above detailed playback file approach would/could be made to cover the moving objects scenario?
.
>
> What do you guys think about this ?
>
> I've pushed my test script to github and it lives in the branch called
> testing-freewrl.
> More details about it here
> https://github.com/pecastro/freewrl/commits/testing-freewrl
Thanks Paulo, great work - I'll check it out this weekend.
-Doug