Re: [Flightgear-devel] RFC: Working on a testing strategy for FlightGear.

[Edward d'A. <tru...@gm...>]

On 5 March 2018 at 17:19, James Turner <ja...@fl...> wrote:
>>
>> My idea would be to include it into the 3rdparty/ directory.  The
>> current version from
>> https://www.freedesktop.org/wiki/Software/cppunit/ is pretty light:
>>
>> Removing the examples directory, it is 3.2 Mb.  Comparing to the
>> current 3rdparty packages:
>
> Yeah agreed, if we can go that route it’s fine with me. Just be ware we then need a SYSTEM_CPPUNIT option to keep the distros happy :)

Ok, done - I've pushed this to my latest cppunit/r9 flightgear branch.
I've tested it on MS Windows and I see that the test_flightplan tests
die a horrible death there :S  I think I need to update my debugging
tools on that VM to do anything about it.  I also unfortunately no
longer have a Mac OS X testing system to check this on.  Anyway,
CppUnit is now built into the run_test_suite{.exe} monolithic binary
on all OSes and it is no longer a dependency.


>> I think this simplicity for test writers would make it easier for more
>> potential developers to learn about the fgfs internals.  They could
>> start contributing to fgfs by writing unit or system/functional tests
>> to learn about the fgfs internal design.  If we had good test
>> coverage, then you could say that all new developments need to come
>> with one or more tests.  That is how I do things.
>
> Well, personally I do not think the complexity for new developers (or old ones) is really so different between any of the test frameworks (our own macros or CppUnit or whatever), but, okay, it’s not such a big concern.
>>
>> With the current monolithic design, the binary file is 215 Mb.  But
>> with the current set of 8 unit and simgear tests, the whole test suite
>> execution, including startup time, is 30-40 ms.  It does have more of
>> an effect on gdb and valgrind though.
>
> Yeah that is a slightly concern, since again debugging something like a terraSync issue is much easier in the test harness code than inside FG. So I am slightly concerned by that.

Me too, but I thought that ~5 seconds wasn't too painful for a Debug build:

$ time gdb test_suite/run_test_suite -ex quit
[snip]
Reading symbols from test_suite/run_test_suite...done.

real    0m4.947s
user    0m4.679s
sys     0m0.243s
$

Debugging a single test in this large binary would be no different
from debugging a single light-weight binary for a single test.  For
Valgrind, you would probably need to use the "--gen-suppressions=yes"
option to remove the large number of "possibly lost" and "still
reachable" errors that a blank fgfs instance produces, so that the
report for the single test only shows the errors for that test.
Additional suppressions for external libraries could also be used, OSG
for example [1].  For example in my experimental FGPythonSys branches
I use "--suppressions=/data/python/src/Python-3.5.2/Misc/valgrind-python.supp".


> This is why I went for the shared library approach - do you think that’s really not viable with CppUnit? or do you have some other reason to avoid it?
>
> For me it has some ugly hacks to make the test-lib, but it’s proved relatively low hassle to maintain, and easy to make tests as ‘big’ or ‘small’ as I needed, and still run quickly. And I don’t think the test architecture relates to the test support library we use or don’t use - unless I am missing something?

I actually did have an experimental branch early on where I built a
shared library in exactly the same way as in tests/CMakeLists.txt.
However I decided against it.  The main reason was to shift as much of
the complexity on to the testing framework maintainer and away from
the test writer.  I.e. when adding tests in this shared library
system, the test writer would have to add files to the CMake SOURCES
and HEADERS lists, chasing down the include dependency chain until
everything that was required was added.  I know there are gcc options
to display the dependency tree, but not all new potential developers
might know this, or know the gcc options required to use to make use
of the -H option, or to use -DCMAKE_VERBOSE_MAKEFILE:BOOL=ON to find
those options.  Anyway, in my current design, test writers don't touch
this part.  This design is that the framework is set up to absolutely
minimise what the test writer has to do.

Another reason was because I wanted the binary to mimic the exact
behaviour of fgfs.  As fgfs is monolithic, so is the test suite
binary.  I think this has advantages for system tests where the
interaction between different components is tested.  And finally, I
was having trouble getting the shared library to work in MS Windows.

And finally, if we ever reach 100% test coverage one day, then the
shared library will end up being the whole of the fgfs binary with the
bootstrap.cxx main() function stripped out.  So it won't be too
different from the current monolithic binary once loaded into memory.


>> In this workflow, do you run the individual tests or all tests?  If we
>> end up with 100% test coverage, you would have to run individual tests
>> otherwise it will take too long.  That is the idea with CppUnit, you
>> would write code and then run:
>
> I run all the tests in say catalog_test or test_repository or test_flight_plan - i.e these 4 or 8 or 20 tests should take maybe 15 or 20 seconds maximum, and then I know my hacking worked and I can continue. This is not the only workflow possible of course but it’s certainly the most productive for me.

That sounds very similar to what I do, and what is planned for the
test suite, i.e.:

$ make test_suite
$ ./test_suite/run_test_suite -s [Test_suite]     # A collection of
related tests.
$ ./test_suite/run_test_suite -s [Test_suite].[test_name]     # A single test.

With ctest, each test suite could be wrapped up, but exposing
individual tests would require a huge list in
test_suite/CMakeLists.txt (which could be monitored to see if it
synchronised with the test suite by a special type of software
varification test - a different test category - within the test
suite).  Therefore apart from what is typed, such a workflow is not
really changed.

Regards,

Edward


[1]  http://www.openscenegraph.org/index.php/documentation/guides/user-guides/115-debugging-with-valgrind