[Flightgear-devel] RFC: Working on a testing strategy for FlightGear.

SourceForge Headquarters 225 Broadway Suite 1600 San Diego, CA 92101 +1 (858) 454-5900

Hi,

I have recently found some time to work on my CppUnit test suite
branches for flightgear/simgear, rebasing to the latest code.  These
have been pushed to:

    https://sourceforge.net/u/edauvergne/flightgear/ci/cppunit/r9/~/tree/
    https://sourceforge.net/u/edauvergne/simgear/ci/cppunit/r8/~/tree/

Just add "git.code.sf.net/u/edauvergne/flightgear" or
"git.code.sf.net/u/edauvergne/simgear" as remotes and checkout the
branches "cppunit/r9" and "cppunit/r8" branches respectively (for more
details, see [1]).  As before, I have my FGPythonSys branches rebased
on these which demonstrate the advantages of this CppUnit framework
for quickly writing tests (with 242 unit tests in one test suite, and
one failing test as a demo):

    https://sourceforge.net/u/edauvergne/flightgear/ci/python/r11/~/tree/
    https://sourceforge.net/u/edauvergne/simgear/ci/python/r8/~/tree/
    https://sourceforge.net/u/edauvergne/fgdata/ci/python/r5/~/tree/

Before I work on any more code, I'll like to make a request for
comments.  I have converted one of James' newer ctests to the testing
framework as an example, more will follow if there is interest.  I'll
split this up, as there are unfortunately a lot of details.  For
reference, previous threads on this subject are given below [1].

* Automated testing

Before getting into design details, I would like to discuss options
for automated unit/functional testing, as this would influence how the
test suite is built.  I am assuming that one day we will have
significant test coverage of both fgfs and fgdata, and that running a
full set of tests would take a significant amount of time (say 30 min
to 1 hour).  My idea was that this could be run on a Jenkins VM, or a
similar system, triggered in the same way Gene's Jenkins set up is
currently run, i.e. "Started by an SCM change".  If run on Jenkins,
then it could either be run together with the flightgear build or as a
separate project (with "flightgear" set as an upstream project).  This
would help identify who accidentally broke what.

The current design of this test suite is that a single binary file is
created for all tests.  This is quite different to the current CTest
design where each test is a separate binary, each with a main()
function, and all using a new shared library (built as a subset of
fgfs).  It is located at
${PROJECT_BINARY_DIR}/test_suite/run_test_suite.  This binary is a
Frankenstein version of fgfs.  It is the entire binary with the
bootstrap code and its main() function ripped out and replaced by an
empty shell.  This has a big size advantage for a test suite once test
coverage is extensive.  It can also be easily downloaded and re-run by
any user on any OS.  For a developer testing locally, rebuilding the
test suite is the same as rebuilding fgfs.

As discussed previously, the ability to run groups of tests (test
suites) or individual tests on the command line is planned, but not
yet implemented.  I haven't coded this yet, as I don't want to waste
time on ideas that would be rejected.  So as a developer you would run
a single test with, for example:

$ ./test_suite/run_test_suite -u Flightplan.testRoutePathSkipped

* CTest

Although currently not implemented, the whole run_test_suite binary
could be run as a single CTest.  That would require 2 lines of new
CMake code.  Or, for parallelised execution, each of the test suites
could be a seperate ctest by executing for example "run_test_suite -u
Flightplan".  However for the automated testing where this would
likely be on a single core VM, a single serial test run would be
sufficient.  The "make test_suite" build target would be modified to
only build, and not run the test suite, and then you would run "ctest
-V" (to view the progress).

* James' flight path example

For the latest branches, I would like to take James' flight path
CTest, with 4 embedded tests, as an example.  I have migrated this
from the current CTest set up to the CppUnit framework:

    https://sourceforge.net/u/edauvergne/flightgear/ci/119218c0fbb7ec7331b3c2ccd2827963ada6446e/

Some notes about this migration:

    1)  Note the CMakeList.txt additions required for this.  You only
need to add the new test code and headers.  All fgfs sources and
headers are already there.  So this is a big advantage for test
writers.  You don't need to carefully build your own binary, you just
write test code and include whatever you like.  It is the same as
writing code directly for fgfs (excluding any dummy data and
replacement data structures you'd like to use).

    2)  The simgear testing macros have been replaced by the CppUnit
framework equivalents [2].  This allows for better failure vs. error
diagnostics by the framework.

    3)  The tests are now more robust.  The previous main function was:

int main(int argc, char* argv[])
{
    fgtest::initTestGlobals("flightplan");

    testBasic();
    testRoutePathBasic();
    testRoutePathSkipped();
    testRoutePathTrivialFlightPlan();

    fgtest::shutdownTestGlobals();
}

The new code is run in protected execution mode (you can test for
exceptions with CPPUNIT_ASSERT_THROW), so it would be more equivalent
of the following:

int main(int argc, char* argv[])
{
    protectedExec(fgtest::initTestGlobals("flightplan"));
    protectedExec(testBasic());
    fgtest::shutdownTestGlobals();

    protectedExec(fgtest::initTestGlobals("flightplan"));
    protectedExec(testRoutePathBasic());
    fgtest::shutdownTestGlobals();

    protectedExec(fgtest::initTestGlobals("flightplan"));
    protectedExec(testRoutePathSkipped());
    fgtest::shutdownTestGlobals();

    protectedExec(fgtest::initTestGlobals("flightplan"));
    protectedExec(testRoutePathTrivialFlightPlan());
    fgtest::shutdownTestGlobals();
}

I.e. the "globals" replacement object is reset between every test,
independent of failures or errors in the initialisation or test code.
I.e. the data structures should be clean between tests and one test
cannot contaminate another (assuming the shutdown or tearDown()
function is well written).  Segfaults, memory leaks, etc. are not
caught though, so this can have a dramatic effect on the testing.

    4)  Apart from the 3 points above, the code is identical.

    5)  Memory leaks - passing the test suite through Valgrind shows
four "definite leaks" coming from these flight plan tests
(NavDataCache.cxx:1272, globals.cxx:164, etc.), all triggered by
FlightplanTestCase::setUp().

* Shared data and data structures

Still to do for the flight plan testing is to shift the dummy globals
code from tests/unitTestHelpers.cxx into a test suite subdirectory
(say test_suite/shared_data/).  This subdirectory would contain all
dummy data structures and minimal data sets for the whole test suite.
With good test coverage, this directory will end up being quite large.

Any feedback on these ideas would be appreciated.

Cheers,

Edward

[1]  Previous CppUnit test suite thread include:

* A FlightGear test suite using CppUnit. -
https://sourceforge.net/p/flightgear/mailman/message/35661026/

* Crash in routePath.cxx (#1703) -
https://sourceforge.net/p/flightgear/mailman/flightgear-devel/thread/76b0bc63-98f6-594f-07da-87797c6f4943%40gmail.com/#msg35708020

* How to test C++ code 'automatically'? -
https://sourceforge.net/p/flightgear/mailman/message/35800794/

[2]  https://people.freedesktop.org/~mmohrhard/cppunit/_test_assert_8h.html

[Flightgear-devel] RFC: Working on a testing strategy for FlightGear.

FlightGear Flight Simulator: free open-source multiplatform flight sim

[Flightgear-devel] RFC: Working on a testing strategy for FlightGear.