Re: [Flightgear-devel] Crash in routePath.cxx (#1703)

[James T. <zak...@ma...>]

> On 18 Mar 2017, at 08:22, Edward d'Auvergne <tru...@gm...> wrote:
> 
> This is one disadvantage of CppUnit, having a single executable.  The
> design of CppUnit is that the setUp() and tearDown() functions are
> used to initialise and then reset the state of the executable, rather
> than having individual executables starting from scratch and then
> terminating execution.  Individual tests can be threaded but, when
> global data structures are being accessed or modified, this is not the
> best practice.  Using CppUnit requires that individual tests do not
> run for a long time.  For example I use a rule of max 10 s for
> individual tests in my own projects, with some minor exceptions.  This
> would also mean that testing via Jenkins may be required to keep the
> test suite in a functional state, otherwise a segfault in one test
> will make the test suite non-operational.
> 
> However this is sometimes also an advantage.  For example if there are
> extensive system/functional tests in the future testing out the
> cross-interactions between subsystems, OSG, the NavDataCache, etc., as
> individual executables these would be quite large in size, and the sum
> of all of them could quickly explode (considering the fgfs binary is
> ~200 Mb).  Unit tests are generally small, and system tests are often
> orders of magnitude bigger in size.  The current design is that you
> have a base executable of ~200 Mb consisting of all of fgfs, and the
> tests then just add linearly to that size.
> 
> The other advantage of the current design is that you don't need to do
> any CMake magic to get the test to run.  All of fgfs is there, so this
> would be a great tool for new developers (myself included) to learn
> about the fgfs internals without worrying about integrating via CMake
> the large dependency chains that a single fgfs code path pulls in.
> You just copy a skeleton framework for a series of tests (if not
> already there), and code pure c++ into the setUp(), tearDown() and
> test*() functions.
> 
> So there are advantages and disadvantages to the CppUnit testing
> framework, compared to the CTest testing framework.  I chose CppUnit
> mainly because it is simpler for beginners to dive into the code - and
> this is where my FGPythonSys experiment was aiming for.

Okay, I solved this by building FGFS (except main/boostrap) as a library for the tests to link against. I think the inability to run tests in parallel is a bit daft, but not the end of the world. We already have tests that take longer than 10s to run however, and I would expect many that take 30s or event a minute. Okay, we can live with that, developers only run the tests when they want, and it doesn’t matter if Jenkins builds take a bit longer.

I’m not sure what you mean by 'don't need to do any CMake magic to get the test to run.’ - CTest just runs any executable, and it part of CMake, which people already have to have to build FlightGear, so I don’t think I quite understand your reasoning here? With Florent’s refactor a few months ago of the SimGear tests, I would say CTest + the very basic ‘CppUnit-alike’ (actually I was inspiredt by QTest, but that of course was inspired by CppUnit, which was inspired by JUnit….) is working pretty nicely, of course unit-testing SimGear is a much easier job.

Anyway, not a big concern, I do think it would be worth making the common code a library, then we can different CppUnit test executables and parallelise if the time issue becomes larger.

Kind regards,
James