|
From: James T. <zak...@ma...> - 2017-03-18 11:06:39
|
> On 18 Mar 2017, at 08:22, Edward d'Auvergne <tru...@gm...> wrote: > > This is one disadvantage of CppUnit, having a single executable. The > design of CppUnit is that the setUp() and tearDown() functions are > used to initialise and then reset the state of the executable, rather > than having individual executables starting from scratch and then > terminating execution. Individual tests can be threaded but, when > global data structures are being accessed or modified, this is not the > best practice. Using CppUnit requires that individual tests do not > run for a long time. For example I use a rule of max 10 s for > individual tests in my own projects, with some minor exceptions. This > would also mean that testing via Jenkins may be required to keep the > test suite in a functional state, otherwise a segfault in one test > will make the test suite non-operational. > > However this is sometimes also an advantage. For example if there are > extensive system/functional tests in the future testing out the > cross-interactions between subsystems, OSG, the NavDataCache, etc., as > individual executables these would be quite large in size, and the sum > of all of them could quickly explode (considering the fgfs binary is > ~200 Mb). Unit tests are generally small, and system tests are often > orders of magnitude bigger in size. The current design is that you > have a base executable of ~200 Mb consisting of all of fgfs, and the > tests then just add linearly to that size. > > The other advantage of the current design is that you don't need to do > any CMake magic to get the test to run. All of fgfs is there, so this > would be a great tool for new developers (myself included) to learn > about the fgfs internals without worrying about integrating via CMake > the large dependency chains that a single fgfs code path pulls in. > You just copy a skeleton framework for a series of tests (if not > already there), and code pure c++ into the setUp(), tearDown() and > test*() functions. > > So there are advantages and disadvantages to the CppUnit testing > framework, compared to the CTest testing framework. I chose CppUnit > mainly because it is simpler for beginners to dive into the code - and > this is where my FGPythonSys experiment was aiming for. Okay, I solved this by building FGFS (except main/boostrap) as a library for the tests to link against. I think the inability to run tests in parallel is a bit daft, but not the end of the world. We already have tests that take longer than 10s to run however, and I would expect many that take 30s or event a minute. Okay, we can live with that, developers only run the tests when they want, and it doesn’t matter if Jenkins builds take a bit longer. I’m not sure what you mean by 'don't need to do any CMake magic to get the test to run.’ - CTest just runs any executable, and it part of CMake, which people already have to have to build FlightGear, so I don’t think I quite understand your reasoning here? With Florent’s refactor a few months ago of the SimGear tests, I would say CTest + the very basic ‘CppUnit-alike’ (actually I was inspiredt by QTest, but that of course was inspired by CppUnit, which was inspired by JUnit….) is working pretty nicely, of course unit-testing SimGear is a much easier job. Anyway, not a big concern, I do think it would be worth making the common code a library, then we can different CppUnit test executables and parallelise if the time issue becomes larger. Kind regards, James |