Re: [Flightgear-devel] RFC: Working on a testing strategy for FlightGear.

[James T. <ja...@fl...>]

> On 7 Mar 2018, at 11:20, Edward d'Auvergne <tru...@gm...> wrote:
> 
>> Yeah that is a slightly concern, since again debugging something like a terraSync issue is much easier in the test harness code than inside FG. So I am slightly concerned by that.
> 
> Me too, but I thought that ~5 seconds wasn't too painful for a Debug build:

Yeah that’s okay, no worries.
> 
> I actually did have an experimental branch early on where I built a
> shared library in exactly the same way as in tests/CMakeLists.txt.
> However I decided against it.  The main reason was to shift as much of
> the complexity on to the testing framework maintainer and away from
> the test writer.  I.e. when adding tests in this shared library
> system, the test writer would have to add files to the CMake SOURCES
> and HEADERS lists, chasing down the include dependency chain until
> everything that was required was added.  I know there are gcc options
> to display the dependency tree, but not all new potential developers
> might know this, or know the gcc options required to use to make use
> of the -H option, or to use -DCMAKE_VERBOSE_MAKEFILE:BOOL=ON to find
> those options.  Anyway, in my current design, test writers don't touch
> this part.  This design is that the framework is set up to absolutely
> minimise what the test writer has to do.
> 
> Another reason was because I wanted the binary to mimic the exact
> behaviour of fgfs.  As fgfs is monolithic, so is the test suite
> binary.  I think this has advantages for system tests where the
> interaction between different components is tested.  And finally, I
> was having trouble getting the shared library to work in MS Windows.
> 
> And finally, if we ever reach 100% test coverage one day, then the
> shared library will end up being the whole of the fgfs binary with the
> bootstrap.cxx main() function stripped out.  So it won't be too
> different from the current monolithic binary once loaded into memory.

That’s fair, I sure don’t claim the testlib approach is nice in that regard. However, does this restrict our ability to control / inject things in a test situation? Eg replace defaults.xml or the property tree or the nav-cahce or Nasal/* ? My concern is if it’s the same .o files, we can’t use pre-processor #defines to expose extra data or allow things to be over-ridden in a test situation. I’m nervous of need to make the real binary so configurable that we can use it directly or testing, becuase this might have security implications.

Kind regards,
James