Re: [Cppqed-support] new testsuite

[Raimar S. <rai...@ui...>]

There was a commit not pushed to sourceforge in CCQEDcore, this is now fixed.

Raimar

On Thursday, December 19, 2013 03:40:42 you wrote:


Hi András,

I have integrated a new testsuite into cmake.

Please clone the testing branch of the monolithic repository from the private repository:

git clone -b testing --recursive gi...@ge...:cppqed/complete.git

Then build everything with the monolithic build method (using the toplevel CMakeLists.txt file). 
Please make sure that no outside components are picked up by cmake, probably it is best to 
delete ~/.cmake. Afterwards, in the build directory run ctest -jN where N is the number of cores. 
Please let me know if this all works and if the tests pass.

A few notes to the structure now, an I will try to write a complete documentation soon.

The testsuite uses ctest from within cmake, which means the complete testsuite can be run by 
either 'make test' or the ctest command from above. The ctest command has the advantage of 
being able to run the tests in parallel with the -j switch. CTest itself is rather 'dumb', basically 
one only has to specify a test name and a test command including command line options in 
the CMakeLists.txt file.

In our case, the command is either the executable target[1] named 'tester', which we have built 
with the boost unittest framework[2], or a python script[3] which I wrote as test driver. The 
former is for testing code snippets, your boost tests can be imported that way. The latter is 
basically for running scripts. At the moment the testdriver.py is capable of:

* running a C++QED script in all modes (single, ensemble, master)
* continuing a script
* comparing the outcome of a C++QED script to saved and version controlled output in 
Testing/expected
* comparing the outcome of a C++QED script to the outcome of another script
* check that the compilation of a cmake target fails in an expected way[4] 

and the functionality can be easily extended. The behavior of the testdriver is controlled by 
command line options and configuration files testdriver.conf in the subdirectories. 

Note that neither 'make test' nor 'ctest' will trigger a re-configuration or re-compilation in the 
case of changed CMakeLists.txt files or changed source files. To overcome this problem I 
have defined the target 'check'. This should depend on everything essential and therefore 
trigger re-configuration and compilation. Every subdirectory of the testsuite (at the moment 
boost, compilefail,continue and run) has its own 'check_subdir' target, and additionally every 
single test has its own target. This is very convenient when developing or debugging a feature, 
one does not have to run the whole testsuite all the time but can fine-tune which tests to run by 
adding them to the buildset in kdevelop. Additionally, all the check* targets are a lot more 
verbose, because they use ctest -v internally.

The next steps:

* documentation of testdriver.py, the testdriver.conf syntax and how to add new tests to cmake
* making Testing a standalone project, at the moment the testsuite is only available in 
monolithic builds
* import all the other boost unittests which exist
* import the other CompositeError files into the compilefail submodule (and I think the 
CompositeError2 which I used so far gives additional compilation errors which should not exist, 
but I'm not sure. Please check the output of make CompositeError2.)
* write more tests to cover a broader range of features and design promises
* integrate the physics testsuite
* anything else?

Best regards
Raimar



--------
[1] http://bit.ly/1gHiTC0
[2] http://bit.ly/1gHjFyW
[3] http://bit.ly/1gHj2FA
[4] http://bit.ly/1gHlPi4