Test_System(CTest_Based)

The CTest-based test system

This test system is one of three implemented for PLplot with the other two being the CMake-based test system and the legacy test system. Other wiki links relevant to PLplot testing are a general discussion of PLplot testing and a summary of our comprehensive testing reports.

The following two sets of commands are typical ways to use this test system for the CMake "Unix Makefiles" generator in the core build tree and in the build tree for the installed examples.

# Example of using the CTest-based test system for the core build tree.
# 1. Configure a PLplot core build with CMake as normal in an initially
# empty build tree.
CORE_BUILD_DIR=$HOME/plplot_core_build_dir
rm -rf $CORE_BUILD_DIR
mkdir $CORE_BUILD_DIR
cd $CORE_BUILD_DIR
# N.B. the CTest-based test system does not work in the core build tree unless -DBUILD_TEST=ON.
cmake -DCMAKE_INSTALL_PREFIX=<install prefix> -DBUILD_TEST=ON <other cmake options> <top-level directory of the PLplot source tree> >& cmake.out
# 2. Build the "all" target.  This is necessary before the ctest command below will work.
make VERBOSE=1 -j<jobs> all >& all.out
# 3. Run ctest
ctest -j<jobs> --extra-verbose <other ctest options> >& ctest.out

# Example of using the CTest-based test system for the installed examples build tree.
# After steps 1 and 2 and possibly 3 above, do the following:
# 4. Install PLplot
make VERBOSE=1 -j<jobs> install >& install.out
# 5. Configure the CMake-based build system for the installed examples in an initially empty build tree.
# N.B. note that no CMake options are allowed for this step.
INSTALLED_BUILD_DIR=$HOME/plplot_installed_build_dir
rm -rf $INSTALLED_BUILD_DIR
mkdir $INSTALLED_BUILD_DIR
cd $INSTALLED_BUILD_DIR
# N.B. note there are no parameters allowed for this particular cmake command
cmake <install_prefix>/share/plplot<plplot_version>/examples >& ctest.out
# 6. Build the "all" target.  This is necessary before the ctest command below will work.
make VERBOSE=1 -j<jobs> all >& all.out
# 7. Run ctest
ctest -j<jobs> --extra-verbose <other ctest options> >& ctest.out

The tester should be able to infer from these "Unix Makefiles" examples how to use the CTest-based test system for any other CMake generator as well.

In the above commands:
$HOME is an environment variable pointing to your home directory;
<install_prefix> should point to an initially non-existent disk directory that will become the top-level directory of the install tree;
<other cmake options> can be any additional cmake options you want to specify to control the PLplot configuration;
<top-level directory of the PLplot source tree> is self-explanatory;
<jobs> is the maximum number of parallel jobs that will be executed by the make and ctest commands (for most efficient results this integer should match the number of hardware threads for your box (e.g., it should be 16 for a Ryzen 7 1700 system with 8 real cores and 16 threads);
<other ctest options> can include the "-D Experimental" option which is configured by CMake to submit a dashboard (summary of test results) to our CDash server and other options (normally not used) to further control ctest;
* <plplot_version> is the PLplot version number, e.g., "5.15.0".

You should search the *.out files generated by the above commands for obvious configuration, build, or run-time errors. You should also look for rendering issues associated with the (noninteractive) tests performed by ctest using an appropriate viewer (e.g., the ImageMagick "display" application) to view the files these tests generate in the ctest_examples_output_dir subdirectory of the build tree. Note this location is distinct from the examples/test_examples_output_dir subdirectory used to store plot file results for the CMake-based test system, but the names of the plot files within those two distinct directories are identical for the two test systems (and the file contents should be identical as well except possibly for timestamps). For further remarks about these filenames please consult the filenames subsection of the discussion of the CMake-based test system.

The above possibility of dashboard submission is a strong motivation for porting all noninteractive tests from our CMake-based test system to this one. CMake logic in plplot_test/CMakeLists.txt configures this test system and implements individual tests for it using the CMake add_test command. Such porting is straightforward since "COMMAND" syntax within the add_test command is interpreted identically to the "COMMAND" syntax within the add_custom_command command used to help implement the CMake-based test system in examples/CMakeLists.txt.

The resulting CTest test names for a Linux platform with essentially all PLplot prerequisites installed are as follows:

examples_ada
examples_c
examples_cxx
examples_d
examples_fortran
examples_java
examples_lua
examples_ocaml
examples_octave
examples_python
examples_tcl

examples_compare

examples_bmpqt
examples_epscairo
examples_jpgqt
examples_pdfcairo
examples_pdfqt
examples_pngcairo
examples_pngqt
examples_ppmqt
examples_ps
examples_psc
examples_pscairo
examples_psttf
examples_psttfc
examples_svgcairo
examples_svgqt
examples_tiffqt
examples_xfig

The first group of these CTest tests have the name pattern examples_<language> and correspond to the low-level test targets test_<language>_${PLPLOT_TEST_DEVICE} (where ${PLPLOT_TEST_DEVICE} is the test device which defaults to "svg") for this test system as well as the CMake-based test system. The next CTest test is called "examples_compare" and corresponds to the high-level test_diff_device test target for the CMake-based test system. The final set of CTest tests have the name pattern examples_<device> and correspond to the low-level test targets test_c_<device> in the CMake-based test system.

CTest tests do not depend on CMake targets other than the "all" target so it is necessary (as we see in the examples above) to build the "all" target before running the ctest command. Most of these CTest tests can be run in any order, but the examples_compare test must be run after the preceding tests in the above list are completed (since the results of those tests are compared by the examples_compare test), and that constraint is enforced by setting the DEPENDS property of the examples_compare test with the set_test_properties command.

The content of this page is available under the GNU Free Documentation License 1.2.