Re: [Plplot-devel] Comprehensive testing before the release this weekend

[Alan W. I. <ir...@be...>]

On 2010-09-30 11:42-0700 Alan W. Irwin wrote:

> I have been tweaking scripts/comprehensive_test.sh to produce more
> informative and well-documented results, and I am now (revision 11233)
> fairly satisfied with it.  Thus, I encourage all those here who would
> like to help out with testing before the release this weekend (and who
> have ~20GB of free disk space!) to try out this script.

I have just completed the cmake-2.6.4 test (successfully, I might add)
so this is my first chance to find out the size of files generated by
this comprehensive test for a fully loaded Debian testing platform
which will build and test almost all components of PLplot. That size
turns out to be 34G so the above estimate was considerably too small.
So be careful out there!  OTOH, if you do have that sort of free disk
space, please go ahead and run the script to potentially improve our
release if you find any issues.  Also please report results at
http://www.miscdebris.net/plplot_wiki/index.php?title=Testing_PLplot
to let others know the reliability of PLplot for your particular
platform and also to gain some public credit for taking the time
(about 2 hours) to run the script and babysit the remaining relatively
few keystrokes/mouseclicks you have to perform to get through the
interactive parts of the test for each of the three major
configurations.

Alan






>
> To find out more about what is possible with this script, run
>
> scripts/comprehensive_test.sh --help
>
> There are many options there to cut down the scope of what is tested
> (say if you are short of disk space for the non-interactive tests or
> in case you don't want to do the interactive tests), but I would urge
> testers not to use those options in order to create as comprehensive a
> test as possible.
>
> Here is how I typically set up before running this script
>
> export CFLAGS='-O3 -fvisibility=hidden'
> export CXXFLAGS='-O3 -fvisibility=hidden'
> export FFLAGS='-O3 -fvisibility=hidden'
>
> The -fvisibility=hidden option minimizes the number of symbols in our
> libraries and reduces startup latency for every one of our PLplot
> examples so it should be an efficiency win for these comprehensive
> tests to some degree although I haven't checked how much.  The -O3
> optimization makes the builds longer but the tests shorter.  My
> feeling is it will be a net efficiency win, but I haven't checked that.
>
> I also then put the cmake version I want to test on my PATH then run
>
> scripts/comprehensive_test.sh \
> --cmake_added_options '-DPLD_pdf=ON -G "Unix Makefiles"' \
> --build_command 'make -j4'
>
> The --build_command option is mandatory and allows users to specify
> any make parallel build options that are optimum for their system.
> (-j4 for GNU make seems to be pretty efficient for my own two-cpu
> Intel core duo system.) The --build_command option theoretically (not
> tested yet) also allows some other build command such as nmake to be
> used on Windows systems with bash (either winbash or MSYS bash)
> installed.
>
> The -G "Unix Makefiles" part of the above illustrates how to specify a
> cmake generator, but it is, of course, redundant on Linux since that is
> the default cmake generator.
>
> I specified the -DPLD_pdf=ON option above because I wanted to test
> that non-default driver.  (It is still non-default because of problems
> with the vanilla libharu library upon which it is based, but I have
> patched the libharu build following instructions in
> cmake/external/libharu/README.libharu so I can specify -DPLD_pdf=ON in
> this case without introducing test errors.)
>
> WARNING, by default to above test is comprehensive (as its name
> implies).  For each of the three major build configurations (shared
> library/dynamic devices, shared library/nondynamic devices, and static
> library/nondynamic devices) it does 7 major tests (ctest,
> test_noninterative, and test_interactive in the build tree;
> test_noninterative, and test_interactive in the installed examples
> tree with the cmake-based build system there; and test_noninterative,
> and test_interactive in the installed examples tree using the
> traditional (Makefile + pkg_config) build system there. The 4
> non-interactive tests for the 3 major build configurations consumes
> something like 25GB of disk space for all the plotted file results so
> only those with sufficient spare disk space should run this script.
>
> Also, the point of the interactive part of these tests (run after the
> noninteractive part is completed for each of the three major
> configurations) is simply to check for any major errors (segfaults and
> the like) with all the configured interactive device drivers and
> interactive environments).  So to avoid driving you mad by doing lots
> of clicking to get through these tests, I have used the -np (no pause)
> option wherever I could.  So an awful lot of different interactive
> plots scream by when running these tests so don't expect to use them
> to check for any subtle rendering issues.
>
> For both the non-interactive and interactive part of this testing any
> major errors (e.g., segfaults) are stored in the *.out files the above
> script collects.  So after the script completes you can test for such
> major errors in both the noninteractive and interactive testing as follows:
>
> software@raven> grep -i error \
> comprehensive_test_disposeable/*/output_tree/*.out
> comprehensive_test_disposeable/nondynamic/output_tree/traditional_make_interactive.out:All
> interactive tests completed without major errors
> comprehensive_test_disposeable/shared/output_tree/traditional_make_interactive.out:All
> interactive tests completed without major errors
> comprehensive_test_disposeable/static/output_tree/traditional_make_interactive.out:All
> interactive tests completed without major errors
>
> Last night I completed the above test for cmake-2.8.2 with no major
> errors revealed by the above grep statement.  I am currently going
> through the whole process again with cmake-2.6.4.  (Testing for two
> different versions of cmake is overkill for most testers, but I felt
> it should be done by at least one tester like me for due diligence
> until we bump our minimum CMake version to 2.8.x.) I fully expect
> those cmake-2.6.4 results to also be good, and in any case I intend to
> report all 42 major tests using the simple table at
> http://www.miscdebris.net/plplot_wiki/index.php?title=Testing_PLplot.
> If I have time before the release I may also try running the
> comprehensive test script with -G "MSYS Makefiles" for MinGW/MSYS
> under Wine and, of course, reporting the results on our Wiki as well.
>
> No one tester can test all combinations of cmake options and build
> platforms so I invite anyone else wanting to help out with testing
> this release to run the script (which by default does 21 major tests)
> on our svn trunk version (what is about to become 5.9.7) and report
> the results at the above page on our wiki.
>
> Alan
> __________________________
> Alan W. Irwin
>
> Astronomical research affiliation with Department of Physics and Astronomy,
> University of Victoria (astrowww.phys.uvic.ca).
>
> Programming affiliations with the FreeEOS equation-of-state implementation
> for stellar interiors (freeeos.sf.net); PLplot scientific plotting software
> package (plplot.org); the libLASi project (unifont.org/lasi); the Loads of
> Linux Links project (loll.sf.net); and the Linux Brochure Project
> (lbproject.sf.net).
> __________________________
>
> Linux-powered Science
> __________________________
>
> ------------------------------------------------------------------------------
> Start uncovering the many advantages of virtual appliances
> and start using them to simplify application deployment and
> accelerate your shift to cloud computing.
> http://p.sf.net/sfu/novell-sfdev2dev
> _______________________________________________
> Plplot-devel mailing list
> Plp...@li...
> https://lists.sourceforge.net/lists/listinfo/plplot-devel
>

__________________________
Alan W. Irwin

Astronomical research affiliation with Department of Physics and Astronomy,
University of Victoria (astrowww.phys.uvic.ca).

Programming affiliations with the FreeEOS equation-of-state implementation
for stellar interiors (freeeos.sf.net); PLplot scientific plotting software
package (plplot.org); the libLASi project (unifont.org/lasi); the Loads of
Linux Links project (loll.sf.net); and the Linux Brochure Project
(lbproject.sf.net).
__________________________

Linux-powered Science
__________________________