Re: [Plplot-devel] Bug in notcrossed() function of plfill.c

[Alan W. I. <ir...@be...>]

On 2016-09-13 10:44+0100 Phil Rosenberg wrote:

> On 13 September 2016 at 03:27, Alan W. Irwin <ir...@be...> wrote:
>> There is a substantial testing requirement for fill changes. The
>> reason for that is we have had years of mostly good user experience
>> with the present fill algorithm so I feel it is important that anybody
>> that changes the fill algorithm in any further way needs to
>> demonstrate with substantial testing that their change does not
>> introduce fill regressions.
>>
>> I have alluded previously to automated fill regression testing (for
>> all our standard examples that directly or indirectly use fill and all
>> our file devices).  The core of the idea is to compare old and updated
>> plot file results using imagemagick image differences.  Do you agree
>> this general approach (supplemented by actually looking at the "fill"
>> subset of our examples for each of our interactive devices) is the way
>> forward to insure that our further fill changes do not introduce more
>> issues than they fix?
>
> I think this is a good idea in general to avoid all sorts of
> regressions.

Good.

> Would it be easier to use the svg output and diff as the
> svg examples can be built on all systems?

Presumably you are aware of this already, but it should be mentioned
that one issue with diff results is you cannot reliably compare
results from two different floating-point (FP) platforms because FP
errors propagate to rounded results on two such platforms in different
ways.  So it is essential to calculate both the old and revised fill
result with exactly the same FP platform (i.e., you cannot store plot
results from one FP platform to compare with plot results from other
FP platforms).

Another issue with using diff even for the same FP platform is any
significant change in the fill code is likely to cause real
differences, e.g., the effective boundaries of the filled regions will
be slightly different.  In fact, this is a problem for all fully
automated methods I can think of to test for fill regressions; there
are going to be large local changes (as boundaries change slightly) in
results, but so long as the final result looks OK (i.e., no obvious
holes in any fill region and no significantly overfilled regions) then
the change does not count as a significant fill regression.

As a result of this concern, I am now thinking along the lines of an
interactive script (i.e., not completely automated so it will be
tiring to use unless we are careful with the selection of what images
to compare).  At the heart of the script, imagemagick would calculate
the normalized difference of the old and modified image results, and
then simultanously display old, modified, and difference image results
to help evaluate by visual means (as opposed to fully automated means)
whether any significant regression has occurred.  (Of course, for
those cases where the normalized difference image is zero, the script
could simply skip the visual comparison to speed up the comparison).

I have mentioned imagemagick here because I am reasonably familiar
with it, and it is one of the primary collections of software that is
used on Unix to manipulate and display images. However, you should
also note that imagemagick is available as a binary download on
Windows (see <http://www.imagemagick.org/script/binary-releases.php>).
Furthermore, even though it appears not to be specifically packaged
for Cygwin such packaging has occurred for the MinGW-w64/MSYS2
platform so if the generic windows binary download does not work,
there is that other Windows platform alternative to try as well.

Alan
__________________________
Alan W. Irwin

Astronomical research affiliation with Department of Physics and Astronomy,
University of Victoria (astrowww.phys.uvic.ca).

Programming affiliations with the FreeEOS equation-of-state
implementation for stellar interiors (freeeos.sf.net); the Time
Ephemerides project (timeephem.sf.net); PLplot scientific plotting
software package (plplot.sf.net); the libLASi project
(unifont.org/lasi); the Loads of Linux Links project (loll.sf.net);
and the Linux Brochure Project (lbproject.sf.net).
__________________________

Linux-powered Science
__________________________