Re: Missing/invalid data points

[Ethan M. <merritt@u.washington.edu>]

NB:   This discussion is with regard to possible changes
      or extensions _after_ 4.2

On Tuesday 29 August 2006 10:25 am, Hans-Bernhard Br=F6ker wrote:
> Ethan Merritt wrote:
> > The second case keeps the point, marking it as undefined.
> > That is better, but it would be better yet if the information
> > stored was
> >    0.3 NaN 3.33 u
> > If nothing else, that would allow the tabular output line
> > to match the original input line.  Beyond that, the extra
> > info may be of use in auto-scaling the axes,
>
> Hardly --- autoscaling should never react to points that aren't
> actually on the plot.

I am not so sure.  Consider the "using 1:2:3 with pm3d" plots that
are being discussed.  Because of the oddities in missing/NaN
handling, the limits of the grid are not determined correctly.=20
The coordinate information may be necessary for gridding, even
if some of the points were unmeasured ('missing') or mis-measured
(NaN or Inf).

> But that's not the actual point.  The core issue is that there's
> only *one* "undefined" flag per data point.  To use the non-NaN
> data values safely, datafile.c would have to record which of them are
> usable, i.e. which caused the DF_UNDEFINED, and which didn't.  And it
> would have to continue reading / filling after a NaN or missing
> column.
>
> That's a rewrite from scratch of datafile.c and a good portion of all
> the code using it you've just outlined.

Nevertheless, that is what I am proposing.
I already said up-thread that this requires changes throughout
datafile.c

However, it is not quite as bad as you make it sound.
The change can be incremental.=20

1) The current code in datafile.c is prepared to fill in all the
relevant columns for return to the caller.
The issue is whether it bails out before doing so, or after doing so.
We can first change it to fill in all columns possible before
returning.  The caller will still see the DF_UNDEFINED return code,
and will continue to behave as before.

2) Individual callers, for example the gridding code, can then be
taught to use the additional information that is passed back to them.

By the way, the histogram code also suffers from a similar problem.
The current code contains work-arounds to try to handle the problem,
but a cleaner protocol for returning information from get_data()=20
would allow cleanup in the higher-level histogram code, and=20
additional flexibility in specifying the x-coordinate for histogram
plot mode (the subject of past feature requests and a contributed
patch).=20

=2D-=20
Ethan A Merritt
Biomolecular Structure Center
University of Washington, Seattle WA