Re: MISSING and UNDEFINED

[Daniel J S. <dan...@ie...>]

Ethan A Merritt wrote:
> On Sunday 18 June 2006 11:19 am, Daniel J Sebald wrote:
> 
>>There are about four bug reports related to handling of 
>>MISSING and/or UNDEFINED data points in a file.
> 
> 
> This was explicitly listed in my original summary of 
> "What's left for a 4.2 release?".
> 
> 
>>I think the patch I sent yesterday is close to a solution.
> 
> 
> The main problem is that there is no clear agreement on
> what the behavior is *supposed* to be.  Without such a 
> statement, it's pointless to produce patches.

I agree with that, and that's why I clarified the way I configured the patch.  Defining what the behavior should be is the big issue, so let's have the list do that in the next day or two.

Currently the documentation does not specify all conditions, I think.

> 
> Version 4.0 does not behave as the docs state.
> Is this a bug in the code, or in the docs?
> Which do we want to fix?
> 
> Disregarding what the docs say, is the handling of
> missing/undefined/garbage data consistent in all code paths?
> 
> FWIW, my own feeling is that a data line containing a field marked
> MISSING  (i.e. it matches the string set by `set datafile missing "X"`)
> should be treated as if the entire line were not there at all, 

You mean "line", as in line of ascii text in the file, I assume.

> save maybe for incrementing the line count.

Hmm, good question, because there is the issue of using the datum number as one of the plotting variables.  In some cases this is desirable (e.g., the geographic statistics on population or somethere simply where there might be a city where information is not available) in others not (e.g., a function relationship).

>  This is not necessarily
> the same as treating it as "undefined" or "garbage in the input field".
> In other words, I think the description in the docs should be changed.
> That is not to say that the code is bug-free, however. 

I don't think it is bug free.  It looks pretty likely that plot3d.c does not behave the same as plot2d.c.

(***) I think one conceptual or philosophical thing we should decide is whether datafile.c can simply toss out an ascii line that is data (i.e., not comment line, or blank line, etc.; those it can ignore in my opinion) or whether it should always go back to the calling code and indicate either DF_MISSING, DF_UNDEFINED or DF_BAD (which looks like something you've added at some point).

My feeling is that good programming practice would be to always go back to the calling routine with some indication of whether this is a DF_MISSING, DF_UNDEFINED or DF_BAD point and let the calling routine handle it.  Now, this is a philosophical issue of having to repeat code vs. the flexibility to have the calling routine behave accordingly if it needs to.  I'm going to go with more flexibility on this one, for no good reason I guess other than future modifications.

...

Perhaps we need more than one character definition.  Maybe we need

set datafile missing "NA"
set datafile undefined "NaN"

Could the difference between these two be that in the case of finding the missing string the line count is not incremented?  In the case of an undefined string it is?  The two can't be the same, of course.

...

Here is how it is for the patch in 2D

plot '-'
1 10
2 20
3 ?
4 40
5 50
e

will treat the third point as (2,3) whether "missing" is properly set or not.  This line count is used as one variable.

But how about for a single column?

gnuplot> plot '-'
input data ('e' ends) > 10
input data ('e' ends) > 20
input data ('e' ends) > ?
              ^
         Bad data on line 3

Ooop, that happens when missing is not defined AND when it is defined.  I would say that it should be ignored in the case of missing being defined... so there is a bug right there even with the patch in my mind.

Let me stop here.  I think the two column behavior with MISSING works in a logical way, but with there being issues I haven't accounted for already for single column data I can see that the approach of discussing the whole thing first isn't going to be the best approach.

How about we do the following?  Let's have a discussion on the topic (***) above about whether we should always return to the calling code as opposed to using the "line_okay" continue method for data.

And let's put together a demo illustrating how MISSING and UNDEFINED works for all the various situations.  We'll move that demo into CVS right away and then people on the list can download, compile and THEN get into a discussion about what behavior should be.  The demo will be plotted examples with "linespoints" set illustrating side-by-side when missing is properly defined and not.  One column and two column data for 2D plots, multiple column data for 3D plots.

Should I put together a demo?

Dan