Menu
▾
▴
gnuplot-beta
|
From: Daniel J S. <dan...@ie...> - 2006-06-18 18:10:11
|
There are about four bug reports related to handling of MISSING and/or UNDEFINED data points in a file. I think the patch I sent yesterday is close to a solution. Ethan, could you please have a look at that and maybe we can clear out several bug reports with one patch. Dan |
|
From: Ethan A M. <merritt@u.washington.edu> - 2006-06-18 18:36:58
|
On Sunday 18 June 2006 11:19 am, Daniel J Sebald wrote: > There are about four bug reports related to handling of > MISSING and/or UNDEFINED data points in a file. This was explicitly listed in my original summary of "What's left for a 4.2 release?". > I think the patch I sent yesterday is close to a solution. The main problem is that there is no clear agreement on what the behavior is *supposed* to be. Without such a statement, it's pointless to produce patches. Version 4.0 does not behave as the docs state. Is this a bug in the code, or in the docs? Which do we want to fix? Disregarding what the docs say, is the handling of missing/undefined/garbage data consistent in all code paths? FWIW, my own feeling is that a data line containing a field marked MISSING (i.e. it matches the string set by `set datafile missing "X"`) should be treated as if the entire line were not there at all, save maybe for incrementing the line count. This is not necessarily the same as treating it as "undefined" or "garbage in the input field". In other words, I think the description in the docs should be changed. That is not to say that the code is bug-free, however. > Ethan, could you please have a look at that and maybe > we can clear out several bug reports with one patch. That has been the idea since the beginning. But first I'd like to see if the "one patch" can be a patch to gnuplot.doc, or whether we need to patch the code anyway. -- Ethan A Merritt Biomolecular Structure Center University of Washington, Seattle 98195-7742 |
|
From: Daniel J S. <dan...@ie...> - 2006-06-18 19:19:43
|
Ethan A Merritt wrote:
> On Sunday 18 June 2006 11:19 am, Daniel J Sebald wrote:
>
>>There are about four bug reports related to handling of
>>MISSING and/or UNDEFINED data points in a file.
>
>
> This was explicitly listed in my original summary of
> "What's left for a 4.2 release?".
>
>
>>I think the patch I sent yesterday is close to a solution.
>
>
> The main problem is that there is no clear agreement on
> what the behavior is *supposed* to be. Without such a
> statement, it's pointless to produce patches.
I agree with that, and that's why I clarified the way I configured the patch. Defining what the behavior should be is the big issue, so let's have the list do that in the next day or two.
Currently the documentation does not specify all conditions, I think.
>
> Version 4.0 does not behave as the docs state.
> Is this a bug in the code, or in the docs?
> Which do we want to fix?
>
> Disregarding what the docs say, is the handling of
> missing/undefined/garbage data consistent in all code paths?
>
> FWIW, my own feeling is that a data line containing a field marked
> MISSING (i.e. it matches the string set by `set datafile missing "X"`)
> should be treated as if the entire line were not there at all,
You mean "line", as in line of ascii text in the file, I assume.
> save maybe for incrementing the line count.
Hmm, good question, because there is the issue of using the datum number as one of the plotting variables. In some cases this is desirable (e.g., the geographic statistics on population or somethere simply where there might be a city where information is not available) in others not (e.g., a function relationship).
> This is not necessarily
> the same as treating it as "undefined" or "garbage in the input field".
> In other words, I think the description in the docs should be changed.
> That is not to say that the code is bug-free, however.
I don't think it is bug free. It looks pretty likely that plot3d.c does not behave the same as plot2d.c.
(***) I think one conceptual or philosophical thing we should decide is whether datafile.c can simply toss out an ascii line that is data (i.e., not comment line, or blank line, etc.; those it can ignore in my opinion) or whether it should always go back to the calling code and indicate either DF_MISSING, DF_UNDEFINED or DF_BAD (which looks like something you've added at some point).
My feeling is that good programming practice would be to always go back to the calling routine with some indication of whether this is a DF_MISSING, DF_UNDEFINED or DF_BAD point and let the calling routine handle it. Now, this is a philosophical issue of having to repeat code vs. the flexibility to have the calling routine behave accordingly if it needs to. I'm going to go with more flexibility on this one, for no good reason I guess other than future modifications.
...
Perhaps we need more than one character definition. Maybe we need
set datafile missing "NA"
set datafile undefined "NaN"
Could the difference between these two be that in the case of finding the missing string the line count is not incremented? In the case of an undefined string it is? The two can't be the same, of course.
...
Here is how it is for the patch in 2D
plot '-'
1 10
2 20
3 ?
4 40
5 50
e
will treat the third point as (2,3) whether "missing" is properly set or not. This line count is used as one variable.
But how about for a single column?
gnuplot> plot '-'
input data ('e' ends) > 10
input data ('e' ends) > 20
input data ('e' ends) > ?
^
Bad data on line 3
Ooop, that happens when missing is not defined AND when it is defined. I would say that it should be ignored in the case of missing being defined... so there is a bug right there even with the patch in my mind.
Let me stop here. I think the two column behavior with MISSING works in a logical way, but with there being issues I haven't accounted for already for single column data I can see that the approach of discussing the whole thing first isn't going to be the best approach.
How about we do the following? Let's have a discussion on the topic (***) above about whether we should always return to the calling code as opposed to using the "line_okay" continue method for data.
And let's put together a demo illustrating how MISSING and UNDEFINED works for all the various situations. We'll move that demo into CVS right away and then people on the list can download, compile and THEN get into a discussion about what behavior should be. The demo will be plotted examples with "linespoints" set illustrating side-by-side when missing is properly defined and not. One column and two column data for 2D plots, multiple column data for 3D plots.
Should I put together a demo?
Dan
|
|
From: Ethan A M. <merritt@u.washington.edu> - 2006-06-18 22:15:20
|
On Sunday 18 June 2006 12:28 pm, you wrote:
> My feeling is that good programming practice would be to always go
> back to the calling routine with some indication of whether this is
> a DF_MISSING, DF_UNDEFINED or DF_BAD point and let the calling
> routine handle it.
MISSING and UNDEFINED are indeed returned to the caller, but so far
the callers treat them the same way (plot2d.cline 446, plot3d.c line 688).
> Here is how it is for the patch in 2D
>
> plot '-'
> 1 10
> 2 20
> 3 ?
> 4 40
> 5 50
> e
>
> will treat the third point as (2,3) whether "missing" is properly
> set or not.
I consider that a bug. But it's been documented as behaving like that
since forever, so I suppose we may be stuck with it. To me it makes
no sense to interpret the different lines of input as having different
formats. I.e. if you read 2 values, X and Y, from lines 1 and 2 then
it's crazy to suddenly switch modes and interpret line 3 as having
an implicit X value and Y in column 1. But this is a digression.
> But how about for a single column?
>
> gnuplot> plot '-'
> input data ('e' ends) > 10
> input data ('e' ends) > 20
> input data ('e' ends) > ?
> ^
> Bad data on line 3
>
> Ooop, that happens when missing is not defined AND when it is defined.
It works properly here. Are you sure you correctly set missing?
> I would say that it should be ignored in the case of missing being defined...
> so there is a bug right there even with the patch in my mind.
Um. I see no bug. It works exactly as you say it should.
> Should I put together a demo?
I'm less interested in a demo than in a re-written section for the
docs. Once we have agreed on a statement of what is supposed to
happen, then we can do any required bug fixing or demo-writing.
--
Ethan A Merritt
Biomolecular Structure Center
University of Washington, Seattle 98195-7742
|
|
From: Daniel J S. <dan...@ie...> - 2006-06-18 22:31:55
|
Ethan A Merritt wrote:
> On Sunday 18 June 2006 12:28 pm, you wrote:
>
>>My feeling is that good programming practice would be to always go
>>back to the calling routine with some indication of whether this is
>>a DF_MISSING, DF_UNDEFINED or DF_BAD point and let the calling
>>routine handle it.
>
>
> MISSING and UNDEFINED are indeed returned to the caller, but so far
> the callers treat them the same way (plot2d.cline 446, plot3d.c line 688).
Yes and no. Yes, currently treated the same way. (I'm arguing they shouldn't be.) No, in some instances df_readascii() will decide "this is missing data, so I'm going to go back to the top and read another line of data". It does this via "line_okay". I say it should always return something if a non-comment is found.
>>will treat the third point as (2,3) whether "missing" is properly
>>set or not.
>
>
> I consider that a bug.
I'm in agreement with you.
> But it's been documented as behaving like that
> since forever, so I suppose we may be stuck with it.
It is, but as far as being stuck with it. I think the documentation was one of describing how it just happens to work because it was dealt with specifically. I see no use to the way that behaves so why have such behavior? (I know I'm a bit more willing to toss precedent to the wind than others.)
> To me it makes
> no sense to interpret the different lines of input as having different
> formats. I.e. if you read 2 values, X and Y, from lines 1 and 2 then
> it's crazy to suddenly switch modes and interpret line 3 as having
> an implicit X value and Y in column 1. But this is a digression.
Exactly. I don't think it is a digression.
>>But how about for a single column?
>>
>>gnuplot> plot '-'
>>input data ('e' ends) > 10
>>input data ('e' ends) > 20
>>input data ('e' ends) > ?
>> ^
>> Bad data on line 3
>>
>>Ooop, that happens when missing is not defined AND when it is defined.
>
>
> It works properly here. Are you sure you correctly set missing?
In the patch I had an extra unnecessary test condition.
>>Should I put together a demo?
>
>
> I'm less interested in a demo than in a re-written section for the
> docs. Once we have agreed on a statement of what is supposed to
> happen, then we can do any required bug fixing or demo-writing.
I just about have one. I think it is worthwhile to get a discussion going and will serve as a tutorial to the user so we can prevent such bug reports in the future. Ten minutes...
Dan
|
|
From: Daniel J S. <dan...@ie...> - 2006-06-18 22:51:54
Attachments:
missing1.png
missing2.png
|
Daniel J Sebald wrote: > I just about have one. I think it is worthwhile to get a discussion going and will serve as a tutorial to the user so we can prevent such bug reports in the future. Ten minutes... I placed the patch on S.F. under a new patch entry. I like these sorts of patches because they are tutorial and a good check some times for when bugs are introduced. We've caught several that way in the past. I still am wondering if there is a purpose to having both a datafile missing string and datafile undefined string. Somehow it seems to me the user could have both such items at the same time and want there to be different behavior based upon type. Anyway, attached are the PNG images for those not tuned in to the discussion (AFTER THE PATCH). We can see what Ethan is talking about for the example where the line dips down to (3,2) in the two column data case. I'd like to add an example or two for the 3D case so we can modify and test proper behavior, but don't want to have so many data points inside the demo file. Dan |
|
From: Daniel J S. <dan...@ie...> - 2006-06-18 23:13:38
|
Daniel J Sebald wrote: > I still am wondering if there is a purpose to having both a datafile > missing string and datafile undefined string. Somehow it seems to me > the user could have both such items at the same time and want there to > be different behavior based upon type. Oh yeah, that question and also, if you look at the PNG's of the previous email, there is the question of a the line number (or datum) being incremented even when the data point is missing. Are there some situation where that is undesirable? For example, if you look at some of the histogram examples in all.dem, you might imagine a table like "day of week" "number of coots on lake" "Monday" 127 "Tuesday" N/A "Wednesday" 35 "Thursday" 42 "Friday" 73 where the user may not want an empty blank space between the first and second histogram element. (In some sense this illustrates the reason for df_readascii() to always return something if it finds a data point rather than ignoring DF_MISSING. So the higher level routines can have alternative behavior in the future.) Dan |
|
From: Daniel J S. <dan...@ie...> - 2006-06-19 01:24:28
|
Daniel J Sebald wrote: > I still am wondering if there is a purpose to having both a datafile > missing string and datafile undefined string. Somehow it seems to me > the user could have both such items at the same time and want there to > be different behavior based upon type. As I looked at the right side graphs in the PNGs of a couple emails back, I began to wonder even about the behavior of passing a DF_MISSING point through an action table script. Why should a point go from type DF_MISSING to DF_UNDEFINED just because it goes through a function first? I doubt anyone desires or expects that. I hear the compatibility argument, but sometimes improving what looks to have been consequential behavior is rather tempting. I propose there should be two types of classification strings set datafile missing "" set datafile undefined "" If the df_readascii() routine finds a missing data string such as "NA", return DF_MISSING. If it finds an undefined string such as NaN, return DF_UNDEFINED. If the data point is valid but passed through the using function and then becomes undefined then return, what?, DF_UNDEFINED or DF_BAD? These definitions all depend on meaningful interpretation on gnuplot's part. If there are meaningful actions to take based upon whether the data is missing or undefined (and both can be present in the same dataset) then that should be the driving factor. I can't think of any good examples right now, but somehow I think there is meaningful distinction that there could be missing and undefined data points at the same time. I think that then there might be good reason to eventually have some options for how to utilize the classification in the plots. This will sound strange, but maybe there should be a way to configure missing points as undefined OR undefined points as missing (but not interchange both because one could simply redefine to achieve that). That would be in addition to specifications of how to utilize such points in the graph... well, on second thought, if the two classes can be configure in similar ways there really is no need then to have "treat missing as undefined", etc. Anyway, my feeling is that the behavior of the data can be made much more useful without too much pain and should be before 4.2. I can see a missing.dem demo with all kinds of plots illustrating nice handling of missing and undefined data. I really do think that complaints will be few when changing behavior that looks like it wasn't so much planned as it was simply explained. Useful, controllable plotting will assuage any complaints. Dan |
|
From: Daniel J S. <dan...@ie...> - 2006-06-19 03:44:15
|
Just brainstorming here:
> I propose there should be two types of classification strings
>
> set datafile missing ""
> set datafile undefined ""
Syntax:
set datafile missing {"<string>"} {ignore | increment}
{connect | break}
show datafile missing
unset datafile
where "ignore" means to ignore any point containing the string "<string>", "inc" means to increment any datum fields (useful for leaving spaces in histograms); "connect" means to draw over the point for continuous styles as though the point wasn't present connecting points on either side(s), "break" means to not connect any points present on either side(s) (useful for placing discontinuities in lines). The default for "missing" is ignore/connect.
The syntax for "undefined" would be similar except the default would be increment/break.
You've probably noticed that there is no difference here other than titles "missing" and "undefined"; we'd just be providing two flags for the user to control. So really, an alternative would simply be to have a general "flag", e.g., "set datafile flag "NaN" increment break" for which there could be multiple flags. But, in some sense I kind of like the missing/datafile in that it provides the flexibility but provides just a tad of assumptions (the defaults) to help the user.
Dan
|
|
From: <br...@ph...> - 2006-06-20 22:19:36
|
Ethan A Merritt wrote: > Version 4.0 does not behave as the docs state. > Is this a bug in the code, or in the docs? Could be both. In case of doubt, it could be that we have to pick between the last version of gnuplot where code and docs still agreed to some sensible level, on one hand, and plain common sense on the other. > Disregarding what the docs say, is the handling of > missing/undefined/garbage data consistent in all code paths? I would be surprised if it were. The code's internal consistency has degraded substantially over time. |
