gnuplot-beta

Missing/invalid data points in PM3D

[Ethan M. <merritt@u.washington.edu>]

While investigating a bug report on comp.graphics.apps.gnuplot
I discovered what appears to be a long-standing coding bug in
the data input loop of plot3d.c.   In outline, the main loop
looks like this:

  while ((j = df_readline(v,MAXDATACOLS)) != DF_EOF) {
     cp = local_this_iso->points + xdatum;

     ... lots of conditional stuff ...

     come_here_if_undefined:
        ++xdatum;
  }  /* end of whileloop - end of surface */


The tricky bit is the increment of xdatum at the end, which
is the only thing that advances the cp (current point) pointer.
Most of the conditional tests either reach the end of the while
clause naturally, or terminate prematurely via an explicit jump:
  goto come_here_if_undefined;
However, two cases terminate via a "continue" statement instead,
which means that the current point is not incremented.
In particular there is this test:

    if (j == DF_UNDEFINED || j == DF_MISSING) {
        cp->type = UNDEFINED;
        continue;
    }

Setting cp->type to UNDEFINED is useless, because cp itself is not
incremented. So during the next iteration of the while clause it is
over-written with the next data point.  This messes up the gridding
structure, and leads to the problem reported on the newsgroup.

It seems clear to me that at least for gridded surfaces the 
"continue" statement should be replaced by a jump to 
come_here_if_undefined.  That does, in fact, fix the specific 
problem reported on the newsgroup.

However, I am dubious about the affect this has on PM3D output.
I tried replacing one of the data points in triangle.dat with a
nonsense string, or with NaN, and issuing the commands

	splot 'triangle.dat' using 1:2:3 with pm3d
	splot 'triangle.dat' using ($1):($2):($3) with pm3d

The results are odd, to say the least.

Version 4.0:
  Very, very odd behavior (try rotating it!).

The current cvs version:
  1st command distorts the grid, which I would predict,
  2nd command distorts and also produces a strange coloring.

Current cvs patched to increment xdatum before continuing:
  1st command distorts the grid (but why?)
  2nd command has no grid distortion, but loses the entire rest of
      the grid line after the invalid point

No version of gnuplot that I have here acts as I would expect.

What is supposed to happen if pm3d sees an invalid point?
Leave a hole in the surface?
Interpolate the color from the surrounding blocks?
Error exit?
Something else?   (see note)


Note:
I originally understood the documentation under 
"help splot datafile" to mean that in case the 3rd value on the
input line was unusable, the previous z value was used
("last value"). But upon rereading I am not sure that was the
intent. It says:

  If two or four values are provided, `gnuplot` uses the last
  value for calculating the color in pm3d plots.

-- 
Ethan A Merritt
Biomolecular Structure Center
University of Washington, Seattle WA

Re: Missing/invalid data points in PM3D

[Petr M. <mi...@ph...>]

>    if (j == DF_UNDEFINED || j == DF_MISSING) {
>        cp->type = UNDEFINED;
>        continue;
>    }
>
> It seems clear to me that at least for gridded surfaces the
> "continue" statement should be replaced by a jump to
> come_here_if_undefined.  That does, in fact, fix the specific
> problem reported on the newsgroup.
>
> However, I am dubious about the affect this has on PM3D output.
> I tried replacing one of the data points in triangle.dat with a
> nonsense string, or with NaN, and issuing the commands
>
> 	splot 'triangle.dat' using 1:2:3 with pm3d
> 	splot 'triangle.dat' using ($1):($2):($3) with pm3d
>
> The results are odd, to say the least.
>
> What is supposed to happen if pm3d sees an invalid point?

You mean handling of points marked as UNDEFINED in routine pm3d_plot().

If any of the 4 corners to be drawn as a color quadrangle is undefined, then 
its drawing is skipped. It is the "continue" in the last occurence of 
"UNDEFINED" string in pm3d.c.

In pm3d.c, if you uncomment these blocks:
   /* debugging: print scan_array */
   /* debugging: this loop prints properties of all scans */

then all the scans&points gnuplot passes into the pm3d draw routine will be 
printed.

I see that in both cases of a NaN-changed triangle.dat:
  	splot 'triangle.dat' using 1:2:3 with pm3d
  	splot 'triangle.dat' using ($1):($2):($3) with pm3d

no undefined point comes into pm3d_plot() -- thus, they are missing. Then 
the scan with a NaN is 1 point shorter and thus qudrangles instead of 
rectangles for a grid topology are drawn behind that.

With your patch of plot3d.c, all scans have the number of points as 
intended. The rectangle(s) contaning a point with NaN are not drawn. This 
looks like to be what is expected, so I propose you commit the plot3d.c 
patch.

---
PM

Re: Missing/invalid data points in PM3D

[Petr M. <mi...@ph...>]

ad 'help splot datafile':

  Data file organization is essentially the same as for `plot`, except that
  each point is an (x,y,z) triple.  If only a single value is provided, it
  will be used for z, the datablock number will be used for y, and the index
  of the data point in the datablock will be used for x.  If two or four
  values are provided, `gnuplot` uses the last value for calculating the color in
  pm3d plots.  Three values are interpreted as an (x,y,z) triple.  Additional

It seems that in this case of 2 and 4 columns, the last column is the color 
column, and the x and y coordinates are autoindexed.

A trial to improve the docs:

  If two or four values are provided, the last value is used
  for calculating the color in explicit `pm3d` plots.

Further -- what about changed the gnuplot message
gnuplot> splot 't2'
                ^
    2 columns only possible with explicit pm3d style (line 4)


into more user helpful:

    2 columns only possible for 'with pm3d' or 'set pm3d explicit' (line 4)


---
PM

Re: Missing/invalid data points in PM3D

[Ethan A M. <merritt@u.washington.edu>]

On Friday 25 August 2006 12:15 am, Petr Mikulik wrote:
> > It seems clear to me that at least for gridded surfaces the
> > "continue" statement should be replaced by a jump to
> > come_here_if_undefined.  That does, in fact, fix the specific
> > problem reported on the newsgroup.
> >
> > However, I am dubious about the effect this has on PM3D output.

> With your patch of plot3d.c, all scans have the number of points as 
> intended. The rectangle(s) contaning a point with NaN are not drawn. This 
> looks like to be what is expected, so I propose you commit the plot3d.c 
> patch.

I think there is still a problem with the patched version.
I am not sure where it is, though.

Have a look at the attached PNG image compared before + after the patch.
The input data file is "triangle.dat" with two individual values
replaced by "Junk" and "NaN" respectively. 

The patched version gets it almost right, but only in the case
of "splot ... using ($1):($2):($3)"
Even so, instead of skipping a single rectangle, it omits the
entire rest of the scan line.

Any ideas where it could be going wrong?


-- 
Ethan A Merritt
Biomolecular Structure Center
University of Washington, Seattle 98195-7742

Re: Missing/invalid data points in PM3D

[Petr M. <mi...@ph...>]

>> With your patch of plot3d.c, all scans have the number of points as
>> intended. The rectangle(s) contaning a point with NaN are not drawn. This
>> looks like to be what is expected, so I propose you commit the plot3d.c
>> patch.
>
> I think there is still a problem with the patched version.
> I am not sure where it is, though.
>
> Have a look at the attached PNG image compared before + after the patch.
> The input data file is "triangle.dat" with two individual values
> replaced by "Junk" and "NaN" respectively.
>
> The patched version gets it almost right, but only in the case
> of "splot ... using ($1):($2):($3)"
> Even so, instead of skipping a single rectangle, it omits the
> entire rest of the scan line.

You should use 'set pm3d map' to see it better.

In the non-($n) case, the plot should be the same as if you remove these 
junk lines from the datafile.

I tried your patch, and it is correct.


However, there is yet another bug:

0 2 -2
1 2 -1
2 2 0
3 2 1

0 3 0
1 3 6
2 3 2
3 3 3

0 4 1
1 4 2
2 bla 3
3 4 4

set pm3d map
splot 'bla'

=> there is a strange feature, and the debug routine in the beginning of 
pm3d_plot() print the data passed into this routine:

#IsoCurve = scan nb 0, 4 points
#x y z type(in,out,undef)
0 2 -2 i
1 2 -1 i
2 2 0 i
3 2 1 i

#IsoCurve = scan nb 1, 4 points
#x y z type(in,out,undef)
0 3 0 i
1 3 6 i
2 3 2 i
3 3 3 i

#IsoCurve = scan nb 2, 4 points
#x y z type(in,out,undef)
0 4 1 i
1 4 2 i
2 2 2 i     <====== HERE IS BUG!
3 4 4 i


Thus, the reader converted the data file line
 	2 bla 3
into
 	2 2 2 i
Correctly, there should be point (like a missing line), or
 	2 999 3 u
(999 or whatever number). I think the "missing line" behaviour is compatible 
to "plot" command.

---
PM

Re: Missing/invalid data points in PM3D

[Daniel J S. <dan...@ie...>]

Petr Mikulik wrote:
>>>With your patch of plot3d.c, all scans have the number of points as
>>>intended. The rectangle(s) contaning a point with NaN are not drawn. This

This is related, I think, to what we discussed not too long ago on missing and invalid points, i.e., the patch I created illustrating the various behaviors with a new "demo".  If I recall correctly, I fixed up a couple things (but this particular goto issue doesn't ring a bell) and then reached the point where it seemed we would have to make some decisions as to how missing/invalid should behave.  I advocated a default behavior with the ability to redefine behavior.

Dan

Re: Missing/invalid data points

[Petr M. <mi...@ph...>]

I have noticed that if there is at least one blank line at the end of a 
datafile, then
         set table; plot 'bla1'
shows a dummy undefined point at the end:

#Curve 0 of 1, 5 points
#x y type
  1  10  i
  2  20  i
  2  30  i
  4  40  i
  0  0  u

That dummy "0 0 u" does not appear for
         splot 'bla1'
It probably does not hurt anything?

***

> In the non-($n) case, the plot should be the same as if you remove these
> junk lines from the datafile.

I think datafiles should not have "stupid" values, but, amazingly, gnuplot 
does not ignore such lines; try this:

0.1 10 1.111
0.2 20 2.222
0.3 bla 3.333
0.4 40 4.444

gnuplot>  set table; plot 'bla1'

#Curve 0 of 1, 4 points
#x y type
  0.1  10  i
  0.2  20  i
  2  0.3  i
  0.4  40  i

gnuplot>  set table; splot 'bla1'

#Surface 0 of 1 surfaces

#IsoCurve 0, 4 points
#x y z type
  0.1  10  1.111 i
  0.2  20  2.222 i
  2  0  0.3 i
  0.4  40  4.444 i

Even that this has no relation to "set datafile missing", the "replacement" 
values are strange -- 'help missing' even says "erroneously".

Further, the first example in 'help missing' + its description seems to be 
wrong.


---
PM

Re: Missing/invalid data points

[Ethan M. <merritt@u.washington.edu>]

On Monday 28 August 2006 12:43 pm, Petr Mikulik wrote:
> I think datafiles should not have "stupid" values, but, amazingly,
> gnuplot does not ignore such lines; try this:
>
> 0.1 10 1.111
> 0.2 20 2.222
> 0.3 bla 3.333
> 0.4 40 4.444
>
> gnuplot>  set table; plot 'bla1'
>
> #Curve 0 of 1, 4 points
> #x y type
>   0.1  10  i
>   0.2  20  i
>   2  0.3  i
>   0.4  40  i

That is exactly what is documented in the help file.
I agree it is stupid, but it has been that way since forever.
In the absence of a 'using' keyword, the program re-evaluates
the intended meaning of the columns all over again for each line
in the input file. The line
 0.2 20 2.222
is interpreted as having 3 valid columns, therefore the 1st 
column is used for x=0.2 and the second column for y=20.
The line
 0.3 bla 3.333
is interpreted as having only 1 valid column. Therefore the
1st column is interpreted as y=0.3 (*NOT X*) and the x=2 value
is taken as being implicitly determined by the input line #,
equivalent to 'column 0'.


> gnuplot>  set table; splot 'bla1'
>
> #Surface 0 of 1 surfaces
>
> #IsoCurve 0, 4 points
> #x y z type
>   0.1  10  1.111 i
>   0.2  20  2.222 i
>   2  0  0.3 i
>   0.4  40  4.444 i
>
> Even that this has no relation to "set datafile missing", the
> "replacement" values are strange -- 'help missing' even says
> "erroneously".

Again, this is exactly what is documented.
Only 1 valid number is read from the line
  0.3 bla 3.333
So that value is assumed to mean z=0.3, while the x=2 and y=0
values are generated implicitly from the line number
and the isocurve number, respectively.

> Further, the first example in 'help missing' + its description seems
> to be wrong.

It looks correct to me.  

I believe we have reached the point where we should just say
the a 'using' keyword is required.  For backwards compatibility,
in the absence of a 'using' specifier the code will behave as
the documents have always stated, and you show above. 

Therefore, odd as the behavior may be, I think we should not change
the documented behavior for no using spec.

However, I think we very definitely should fix the currently broken
behavior of
	splot 'foo' using 1:2:3
in the presence of nonsense or NaN strings
Unfortunately, I have not found a simple fix for this yet.
I can make it work, but only be substantially re-writing the 
input code in datafile.c.   The presence of the binary read
routines is an additional complication. 



-- 
Ethan A Merritt
Biomolecular Structure Center
University of Washington, Seattle WA

Re: Missing/invalid data points

[Daniel J S. <dan...@ie...>]

Ethan Merritt wrote:

> However, I think we very definitely should fix the currently broken
> behavior of
> 	splot 'foo' using 1:2:3
> in the presence of nonsense or NaN strings
> Unfortunately, I have not found a simple fix for this yet.
> I can make it work, but only be substantially re-writing the 
> input code in datafile.c.

I propose it should be a post 4.2 major redo.  Again, I suspect that not a lot of thought was put into the original behavior for this and any documentation came about as an after thought.  We'd want to keep as much compatibility as possible but I'm guessing that if people had a really nice flexible alternative they'd be happy enough.

Recall, I also raised the issue of flexibility to define how the "datum" should act when finding a "missing" point.  Some times one would like to increment, some times not.

>   The presence of the binary read
> routines is an additional complication. 

I think the binary case can be addressed after the fact, once a nice set up is done in ASCII.  We can have analogous concepts, e.g., specifying that a certain number represents "missing", or "NaN", etc.  (This is something that people often do in practice, e.g., in a data stream they might define 0x8000 of a 16 bit number to represent something special, say saturation.)

Dan

Re: Missing/invalid data points

[Petr M. <mi...@ph...>]

> However, I think we very definitely should fix the currently broken
> behavior of
> 	splot 'foo' using 1:2:3
> in the presence of nonsense or NaN strings

It seems that Inf is handled, so why not NaN?

0.1 10 1.111
0.2 20 2.222
0.3 Inf 3.333
0.4 40 4.444


set table
plot 'a'

#Curve 0 of 1, 4 points
#x y type
  0.1  10  i
  0.2  20  i
  0.3  0.3  u
  0.4  40  i

---
PM

Re: Missing/invalid data points

[<br...@ph...>]

Petr Mikulik wrote:
>> In the non-($n) case, the plot should be the same as if you remove these
>> junk lines from the datafile.
> 
> I think datafiles should not have "stupid" values, but, amazingly, gnuplot 
> does not ignore such lines; try this:
> 
> 0.1 10 1.111
> 0.2 20 2.222
> 0.3 bla 3.333
> 0.4 40 4.444
> 
> gnuplot>  set table; plot 'bla1'

You partly missed Dan's point.  By 'non-($n) case', I think he was 
referring to "plot 'bla1' u 1:2:3".  And yes, that's supposed to behave
differently from "plot 'bla1'".  The documentation even says so, right 
there in 'help using'.

> Further, the first example in 'help missing' + its description seems to be 
> wrong.

No, the example matches the 2D case of your example quite nicely.  It 
doesn't match the 3D case as well.  I suspect that's because splot never
used to have a 2-column default case back in the days when 'missing' and 
'using' still worked as documented (before pm3d, among other things), so 
there's nothing for using-less splot to fall back to, seeing a data 
record with 2 usable entries.

Re: Missing/invalid data points

[Ethan M. <merritt@u.washington.edu>]

On Monday 28 August 2006 02:52 pm, Ethan Merritt wrote:
>
> However, I think we very definitely should fix the currently broken
> behavior of
> 	splot 'foo' using 1:2:3
> in the presence of nonsense or NaN strings

Here is the essence of the problem:

# Data file junk.dat
0.1 10 1.111
0.2 20 2.222
0.3 NaN 3.333
0.4 40 4.444



gnuplot> splot 'junk.dat' using 1:2:3 with pm3d

#IsoCurve 0, 3 points
#x y z type
 0.1  10  1.111 i
 0.2  20  2.222 i
 0.4  40  4.444 i

gnuplot> splot 'junk.dat' using ($1):($2):($3) with pm3d

#IsoCurve 0, 4 points
#x y z type
 0.1  10  1.111 i
 0.2  20  2.222 i
 0  0  0 u
 0.4  40  4.444 i


The first case (using 1:2:3) omits the line altogether,
which unfortunately messes up the gridding and produces
a distorted surface.

The second case keeps the point, marking it as undefined.
That is better, but it would be better yet if the information
stored was
   0.3 NaN 3.33 u
If nothing else, that would allow the tabular output line
to match the original input line.  Beyond that, the extra
info may be of use in auto-scaling the axes, auto-gridding 
(not in this case, but if it were the Z value that was bad),
interpolation of color values, and probably things I haven't
thought of.


-- 
Ethan A Merritt
Biomolecular Structure Center
University of Washington, Seattle WA

Re: Missing/invalid data points

[<br...@ph...>]

Ethan Merritt wrote:

> The second case keeps the point, marking it as undefined.
> That is better, but it would be better yet if the information
> stored was
>    0.3 NaN 3.33 u
> If nothing else, that would allow the tabular output line
> to match the original input line.  Beyond that, the extra
> info may be of use in auto-scaling the axes, 

Hardly --- autoscaling should never react to points that aren't
actually on the plot.

But that's not the actual point.  The core issue is that there's
only *one* "undefined" flag per data point.  To use the non-NaN
data values safely, datafile.c would have to record which of them are 
usable, i.e. which caused the DF_UNDEFINED, and which didn't.  And it 
would have to continue reading / filling after a NaN or missing column.

That's a rewrite from scratch of datafile.c and a good portion of all 
the code using it you've just outlined.

Re: Missing/invalid data points

[Ethan M. <merritt@u.washington.edu>]

NB:   This discussion is with regard to possible changes
      or extensions _after_ 4.2

On Tuesday 29 August 2006 10:25 am, Hans-Bernhard Br=F6ker wrote:
> Ethan Merritt wrote:
> > The second case keeps the point, marking it as undefined.
> > That is better, but it would be better yet if the information
> > stored was
> >    0.3 NaN 3.33 u
> > If nothing else, that would allow the tabular output line
> > to match the original input line.  Beyond that, the extra
> > info may be of use in auto-scaling the axes,
>
> Hardly --- autoscaling should never react to points that aren't
> actually on the plot.

I am not so sure.  Consider the "using 1:2:3 with pm3d" plots that
are being discussed.  Because of the oddities in missing/NaN
handling, the limits of the grid are not determined correctly.=20
The coordinate information may be necessary for gridding, even
if some of the points were unmeasured ('missing') or mis-measured
(NaN or Inf).

> But that's not the actual point.  The core issue is that there's
> only *one* "undefined" flag per data point.  To use the non-NaN
> data values safely, datafile.c would have to record which of them are
> usable, i.e. which caused the DF_UNDEFINED, and which didn't.  And it
> would have to continue reading / filling after a NaN or missing
> column.
>
> That's a rewrite from scratch of datafile.c and a good portion of all
> the code using it you've just outlined.

Nevertheless, that is what I am proposing.
I already said up-thread that this requires changes throughout
datafile.c

However, it is not quite as bad as you make it sound.
The change can be incremental.=20

1) The current code in datafile.c is prepared to fill in all the
relevant columns for return to the caller.
The issue is whether it bails out before doing so, or after doing so.
We can first change it to fill in all columns possible before
returning.  The caller will still see the DF_UNDEFINED return code,
and will continue to behave as before.

2) Individual callers, for example the gridding code, can then be
taught to use the additional information that is passed back to them.

By the way, the histogram code also suffers from a similar problem.
The current code contains work-arounds to try to handle the problem,
but a cleaner protocol for returning information from get_data()=20
would allow cleanup in the higher-level histogram code, and=20
additional flexibility in specifying the x-coordinate for histogram
plot mode (the subject of past feature requests and a contributed
patch).=20

=2D-=20
Ethan A Merritt
Biomolecular Structure Center
University of Washington, Seattle WA

Re: Missing/invalid data points

[Daniel J S. <dan...@ie...>]

Ethan Merritt wrote:
> NB:   This discussion is with regard to possible changes
>       or extensions _after_ 4.2
>=20
> On Tuesday 29 August 2006 10:25 am, Hans-Bernhard Br=F6ker wrote:
>=20
>>Ethan Merritt wrote:
>>
>>>The second case keeps the point, marking it as undefined.
>>>That is better, but it would be better yet if the information
>>>stored was
>>>   0.3 NaN 3.33 u
>>>If nothing else, that would allow the tabular output line
>>>to match the original input line.  Beyond that, the extra
>>>info may be of use in auto-scaling the axes,
>>
>>Hardly --- autoscaling should never react to points that aren't
>>actually on the plot.
>=20
>=20
> I am not so sure.  Consider the "using 1:2:3 with pm3d" plots that
> are being discussed.  Because of the oddities in missing/NaN
> handling, the limits of the grid are not determined correctly.=20
> The coordinate information may be necessary for gridding, even
> if some of the points were unmeasured ('missing') or mis-measured
> (NaN or Inf).
>=20
>=20
>>But that's not the actual point.  The core issue is that there's
>>only *one* "undefined" flag per data point.  To use the non-NaN
>>data values safely, datafile.c would have to record which of them are
>>usable, i.e. which caused the DF_UNDEFINED, and which didn't.  And it
>>would have to continue reading / filling after a NaN or missing
>>column.
>>
>>That's a rewrite from scratch of datafile.c and a good portion of all
>>the code using it you've just outlined.
>=20
>=20
> Nevertheless, that is what I am proposing.
> I already said up-thread that this requires changes throughout
> datafile.c
>=20
> However, it is not quite as bad as you make it sound.
> The change can be incremental.=20

I think so too.  The code may look a bit messy, but we've already narrowe=
d down the means for getting data from a file to df_readline() in plot2d.=
c and plot3d.c--in the case of the new binary data file code.  If we beco=
me comfortable enough to accept the binary data file code, which haas bee=
n running for quite a while now, that eliminates "df_3dmatrix()" and make=
s things a little cleaner.

I don't think what Ethan proposed should be too bad.  (Reaching agreement=
 on behavior is a different matter.)

Dan

Re: Missing/invalid data points

[<br...@ph...>]

Petr Mikulik wrote:
> It seems that Inf is handled, so why not NaN?

Because NaN is worse than Inf.  Infinity doesn't have all the properties 
of a number, but still some of them.  In particular +Inf is larger than 
any other number, -Inf is smaller than any other.

So Inf is generally just out-of-range.   A NaN value, OTOH, is 
completely undefined.  It's not even equal to itself!

> #x y type
>   0.1  10  i
>   0.2  20  i
>   0.3  0.3  u
>   0.4  40  i

Arguably, there's a bug in this result.  The Inf could have
have been stored and marked as "out of range" ('o' instead of 'u').