Re: conflict of separator and missing

[Ethan A M. <sf...@us...>]

On Tuesday, 14 June, 2016 13:10:13 pl...@pi... wrote:
> On 14/06/16 12:52, Allin Cottrell wrote:
> > On Tue, 14 Jun 2016, pl...@pi... wrote:
> >
> >> On 14/06/16 09:43, pl...@pi... wrote:
> >>>
> >>> I have a data csv datafile which uses -99.99 as it missing data value.
> >>>
> >>> with the following settings the 'missing' data are getting plotted.
> >>>
> >>>   set datafile separator ","
> >>>   set datafile missing "-99.99"
> >>>
> >>>
> >>>   show datafile missing
> >>>
> >>>          "-99.99" in datafile is interpreted as missing value
> >>>

I think you are correct that there is a bug in this part of the code.
The full-length string from 'set missing' is tested against the 
start of the field contents (after removing leading whitespace);
then the subsequent character is tested to see if it is whitespace
rather than a continuation of whatever string is in the field.
So it works with a tab-separated *.csv file because <tab> counts
as whitespace, but fails with a comma-separated file because the 
comma is mis-interpreted as part of the field content.

This should be fixed.  
The subsequent character should be tested for 
	<next character is either whitespace or field-separator>.
Or maybe it should be 
	<next character is  field-separator (which might be whitespace)>.
I'm not sure which is correct.

The difference would matter in a case like this:

set datafile separator comma
set datafile missing "ignore"

plot '-' using 1:3
1, 1, 1, 1
2, 2, ignore A, 2
3, 3, ignore B, 3
4, 4, ignore, 4
5, 5, 5, 5
e

In current gnuplot lines 2 and 3 will be treated as missing
but line 4 will not.
Should a fix result in only line 4 being ignored?
Or should all three lines be ignored?

> >>>
> >>> As a wild guess I tried the following and the missing data now get
> >>> correctly removed.
> >>>
> >>>   set datafile missing "-99.99,"
> >>>
> >>>
> >>> This seems to be an illogical order of parsing.

That will only work if there is whitespace following the comma.
So it's not a guaranteed work-around.

	Ethan


> >>>
> >>> Surely the data line needs to be parsed into its constituent data
> >>> columns before trying to detect the missing data string.
> >>>
> >>> Regards, Peter
> >>>
> >>>
> >>> gnuplot> show version
> >>>
> >>>          G N U P L O T
> >>>          Version 5.0 patchlevel 1    last modified 2015-06-07
> >>>
> >>>          Copyright (C) 1986-1993, 1998, 2004, 2007-2015
> >>>          Thomas Williams, Colin Kelley and many others
> >>>
> >>>          gnuplot home:     http://www.gnuplot.info
> >>>          faq, bugs, etc:   type "help FAQ"
> >>>          immediate help:   type "help"  (plot window: hit 'h')
> >>>
> >>>
> >>>
> >>
> >> Just to complete this here is a sample line from the file displaying
> >> this bug.
> >>
> >> 1958, 06,   21351, 1958.4548,   -99.99,   -99.99,     317.25,   315.14,
> >>    317.25,   315.14
> >
> > Doesn't it invite undefined behavior if you set "," as separator but
> > then also include spaces between the values?
> >
> > Allin Cottrell
> >
> 
> I'm not including anything, I have some data provided that I need to 
> plot with gnuplot.
> I've always found gnuplot smart enough to deal with most things that 
> I've thrown at it. If this is not a bug I could always preprocess the 
> data to remove the commas.
> 
> The question remains as to whether this is a bug or not.
> 
> The fourth field in that line is "  -99.99" when using comma separator.
> 
> This data is supplied with a missing VALUE of -99.99 . Will this match 
> gnuplot's datafile missing defined as a string "-99.99"  ?  That depends 
> upon how the equality test is done in a language that has fuzzy 
> variable types.
> 
> But that does not account for the behaviour of it working with datafile 
> missing set to "-99.99,"
> 
> That seems to clearly indicate that there is logical problem here. The 
> comma should no longer be there in the data field since it is the field 
> separator.
> 
> Peter.
> 
> 
> 
> 
> ------------------------------------------------------------------------------
> What NetFlow Analyzer can do for you? Monitors network bandwidth and traffic
> patterns at an interface-level. Reveals which users, apps, and protocols are 
> consuming the most bandwidth. Provides multi-vendor support for NetFlow, 
> J-Flow, sFlow and other flows. Make informed decisions using capacity 
> planning reports. https://ad.doubleclick.net/ddm/clk/305295220;132659582;e
> _______________________________________________
> gnuplot-beta mailing list
> gnu...@li...
> Membership management via: https://lists.sourceforge.net/lists/listinfo/gnuplot-beta