Re: conflict of separator and missing

[<pl...@pi...>]

On 14/06/16 21:00, Ethan A Merritt wrote:
> On Tuesday, 14 June, 2016 13:10:13 pl...@pi... wrote:
>> On 14/06/16 12:52, Allin Cottrell wrote:
>>> On Tue, 14 Jun 2016, pl...@pi... wrote:
>>>
>>>> On 14/06/16 09:43, pl...@pi... wrote:
>>>>>
>>>>> I have a data csv datafile which uses -99.99 as it missing data value.
>>>>>
>>>>> with the following settings the 'missing' data are getting plotted.
>>>>>
>>>>>   set datafile separator ","
>>>>>   set datafile missing "-99.99"
>>>>>
>>>>>
>>>>>   show datafile missing
>>>>>
>>>>>          "-99.99" in datafile is interpreted as missing value
>>>>>
>
> I think you are correct that there is a bug in this part of the code.
> The full-length string from 'set missing' is tested against the
> start of the field contents (after removing leading whitespace);
> then the subsequent character is tested to see if it is whitespace
> rather than a continuation of whatever string is in the field.
> So it works with a tab-separated *.csv file because <tab> counts
> as whitespace, but fails with a comma-separated file because the
> comma is mis-interpreted as part of the field content.

Thanks Ethan,

First comment: a tab separated file is not a CSV file. The C mean comma 
separated.

Your explanation seems to confirm my intuitive guess about how this was 
being processed. There should never be a question of the 'missing' test 
seeing the following comma since it is not the content of a field and I 
think that is the origin of the bug.

I would suggest that the correct, structured way to do this is to break 
into fields using the current field separator, then test whether any 
fields match the missing string ( with the white-space caveats ).

If I follow your explanation, it would seem that currently the whole 
line is scanned for the 'missing' string before it is split into fields, 
or it is being parsed twice.

It seems logical that the line be split into fields before trying to 
test the value of any field for any condition. This appears not to be 
the case at the moment.


>
> This should be fixed.
> The subsequent character should be tested for
> 	<next character is either whitespace or field-separator>.
> Or maybe it should be
> 	<next character is  field-separator (which might be whitespace)>.
> I'm not sure which is correct.
>
> The difference would matter in a case like this:
>
> set datafile separator comma
> set datafile missing "ignore"
>
> plot '-' using 1:3
> 1, 1, 1, 1
> 2, 2, ignore A, 2
> 3, 3, ignore B, 3
> 4, 4, ignore, 4
> 5, 5, 5, 5
> e
>
> In current gnuplot lines 2 and 3 will be treated as missing
> but line 4 will not.
> Should a fix result in only line 4 being ignored?
> Or should all three lines be ignored?
>
>>>>>
>>>>> As a wild guess I tried the following and the missing data now get
>>>>> correctly removed.
>>>>>
>>>>>   set datafile missing "-99.99,"
>>>>>
>>>>>
>>>>> This seems to be an illogical order of parsing.
>
> That will only work if there is whitespace following the comma.
> So it's not a guaranteed work-around.
>
> 	Ethan
>
>
>>>>>
>>>>> Surely the data line needs to be parsed into its constituent data
>>>>> columns before trying to detect the missing data string.
>>>>>
>>>>> Regards, Peter
>>>>>
>>>>>
>>>>> gnuplot> show version
>>>>>
>>>>>          G N U P L O T
>>>>>          Version 5.0 patchlevel 1    last modified 2015-06-07
>>>>>
>>>>>          Copyright (C) 1986-1993, 1998, 2004, 2007-2015
>>>>>          Thomas Williams, Colin Kelley and many others
>>>>>
>>>>>          gnuplot home:     http://www.gnuplot.info
>>>>>          faq, bugs, etc:   type "help FAQ"
>>>>>          immediate help:   type "help"  (plot window: hit 'h')
>>>>>
>>>>>
>>>>>
>>>>
>>>> Just to complete this here is a sample line from the file displaying
>>>> this bug.
>>>>
>>>> 1958, 06,   21351, 1958.4548,   -99.99,   -99.99,     317.25,   315.14,
>>>>    317.25,   315.14
>>>
>>> Doesn't it invite undefined behavior if you set "," as separator but
>>> then also include spaces between the values?
>>>
>>> Allin Cottrell
>>>
>>
>> I'm not including anything, I have some data provided that I need to
>> plot with gnuplot.
>> I've always found gnuplot smart enough to deal with most things that
>> I've thrown at it. If this is not a bug I could always preprocess the
>> data to remove the commas.
>>
>> The question remains as to whether this is a bug or not.
>>
>> The fourth field in that line is "  -99.99" when using comma separator.
>>
>> This data is supplied with a missing VALUE of -99.99 . Will this match
>> gnuplot's datafile missing defined as a string "-99.99"  ?  That depends
>> upon how the equality test is done in a language that has fuzzy
>> variable types.
>>
>> But that does not account for the behaviour of it working with datafile
>> missing set to "-99.99,"
>>
>> That seems to clearly indicate that there is logical problem here. The
>> comma should no longer be there in the data field since it is the field
>> separator.
>>
>> Peter.
>>
>>
>>

Thanks.


2, 2, ignore A, 2
3, 3, ignore B, 3
4, 4, ignore, 4


IMO 2 and 3 should not match since the field is not equal to the 
'missing'  string but simply contains it. This sounds like asking for 
trouble. Allowing white space seems sensible flexibility on insisting on 
an exact match since it is often added for human readability, as is the 
case here.

Only something which IS the 'missing' string or the string with leading 
and/or trailing white-space should match, IMO.

Peter.