Re: conflict of separator and missing

[<pl...@pi...>]

On 15/06/16 00:06, Ethan A Merritt wrote:
> On Tuesday, 14 June, 2016 23:00:19 pl...@pi... wrote:
>> On 14/06/16 21:00, Ethan A Merritt wrote:
>>> On Tuesday, 14 June, 2016 13:10:13 pl...@pi... wrote:
>>>> On 14/06/16 12:52, Allin Cottrell wrote:
>>>>> On Tue, 14 Jun 2016, pl...@pi... wrote:
>>>>>
>>>>>> On 14/06/16 09:43, pl...@pi... wrote:
>>>>>>>
>>>>>>> I have a data csv datafile which uses -99.99 as it missing data value.
>>>>>>>
>>>>>>> with the following settings the 'missing' data are getting plotted.
>>>>>>>
>>>>>>>   set datafile separator ","
>>>>>>>   set datafile missing "-99.99"
>>>>>>>
>>>>>>>
>>>>>>>   show datafile missing
>>>>>>>
>>>>>>>          "-99.99" in datafile is interpreted as missing value
>>>>>>>
>>>
>>> I think you are correct that there is a bug in this part of the code.
>>> The full-length string from 'set missing' is tested against the
>>> start of the field contents (after removing leading whitespace);
>>> then the subsequent character is tested to see if it is whitespace
>>> rather than a continuation of whatever string is in the field.
>>> So it works with a tab-separated *.csv file because <tab> counts
>>> as whitespace, but fails with a comma-separated file because the
>>> comma is mis-interpreted as part of the field content.
>>
>> Thanks Ethan,
>>
>> First comment: a tab separated file is not a CSV file. The C mean comma
>> separated.
>
> In practice this is not true.  Pretty much any program I know of that
> supports *.csv files allows you to specify what character is used as
> a field separator.
>
> Quoting Wikipedia:
>
>  "the term "CSV" also denotes some closely related delimiter-separated
>   formats that use different field delimiters. These include tab-separated
>   values and space-separated values. A delimiter that is not present in
>   the field data (such as tab) keeps the format parsing simple.
>   These alternate delimiter-separated files are often even given a
>   .csv extension, despite the use of a non-comma field separator."
>
>> I would suggest that the correct, structured way to do this is to break
>> into fields using the current field separator, then test whether any
>> fields match the missing string ( with the white-space caveats ).
>>
>> If I follow your explanation, it would seem that currently the whole
>> line is scanned for the 'missing' string before it is split into fields,
>> or it is being parsed twice.
>
> Not quite.  The input line is scanned for field separators, the start
> of each field is noted, then it goes back to process them one-by-one.
>
>>
>> 2, 2, ignore A, 2
>> 3, 3, ignore B, 3
>> 4, 4, ignore, 4
>>
>>
>> IMO 2 and 3 should not match since the field is not equal to the
>> 'missing'  string but simply contains it. This sounds like asking for
>> trouble. Allowing white space seems sensible flexibility on insisting on
>> an exact match since it is often added for human readability, as is the
>> case here.
>>
>> Only something which IS the 'missing' string or the string with leading
>> and/or trailing white-space should match, IMO.
>
> The conventional indication of missing data in a *.csv file is simply
> an empty field.  This obviously is not possible in a whitespace-separated
> file.  Gnuplot's use of "set missing" is outside any standard practice
> I know of for csv files, so anything we choose is likely to strike
> someone as wrong.
>
> For instance, RFC-4180, the closest thing to a csv standard, states that
> "any field may be quoted with double quotes".  So in the example above,
> should we ignore this line?
>   5, 5, "ignore", 5
> This one?
>   5, 5, " ignore ", 5
>
>
> 	Ethan
>
>>
>> Peter.
>
>


Ok, in the absence of any properly defined standard , where software 
like Excel ( probably the most common source of "CSV" files for a lot of 
people ) produces comma separated variables without using commas, it is 
likely to be messy.

  5, 5, "ignore", 5

This seems a bit of a contrived case, what software will quote one field 
in a line but not the others?

How would gnuplot cope with :

  "5","5", "ignore", "5"

Looking at the bug I reported may be a chance to review this whole messy 
subject but it seems like a diversion from the clear bug case.

If gnuplot scans for the position of the field separators, it should be 
stopping BEFORE it gets to the next one when testing for occurrences of 
the  'missing' string.

That seems to be a simple bug that does not open a whole can of csv worms.

Peter.