Re: conflict of separator and missing

[Ethan A M. <sf...@us...>]

On Tuesday, 14 June, 2016 23:00:19 pl...@pi... wrote:
> On 14/06/16 21:00, Ethan A Merritt wrote:
> > On Tuesday, 14 June, 2016 13:10:13 pl...@pi... wrote:
> >> On 14/06/16 12:52, Allin Cottrell wrote:
> >>> On Tue, 14 Jun 2016, pl...@pi... wrote:
> >>>
> >>>> On 14/06/16 09:43, pl...@pi... wrote:
> >>>>>
> >>>>> I have a data csv datafile which uses -99.99 as it missing data value.
> >>>>>
> >>>>> with the following settings the 'missing' data are getting plotted.
> >>>>>
> >>>>>   set datafile separator ","
> >>>>>   set datafile missing "-99.99"
> >>>>>
> >>>>>
> >>>>>   show datafile missing
> >>>>>
> >>>>>          "-99.99" in datafile is interpreted as missing value
> >>>>>
> >
> > I think you are correct that there is a bug in this part of the code.
> > The full-length string from 'set missing' is tested against the
> > start of the field contents (after removing leading whitespace);
> > then the subsequent character is tested to see if it is whitespace
> > rather than a continuation of whatever string is in the field.
> > So it works with a tab-separated *.csv file because <tab> counts
> > as whitespace, but fails with a comma-separated file because the
> > comma is mis-interpreted as part of the field content.
> 
> Thanks Ethan,
> 
> First comment: a tab separated file is not a CSV file. The C mean comma 
> separated.

In practice this is not true.  Pretty much any program I know of that
supports *.csv files allows you to specify what character is used as
a field separator.

Quoting Wikipedia:

 "the term "CSV" also denotes some closely related delimiter-separated
  formats that use different field delimiters. These include tab-separated
  values and space-separated values. A delimiter that is not present in
  the field data (such as tab) keeps the format parsing simple.
  These alternate delimiter-separated files are often even given a 
  .csv extension, despite the use of a non-comma field separator."

> I would suggest that the correct, structured way to do this is to break 
> into fields using the current field separator, then test whether any 
> fields match the missing string ( with the white-space caveats ).
> 
> If I follow your explanation, it would seem that currently the whole 
> line is scanned for the 'missing' string before it is split into fields, 
> or it is being parsed twice.

Not quite.  The input line is scanned for field separators, the start
of each field is noted, then it goes back to process them one-by-one.

> 
> 2, 2, ignore A, 2
> 3, 3, ignore B, 3
> 4, 4, ignore, 4
> 
> 
> IMO 2 and 3 should not match since the field is not equal to the 
> 'missing'  string but simply contains it. This sounds like asking for 
> trouble. Allowing white space seems sensible flexibility on insisting on 
> an exact match since it is often added for human readability, as is the 
> case here.
> 
> Only something which IS the 'missing' string or the string with leading 
> and/or trailing white-space should match, IMO.

The conventional indication of missing data in a *.csv file is simply
an empty field.  This obviously is not possible in a whitespace-separated
file.  Gnuplot's use of "set missing" is outside any standard practice
I know of for csv files, so anything we choose is likely to strike
someone as wrong.

For instance, RFC-4180, the closest thing to a csv standard, states that
"any field may be quoted with double quotes".  So in the example above,
should we ignore this line?
  5, 5, "ignore", 5
This one?
  5, 5, " ignore ", 5


	Ethan

> 
> Peter.