Re: conflict of separator and missing

[Tait <gnu...@t4...>]

> > The conventional indication of missing data in a *.csv file is simply
> > an empty field.  This obviously is not possible in a whitespace-separated
> > file.  Gnuplot's use of "set missing" is outside any standard practice
> > I know of for csv files, so anything we choose is likely to strike
> > someone as wrong.

I don't follow the comment in the second sentence. Empty fields in a
TSV file* are indicated by having no data in between the field
separators. Not only is it possible, but it's quite intuitive, I
think. I'm adding spaces for clarity, but those spaces wouldn't be
in the actual file:

  header1 \t header2 \t header3 \t header4
  data1 \t data2 \t data3 \t data4
  data5 \t       \t data7 \t data8
  ...

Where data6 would be, is an empty field.

(* as an aside, TSV or "tab-separated text" is the term I always see
used for tab-separated values. I've never heard of someone refer to
a tab-separated file as "CSV".)

> > For instance, RFC-4180, the closest thing to a csv standard, states that
> > "any field may be quoted with double quotes".  So in the example above,
> > should we ignore this line?
> >   5, 5, "ignore", 5
> > This one?
> >   5, 5, " ignore ", 5
> 
> Ok, in the absence of any properly defined standard , where software 
> like Excel ( probably the most common source of "CSV" files for a lot of 
> people ) produces comma separated variables without using commas, it is 
> likely to be messy.
> 
>   5, 5, "ignore", 5
> 
> This seems a bit of a contrived case, what software will quote one field 
> in a line but not the others?

Excel does exactly this. Fields are unquoted in general, but (only)
if they contain delimiter or quoting characters, then they are
quoted. If they contain quote characters, quotes are double-quoted.
Delimiter characters are not just "," for CSV, but also newlines.
Consider three rows of data, each containing two fields:

  row 1: ab    cd
  row 2: e\nf  g,h
  row 3: i"j   k<space>m

Excel will produce a CSV that looks like this:

  ab,cd
  "e
  f","g,h"
  "i""j",k m

This is obviously a contrived pathological case, but it's
illustrative of what common software "out there" might do.
Of course, backslash-escaping is also a common convention,
and for the same input, it might produce a CSV like:

  ab,cd
  e\
  f,g\,h
  i"j,k m

As Ethan mentioned, any convention will break some
expectations/compatibility, unless the plan is to build in
a wide range of application- or convention-specific input
filters. (And those filters implemented in Perl is usually
how I get by and produce the format gnuplot expects.)