Re: conflict of separator and missing

[sfeam <sf...@us...>]

On Wednesday, 15 June 2016 06:36:53 PM Tait wrote:
> 
> > > The conventional indication of missing data in a *.csv file is simply
> > > an empty field.  This obviously is not possible in a whitespace-separated
> > > file.  Gnuplot's use of "set missing" is outside any standard practice
> > > I know of for csv files, so anything we choose is likely to strike
> > > someone as wrong.
> 
> I don't follow the comment in the second sentence.

I was contrasting csv files to whitespace-separated files.

If you have only whitespace to separate values, then you can't simply
omit a field because this is indistinguishable from shifting all the
remaining fields over by one.  That's why we need a "missing"
placeholder.  In a csv file you shouldn't really need a "missing" 
placeholder because an empty field is unambiguous.

	Ethan

> Empty fields in a
> TSV file* are indicated by having no data in between the field
> separators. Not only is it possible, but it's quite intuitive, I
> think. I'm adding spaces for clarity, but those spaces wouldn't be
> in the actual file:
> 
>   header1 \t header2 \t header3 \t header4
>   data1 \t data2 \t data3 \t data4
>   data5 \t       \t data7 \t data8
>   ...
> 
> Where data6 would be, is an empty field.
> 
> (* as an aside, TSV or "tab-separated text" is the term I always see
> used for tab-separated values. I've never heard of someone refer to
> a tab-separated file as "CSV".)
> 
> > > For instance, RFC-4180, the closest thing to a csv standard, states that
> > > "any field may be quoted with double quotes".  So in the example above,
> > > should we ignore this line?
> > >   5, 5, "ignore", 5
> > > This one?
> > >   5, 5, " ignore ", 5
> > 
> > Ok, in the absence of any properly defined standard , where software 
> > like Excel ( probably the most common source of "CSV" files for a lot of 
> > people ) produces comma separated variables without using commas, it is 
> > likely to be messy.
> > 
> >   5, 5, "ignore", 5
> > 
> > This seems a bit of a contrived case, what software will quote one field 
> > in a line but not the others?
> 
> Excel does exactly this. Fields are unquoted in general, but (only)
> if they contain delimiter or quoting characters, then they are
> quoted. If they contain quote characters, quotes are double-quoted.
> Delimiter characters are not just "," for CSV, but also newlines.
> Consider three rows of data, each containing two fields:
> 
>   row 1: ab    cd
>   row 2: e\nf  g,h
>   row 3: i"j   k<space>m
> 
> Excel will produce a CSV that looks like this:
> 
>   ab,cd
>   "e
>   f","g,h"
>   "i""j",k m
> 
> This is obviously a contrived pathological case, but it's
> illustrative of what common software "out there" might do.
> Of course, backslash-escaping is also a common convention,
> and for the same input, it might produce a CSV like:
> 
>   ab,cd
>   e\
>   f,g\,h
>   i"j,k m
> 
> As Ethan mentioned, any convention will break some
> expectations/compatibility, unless the plan is to build in
> a wide range of application- or convention-specific input
> filters. (And those filters implemented in Perl is usually
> how I get by and produce the format gnuplot expects.)
>