Re: conflict of separator and missing

[<pl...@pi...>]

On 15/06/16 20:11, sfeam wrote:
> On Wednesday, 15 June 2016 06:36:53 PM Tait wrote:
>>
>>>> The conventional indication of missing data in a *.csv file is simply
>>>> an empty field.  This obviously is not possible in a whitespace-separated
>>>> file.  Gnuplot's use of "set missing" is outside any standard practice
>>>> I know of for csv files, so anything we choose is likely to strike
>>>> someone as wrong.
>>
>> I don't follow the comment in the second sentence.
>
> I was contrasting csv files to whitespace-separated files.
>
> If you have only whitespace to separate values, then you can't simply
> omit a field because this is indistinguishable from shifting all the
> remaining fields over by one.  That's why we need a "missing"
> placeholder.  In a csv file you shouldn't really need a "missing"
> placeholder because an empty field is unambiguous.
>
> 	Ethan

It should be remembered that csv are rarely a data storage object but 
simply a text dump of something else. I don't think anyone would chose 
csv as a working file format. It's more a means of transmission.

So the question is what is the source data format that the csv is a dump 
of.

Missing value flags like -999 etc are commonly used in software storing 
data in numerical arrays where every datum must have a finite value 
assigned. An array value cannot be 'empty'. A csv dump from such 
software will most likely preserve the missing value flag rather then 
strip them out.

Even though a spreadsheet can have an empty cell, it is often useful to 
have an affirmative flag that indicates that the cell has been processed 
and determined to be a missing datum, rather than just  having been 
overlooked or not yet processed.

So even though a csv file can represent an empty value, this does not 
mean missing value marker is not necessary.

Peter.





>
>> Empty fields in a
>> TSV file* are indicated by having no data in between the field
>> separators. Not only is it possible, but it's quite intuitive, I
>> think. I'm adding spaces for clarity, but those spaces wouldn't be
>> in the actual file:
>>
>>   header1 \t header2 \t header3 \t header4
>>   data1 \t data2 \t data3 \t data4
>>   data5 \t       \t data7 \t data8
>>   ...
>>
>> Where data6 would be, is an empty field.
>>
>> (* as an aside, TSV or "tab-separated text" is the term I always see
>> used for tab-separated values. I've never heard of someone refer to
>> a tab-separated file as "CSV".)
>>
>>>> For instance, RFC-4180, the closest thing to a csv standard, states that
>>>> "any field may be quoted with double quotes".  So in the example above,
>>>> should we ignore this line?
>>>>   5, 5, "ignore", 5
>>>> This one?
>>>>   5, 5, " ignore ", 5
>>>
>>> Ok, in the absence of any properly defined standard , where software
>>> like Excel ( probably the most common source of "CSV" files for a lot of
>>> people ) produces comma separated variables without using commas, it is
>>> likely to be messy.
>>>
>>>   5, 5, "ignore", 5
>>>
>>> This seems a bit of a contrived case, what software will quote one field
>>> in a line but not the others?
>>
>> Excel does exactly this. Fields are unquoted in general, but (only)
>> if they contain delimiter or quoting characters, then they are
>> quoted. If they contain quote characters, quotes are double-quoted.
>> Delimiter characters are not just "," for CSV, but also newlines.
>> Consider three rows of data, each containing two fields:
>>
>>   row 1: ab    cd
>>   row 2: e\nf  g,h
>>   row 3: i"j   k<space>m
>>
>> Excel will produce a CSV that looks like this:
>>
>>   ab,cd
>>   "e
>>   f","g,h"
>>   "i""j",k m
>>
>> This is obviously a contrived pathological case, but it's
>> illustrative of what common software "out there" might do.
>> Of course, backslash-escaping is also a common convention,
>> and for the same input, it might produce a CSV like:
>>
>>   ab,cd
>>   e\
>>   f,g\,h
>>   i"j,k m
>>
>> As Ethan mentioned, any convention will break some
>> expectations/compatibility, unless the plan is to build in
>> a wide range of application- or convention-specific input
>> filters. (And those filters implemented in Perl is usually
>> how I get by and produce the format gnuplot expects.)
>>
>
>