Re: conflict of separator and missing

[<pl...@pi...>]

On 17/06/16 00:11, Ethan A Merritt wrote:
> On Thursday, 16 June, 2016 20:46:00 pl...@pi... wrote:
>> On 16/06/16 19:44, Ethan A Merritt wrote:
>>> On Thursday, 16 June, 2016 18:00:00 pl...@pi... wrote:
>>>> On 16/06/16 17:23, sfeam wrote:
>>>>
>>>>
>>>>> -   The comparison is to a string, not a numerical value, so -99.00 ne -99.0 ne -99
>>>> Sounds reasonable. From the gnuplot POV it is a missing *string* ; if
>>>> the cvs is output by a spreadsheet or other software it seems reasonable
>>>> to expect consistent string formatting ( although Excel could have
>>>> different cell formats, that is probably too much to try and anticipate.
>>>>
>>>>
>>>>  >-   Leading whitespace is ignored but trailing whitespace is not.
>>>>
>>>> Seems inconsistent. Was this a programming convenience for minimal
>>>> coding changes or is there a functional logic behind this?
>>>
>>> I though the consensus from a couple of days ago was that any difference
>>> in the remainder of the field was significant, hence extra trailing
>>> characters would mean that the match was imperfect.
>>> Previously "missing A" and "missing B" were both matched as "missing".
>>> Now they are not, even if A is a <tab> or '\n' or '\r'.
>>
>> I don't know what the consensus was but my comment on that was that
>> trailing WS should be stripped, as it is with leading WS.  I was
>> suggesting that any non-WS following the missing string meant the match
>> failed. Specifically relating to your  " ignore A" case.  I did not
>> suggest  WS"ignore"WS should fail.
>>
>> If leading space is stripped, I'm not sure I see why trailing is not
>> also stripped.
>>
>>
>>
>>>
>>>>  > ... and requires that the next character is a field-terminator.
>>>>
>>>> I presume field-terminator.means FS or EOL.
>>>
>>> Separator or null.
>>> EOL is legal with in a csv field, although if you have such a file good
>>> luck to you. When a line of data is read in to gnuplot it is transferred
>>> to a null-terminated string, so the check for null should catch the true
>>> end-of-line.
>>>
>>>> Does this cater for WS at end of line without an explicit FS,
>>>> or does this fall foul of previous point?
>>>
>>> You mean like a DOS-style file with <cr><nl> at the end of the line?
>>> So far as I know this is properly handled by stripping away both line
>>> termination characters on input.  But more testing wouldn't hurt.
>>>
>>> 	Ethan
>>>
>>
>> No , I was not talking about CRLF end of line.
>>
>> It is quite common to have 'invisible' WS   after the last field and
>> being the last field probably no FS.  This is especially the case if
>> there was a comment :  WS to provide visual separation or align  comments:
>>
>> 1,2,3,-999             # last column data got lost in paper records !
>
> A comment character is only valid at the start of a data line.
> A trailing comment like the one you show will be treated as
> extraneous garbage in the last field.
>
> Prior to yesterday's change, if "missing" were set to "-999" then the
> rest of the field it would be ignored because of the intervening whitespace.
> Since today the "missing" test will fail because "-999" is not followed
> immediately by a field separator.
>
> Do you think it should revert to terminating the "missing" check
> on whitespace?  That was the example I tried to give by showing
> that
>   set datafile missing "missing"
> would catch both fields 2 and 3 in a line containing
>    1, missing A, missing B, 4
>
>> Once the # is replaced by #0 to truncate out the comment ,
>
> Such replacement does not happen.

Oops. It seems like I've been relying " extraneous garbage in the last 
field" for my comments fro some time !  This may be why:

gnuplot> help comment
  Comments are supported as follows: a `#` may appear in most places in 
a line
  and `gnuplot` will ignore the rest of the line.  It will not have this 
effect
  inside quotes, inside numbers (including complex numbers), inside command
  substitutions, etc.  In short, it works anywhere it makes sense to work.

That behaviour should probably be consistent w.r.t changes in separator. 
If  a `#` may appear in most places in a line will get cropped for WS 
files , it should work for CSV files. The example data file line that I 
gave should truncate in both formats.

Gnuplot is remarkably good at sorting out almost any file in what seems 
like an intuitive way and that is something that I find quite impressive.

Is there any reason not to support comments at end of line by simply 
changing # ( or whatever the commentchars are set to ) to #0 as I 
incorrectly thought was being done?

Moving towards a consistent behaviour would seem preferable to reverting 
because of this difference.


Can you comment on whether there is a reason to strip leading WS but not 
strip trailing WS ( as is currently done )?  This seems a little odd and 
is unlikely to be a combination that one would expect.

Unless there is a positive reason for this choice or downside that I'm 
missing, It would seem more consistent to strip both ends.

It seems that this choice may have been motived by the comment issue and 
maybe making the comment behaviour more consistent, as outlined above, 
would neatly resolve both.

Peter.



>
>> this line  would fall foul of your new scheme I think.
>
> Yes, it will fail.
> So should I partially revert the change to restore checking for
> "missing" only up to the first whitespace?
>
> 	Ethan
>
>
>
>
>> I see no real reason not to remove the tailing WS , it seems a little
>> odd to strip one end an not the other.
>>
>> CSV is pretty illegible at the best of times. If need to dump a
>> spreadsheet to CSV I often separate with " , " to make the result a
>> little easier to read afterwards.
>>
>> Unless I'm missing something , I don't see any reason or advantage to
>> not stripping trailing WS.
>>
>>
>> Not wishing to be finicky, but you seemed interesting is considering any
>> corner cases.
>>
>> Peter.
>>
>>
>>
>>
>>>
>>>
>>>
>>> 	Ethan
>>>
>>
>>