Re: conflict of separator and missing

[Ethan A M. <sf...@us...>]

On Thursday, 16 June, 2016 20:46:00 pl...@pi... wrote:
> On 16/06/16 19:44, Ethan A Merritt wrote:
> > On Thursday, 16 June, 2016 18:00:00 pl...@pi... wrote:
> >> On 16/06/16 17:23, sfeam wrote:
> >>
> >>
> >>> -   The comparison is to a string, not a numerical value, so -99.00 ne -99.0 ne -99
> >> Sounds reasonable. From the gnuplot POV it is a missing *string* ; if
> >> the cvs is output by a spreadsheet or other software it seems reasonable
> >> to expect consistent string formatting ( although Excel could have
> >> different cell formats, that is probably too much to try and anticipate.
> >>
> >>
> >>  >-   Leading whitespace is ignored but trailing whitespace is not.
> >>
> >> Seems inconsistent. Was this a programming convenience for minimal
> >> coding changes or is there a functional logic behind this?
> >
> > I though the consensus from a couple of days ago was that any difference
> > in the remainder of the field was significant, hence extra trailing
> > characters would mean that the match was imperfect.
> > Previously "missing A" and "missing B" were both matched as "missing".
> > Now they are not, even if A is a <tab> or '\n' or '\r'.
> 
> I don't know what the consensus was but my comment on that was that 
> trailing WS should be stripped, as it is with leading WS.  I was 
> suggesting that any non-WS following the missing string meant the match 
> failed. Specifically relating to your  " ignore A" case.  I did not 
> suggest  WS"ignore"WS should fail.
> 
> If leading space is stripped, I'm not sure I see why trailing is not 
> also stripped.
> 
> 
> 
> >
> >>  > ... and requires that the next character is a field-terminator.
> >>
> >> I presume field-terminator.means FS or EOL.
> >
> > Separator or null.
> > EOL is legal with in a csv field, although if you have such a file good
> > luck to you. When a line of data is read in to gnuplot it is transferred
> > to a null-terminated string, so the check for null should catch the true
> > end-of-line.
> >
> >> Does this cater for WS at end of line without an explicit FS,
> >> or does this fall foul of previous point?
> >
> > You mean like a DOS-style file with <cr><nl> at the end of the line?
> > So far as I know this is properly handled by stripping away both line
> > termination characters on input.  But more testing wouldn't hurt.
> >
> > 	Ethan
> >
> 
> No , I was not talking about CRLF end of line.
> 
> It is quite common to have 'invisible' WS   after the last field and 
> being the last field probably no FS.  This is especially the case if 
> there was a comment :  WS to provide visual separation or align  comments:
> 
> 1,2,3,-999             # last column data got lost in paper records !

A comment character is only valid at the start of a data line.
A trailing comment like the one you show will be treated as
extraneous garbage in the last field. 

Prior to yesterday's change, if "missing" were set to "-999" then the 
rest of the field it would be ignored because of the intervening whitespace.
Since today the "missing" test will fail because "-999" is not followed
immediately by a field separator.

Do you think it should revert to terminating the "missing" check
on whitespace?  That was the example I tried to give by showing
that 
  set datafile missing "missing"
would catch both fields 2 and 3 in a line containing
   1, missing A, missing B, 4

> Once the # is replaced by #0 to truncate out the comment , 

Such replacement does not happen.

> this line  would fall foul of your new scheme I think.

Yes, it will fail.
So should I partially revert the change to restore checking for
"missing" only up to the first whitespace?

	Ethan




> I see no real reason not to remove the tailing WS , it seems a little 
> odd to strip one end an not the other.
>
> CSV is pretty illegible at the best of times. If need to dump a 
> spreadsheet to CSV I often separate with " , " to make the result a 
> little easier to read afterwards.
> 
> Unless I'm missing something , I don't see any reason or advantage to 
> not stripping trailing WS.
> 
> 
> Not wishing to be finicky, but you seemed interesting is considering any 
> corner cases.
> 
> Peter.
> 
> 
> 
> 
> >
> >
> >
> > 	Ethan
> >
> 
> 
> ------------------------------------------------------------------------------
> What NetFlow Analyzer can do for you? Monitors network bandwidth and traffic
> patterns at an interface-level. Reveals which users, apps, and protocols are 
> consuming the most bandwidth. Provides multi-vendor support for NetFlow, 
> J-Flow, sFlow and other flows. Make informed decisions using capacity planning
> reports. http://sdm.link/zohomanageengine
> _______________________________________________
> gnuplot-beta mailing list
> gnu...@li...
> Membership management via: https://lists.sourceforge.net/lists/listinfo/gnuplot-beta