Re: conflict of separator and missing

[Ethan A M. <sf...@us...>]

On Saturday, 18 June, 2016 00:18:31 pl...@pi... wrote:
> On 17/06/16 20:19, Ethan A Merritt wrote:
> > On Friday, 17 June, 2016 08:42:46 pl...@pi... wrote:
> >
> >> Can you comment on whether there is a reason to strip leading WS but not
> >> strip trailing WS ( as is currently done )?  This seems a little odd and
> >> is unlikely to be a combination that one would expect.
> >>
> >> Unless there is a positive reason for this choice or downside that I'm
> >> missing, It would seem more consistent to strip both ends.
> >
> > You may be over-thinking this.
> > Gnuplot does no "stripping" or other pre-processing of the input line.
> > Successive fields are read by standard calls to the C library.
> >
> > For a numeric field this is either atod() or sscanf().
> > In either case the C language formatted input routine
> > 1) skips over any leading whitespace,
> > 2) parses the number, and
> > 3) stops at the first character that is not part of the number.
> > That next character could be anything.
> > In other words, skipping any leading whitespace and ignoring any
> > trailing garbage is all normal behaviour for the libc input routines.
> >
> > For a string field (e.g. 'plot with labels') it's a bit more complicated.
> > In this case yes, unquoted leading and trailing whitespace is eventually
> > stripped but this happens at a later stage, not while parsing the input.
> > Also some escape-character sequences are translated.
> >
> > For detection of a "missing" flag? Well, that's what we're discussing.
> > This thread started with the example of using a numeric value as a
> > "missing" flag, which adds an additional layer of ambiguity.
> > If you think of it as substituting for a number, then trailing characters
> > should be ignored.  If you think of it as a string, then trailing
> > characters including whitespace are potentially significant.  If it
> > really were a string then at a later stage any trailing whitespace would
> > be deleted but nothing in the current input layer (datafile.c) does this,
> > and this is the layer that has to decide whether the current record
> > is missing or not.
> >
> >> It seems that this choice may have been motived by the comment issue and
> >> maybe making the comment behaviour more consistent, as outlined above,
> >> would neatly resolve both.
> >
> > The only comment issue I see is that the documentation could be more
> > clear that it is talking about command lines rather than data files.
> > That much is easy to fix.
> >
> > 	Ethan
> >
> 
> Thanks Ethan.
> 
> last things first:
> 
> gnuplot> help comment
>   Comments are supported as follows: a `#` may appear in most places in 
> a line
>   and `gnuplot` will ignore the rest of the line.  It will not have this 
> effect
>   inside quotes, inside numbers (including complex numbers), inside command
>   substitutions, etc.  In short, it works anywhere it makes sense to work.
> 
>   See also `set datafile commentschars` for specifying comment characters in
>   data files.  Note that if a comment line ends in '\' then the subsequent
>   line is also treated as a comment.
> 
> 
> I don't understand your comment that this is about command lines. Here 
> it is explicitly referring to 'datafile'.

The section you quote is talking only about comments in a command line.
I amended the text earlier today to make this more clear.

It then refers you to a separate section about comments in data files,
which says
%%%%%
 `set datafile commentschars` tells `gnuplot` what characters are used in a
 data file to begin comment lines. If the first non-blank character on a line
 is one of the specified characters then the rest of the input line is ignored.
%%%%%

> What you have previously referred to as  end of line 'garbage' is 
> documented as a feature. This contradicts what you previously said about 
> comment char only being legit at the beginning of a line. The help says 
> : `#` may appear in most places in a line.  I was not mistaken in doing 
> this, I was following the doc.

I hope the amended documentation is clearer.
The key points are

1) A comment can start anywhere on a command line, and may affect
multiple physical lines if the initial lines end in a backslash \.

2) The only comment option in a data file is to comment out an entire 
single line by putting a special character at the start of the line.
A backslash at the end of a data file line has no special meaning.

3) The comment character used for datafiles can be something other
than # but still it is only looked for as the first character on the line.

	Ethan


> 
> 
> 
> Secondly, 'missing' flag:
> 
> To clarify , I never thought the missing string was a number even 
> though, in this case it represents a number. I have always been clear 
> that it is treated as a string.
> set datafile missing "-99.99"    # it's a string.
> 
> 
> Thanks for the detailed explanation, I see the problem: evolutionary 
> growth of the code.
> 
>  > If it really were a string then at a later stage any trailing 
> whitespace would
>  > be deleted but nothing in the current input layer (datafile.c) does this,
>  > and this is the layer that has to decide whether the current record
>  > is missing or not.
> 
> Then this is crux of the problem.
> 
> If all separators cases were made to follow what is described in help: 
> ie commentchars can appear 'almost anywhere', then thing would get 
> simpler and be more consistent. The input line needs to be truncated at 
> the first occurrence of  'commentschars' and this new position becomes 
> an implicit field terminator when the line is parsed for field positions.
> 
> This sounds like one line of code.
> 
> Explicitly stripping off comments at an early stage would seem to be 
> desirable, since hoping they will fall of the end as 'garbage' is non 
> consistent between WS and non-WS separated files.
> 
> Then, since 'missing' is a string, it needs to be processed as a string 
> in a similar way that you say is done later for real strings.
> 
> Once a particular field has been isolated between two field terminators, 
> it needs to be checked against 'missing' string.  Std functions will 
> test for presence of missing in the field. Then a check needs to be made 
> to see whether there is anything other than WS before or after.
> 
> set datafile missing "ignore"
> 1,2,3,4, ignore
> 1,2,3,4, ignore\t
> 1,2,3,4, ignore   # comment
> 1,2,3,4, ignore_not
> 1,2,3,4, ignore not
> 
> To my mind, in  the last two examples the last field does not match 
> missing.
> 
> The last line does not match for the same reason the following line does 
> not. Both contain invalid data.
> 1,2,3,4, ignore not,6
> 
> 
> It seems that ensuring comments work as documented even for csv files 
> actually helps in producing overall consistent behaviour for nearly zero 
> effort and processing time.
> 
> Help should indicate that leading or trailing space is not  allowed in 
> 'set datafile missing'.
> 
> /my2c/
> 
> Peter.