Re: conflict of separator and missing

[<pl...@pi...>]

On 17/06/16 20:19, Ethan A Merritt wrote:
> On Friday, 17 June, 2016 08:42:46 pl...@pi... wrote:
>
>> Can you comment on whether there is a reason to strip leading WS but not
>> strip trailing WS ( as is currently done )?  This seems a little odd and
>> is unlikely to be a combination that one would expect.
>>
>> Unless there is a positive reason for this choice or downside that I'm
>> missing, It would seem more consistent to strip both ends.
>
> You may be over-thinking this.
> Gnuplot does no "stripping" or other pre-processing of the input line.
> Successive fields are read by standard calls to the C library.
>
> For a numeric field this is either atod() or sscanf().
> In either case the C language formatted input routine
> 1) skips over any leading whitespace,
> 2) parses the number, and
> 3) stops at the first character that is not part of the number.
> That next character could be anything.
> In other words, skipping any leading whitespace and ignoring any
> trailing garbage is all normal behaviour for the libc input routines.
>
> For a string field (e.g. 'plot with labels') it's a bit more complicated.
> In this case yes, unquoted leading and trailing whitespace is eventually
> stripped but this happens at a later stage, not while parsing the input.
> Also some escape-character sequences are translated.
>
> For detection of a "missing" flag? Well, that's what we're discussing.
> This thread started with the example of using a numeric value as a
> "missing" flag, which adds an additional layer of ambiguity.
> If you think of it as substituting for a number, then trailing characters
> should be ignored.  If you think of it as a string, then trailing
> characters including whitespace are potentially significant.  If it
> really were a string then at a later stage any trailing whitespace would
> be deleted but nothing in the current input layer (datafile.c) does this,
> and this is the layer that has to decide whether the current record
> is missing or not.
>
>> It seems that this choice may have been motived by the comment issue and
>> maybe making the comment behaviour more consistent, as outlined above,
>> would neatly resolve both.
>
> The only comment issue I see is that the documentation could be more
> clear that it is talking about command lines rather than data files.
> That much is easy to fix.
>
> 	Ethan
>

Thanks Ethan.

last things first:

gnuplot> help comment
  Comments are supported as follows: a `#` may appear in most places in 
a line
  and `gnuplot` will ignore the rest of the line.  It will not have this 
effect
  inside quotes, inside numbers (including complex numbers), inside command
  substitutions, etc.  In short, it works anywhere it makes sense to work.

  See also `set datafile commentschars` for specifying comment characters in
  data files.  Note that if a comment line ends in '\' then the subsequent
  line is also treated as a comment.


I don't understand your comment that this is about command lines. Here 
it is explicitly referring to 'datafile'.

What you have previously referred to as  end of line 'garbage' is 
documented as a feature. This contradicts what you previously said about 
comment char only being legit at the beginning of a line. The help says 
: `#` may appear in most places in a line.  I was not mistaken in doing 
this, I was following the doc.



Secondly, 'missing' flag:

To clarify , I never thought the missing string was a number even 
though, in this case it represents a number. I have always been clear 
that it is treated as a string.
set datafile missing "-99.99"    # it's a string.


Thanks for the detailed explanation, I see the problem: evolutionary 
growth of the code.

 > If it really were a string then at a later stage any trailing 
whitespace would
 > be deleted but nothing in the current input layer (datafile.c) does this,
 > and this is the layer that has to decide whether the current record
 > is missing or not.

Then this is crux of the problem.

If all separators cases were made to follow what is described in help: 
ie commentchars can appear 'almost anywhere', then thing would get 
simpler and be more consistent. The input line needs to be truncated at 
the first occurrence of  'commentschars' and this new position becomes 
an implicit field terminator when the line is parsed for field positions.

This sounds like one line of code.

Explicitly stripping off comments at an early stage would seem to be 
desirable, since hoping they will fall of the end as 'garbage' is non 
consistent between WS and non-WS separated files.

Then, since 'missing' is a string, it needs to be processed as a string 
in a similar way that you say is done later for real strings.

Once a particular field has been isolated between two field terminators, 
it needs to be checked against 'missing' string.  Std functions will 
test for presence of missing in the field. Then a check needs to be made 
to see whether there is anything other than WS before or after.

set datafile missing "ignore"
1,2,3,4, ignore
1,2,3,4, ignore\t
1,2,3,4, ignore   # comment
1,2,3,4, ignore_not
1,2,3,4, ignore not

To my mind, in  the last two examples the last field does not match 
missing.

The last line does not match for the same reason the following line does 
not. Both contain invalid data.
1,2,3,4, ignore not,6


It seems that ensuring comments work as documented even for csv files 
actually helps in producing overall consistent behaviour for nearly zero 
effort and processing time.

Help should indicate that leading or trailing space is not  allowed in 
'set datafile missing'.

/my2c/

Peter.