Re: Weirdness in column specifications, column() vs dollar

[Juhász P. <pet...@gm...>]

On Thu, 2022-03-10 at 20:42 -0800, Ethan A Merritt wrote:
> On Friday, 4 March 2022 00:17:15 PST Ethan A Merritt wrote:
> 
> > I dislike the idea of going back to either the original behaviour
> > (always skip input line if a column is NaN or missing) or the
> > version 5.2
> > behaviour (see Bug #2042).  
> > 
> > You have now pointed out the very real bug that one cannot use 
> > valid(N) in an expression containing $N because the entire
> > expression
> > will be skipped.
> > So I dislike the way it is now, also.
> > 
> > Possible options:
> > 
> > - Revert commit b8304eaf.  That would re-introduce Bug #2042 but
> > would
> >   allow use of valid() to avoid it.
> > 
> > - Better documentation. Trying to explain all this is probably much
> > too
> >   difficult, but we could at least warn that if you use valid(N)
> > then
> >   you must also use column(N) rather than $N.
> > 
> > - At the cost of some hackery, I could add a check for the valid()
> > function
> >   itself immediately before the check for $N added by the commit
> > shown above.
> >   The logic would be "if we see the user is doing their own
> > validity checks,
> >   we will not use the known-imperfect check that looks for $
> > signs".
> >   I could cook up a patch for that if you want to test it.
> 
> Eventually I realized that the valid() function goes all the way back
> to gnuplot version 3.something, and it always had the property that
> it couldn't catch "missing" values because they were discarded at an
> earlier stage of the input.
> 
> I have come around to thinking that the only major issue here
> is the one that Peter originally pointed out: that $N and column(N)
> are documented as being identical but they were not.
> 
> So commit c8d468de adds the same initial check to column(N) that
> already exists for $N.  In both cases if column N contains a 
> missing value flag then the data point is skipped prior to
> evaluation of the expression in the "using" specifier.
> So you cannot in practice catch such points by using 
>     valid(N) ? $N : <foo>

I still think that this behavior is hostile to the user - at least it's
consistently hostile now, I guess.

Maybe I'm overreacting, and for most use cases this is the sensible and
expected behavior. What prompted me to send in the original report is
that in my application the missing column was used deep inside a
sprintf for  a hypertext label, only visible on mouseover, and for the
longest time I couldn't understand why such an inconsequential part of
the using spec was causing part of the input to be silently dropped.

Anyway, thanks for looking into this.

best regards,
Peter Juhasz