Re: Weirdness in column specifications, column() vs dollar

[Juhász P. <pet...@gm...>]

On Wed, 2022-03-02 at 14:36 -0800, Ethan A Merritt wrote:
> On Wednesday, 2 March 2022 06:37:26 PST Peter Juhasz wrote:
> > On Wed, Mar 2, 2022 at 5:22 AM Ethan A Merritt <me...@uw...>
> > wrote:
> > 
> > > > Observations:
> > > > - it's as if the mere presence of a $X in the specification
> > > > causes the
> > > > datum to be marked as invalid, and dropped entirely, if column
> > > > X
> > > > doesn't contain data, no matter what the rest of the
> > > > specification is.
> 
> This is intentional.
> 
[... detailed explanation snipped ...]

This all sounds eminently horrible. I admit I never had to think into
the various edge cases like `column($2)` and so on, but even
acknowledging these, it seems to me that peeking ahead in the
expression and just throwing away the datum if a column happens to be
empty/invalid is an overly zealous, and in the end, suboptimal,
approach, because it throws away that datum even if the invalid value
ends up not influencing the result.

If I understand you correctly, my use case (plotting a dataset with a
potentially null column with a default value, so that a point is
plotted for every row) is supposed to be impossible, and the method
I've eventually stumbled on (`valid(N) ? column(N) : 0`) only works
because of an implementation detail (that peeking ahead for `column()`
was deemed too hard).

And even accepting all this, the fact remains that there is an
undocumented discrepancy in the user interface (`column(N)` vs `$N`
behaves differently, even though the documentation states that the
latter is just a shortcut to the former), and that all of this is not
easily discoverable and confusing to the user.

> 
> Any suggestions for improvement are welcome.

To try to be constructive:

- the state of affairs should be documented somewhere (though I'm not
sure under which keyword does it belong)

- perhaps there could be a separate, explicit function for plotting a
column with a default value. My subconcious is trying to suggest a
defined-or operator like perl's `//`, but I realise that that would not
help much with the problems that already exist on the level of
expression parsing. But perhaps a two-argument alternative to
`column(X)` could be added, one that allows a default value to be
specified, e.g. `defaultcolumn(1, 3.14)`, or just make `column` accept
an optional second argument, like `timecolumn` does.

I'm not sure how much value this second proposal would add, though,
since the `valid ? column : default` syntax does work, but perhaps a
separate explicit function would be useful if `column` was ever "fixed"
to behave like the dollar operator.


> 
>    cheers,
>            Ethan
> 
> 

best regards,
Peter Juhasz