Re: Should heredocs be included in "save"?

[Philipp K. J. <ja...@ie...>]

Comments below. Others please join, I think this
touches a fundamental question. (At the very end
of this email.)

> > Ethan pointed out (on gp-info) that it is usually
> > used as part of a command script. That makes me 
> > wonder whether the contents of a heredoc should be 
> > included in the information that is written to file
> > using "save".
> > 
> > My rationale is this: traditionally, "save" 
> > persisted the entire session state (excluding
> > terminal settings). Loading the resulting file
> > with "load" recreated the entire session, even
> > if gnuplot had been exited and restarted in the
> > meantime.
> > 
> > Now I imagine an interactive session, in which
> > I define a heredoc (to add some points to a graph,
> > for example). Now doing a "save", exit, and "load"
> > will NOT recreate the session, in fact the "load"
> > will fail, since the data block (=heredoc) is no
> > longer defined!
> 
> I suppose that's one way of thinking about it, but it's
> not the viewpoint that motivated heredoc originally.
> I view it as a mechanism to avoid requiring multiple
> files associated with a reusable script.  It is an
> alternate data source, not part of the session state.

Well, IF the data is presented in the command-script,
I agree. But if the data is entered at the command
prompt, then it seems to be the same as a user-defined
variable, and hence part of the session state.

> 
> "save" doesn't include the content of the last data file
> plotted, and it doesn't include the content of the last
> heredoc plotted.  If you had used in-line data via
> plot '-' then it wouldn't save that either.

Yes, but file contents are NEVER part of the session
state - that is (was?) the big distinction in gnuplot:
session state as opposed to data files. 

But heredocs appear to the user as session-variables
that are able to hold more than a single scalar. It
is odd that they are treated differently.

Let me make a symmetry argument: "save" and "load"
are complementary commands. Since heredocs can be
read with "load", it is only natural that they 
should be written with "save". 

(I actually think this symmetry/consistency argument 
is a "killer" argument.)

> 
> > I think it is undesirable that a command file
> > generated by "save" will fail to "load". Moreover,
> > I strongly think that "re-creatibility" of a plot
> > is an essential feature. But currently, plots
> > involving interactively defined heredocs are not
> > recreateable. 
> 
> That's exactly backwards to the way I see it.  If a script requires

I agree! And on some level, that's a compliment to
the feature: apparently, it has uses quite outside
of the original intent.

> a separate auxilliary data file to run, that introduces an extra
> dependency for reproducing any plot created by that script.
> By instead placing the data in a heredoc in the script itself,
> the plot can be recreated without tracking down an
> additional data file.
> 

Yes, I understand that (and I will admit to having
this problem in the past). 

At the same time, the need to create and maintain
demo scripts is not a widespread activity. I am
concerned that a paradigm-breaking feature has been
introduced only for the maintenance of the demo/
folder! 

But now that it's there, it should be made generally
useful. (Or quietly swept under the rug again, and
kept only as "undocumented maintainer feature".)

> 
> > More broadly speaking, I am wondering about the
> > strategic intent/direction for heredocs. For
> > instance, I would expect to be able to load a 
> > data file into a heredoc "variable", so that I 
> > then don't have to touch the underlying file again.
> 
> > Currently, their primary purpose seems to be to 
> > embed  data in command (demo?) files - which is 
> > not exactly a feature most users need every day.
> 
> Think of a data analysis script that takes as input a data file
> and produces a plot of the experimental data vs a reference
> curve.   If the reference curve is easily generated by an
> analytic function then probably you'd use that.  But if the
> reference curve itself is defined by observed data then
> either you need to provide a separate reference data file or
> place the reference data in a heredoc.

I disagree - if your reference model is given through
a data set, then it's a data set and should be treated
as such.

> Placing it in a heredoc makes the script self-contained.    

I totally disagree, and I wish there would be a much
bigger discussion: we are now breaking the distinction
of data and commands. Is this really a good idea? We
are also abandoning the idea of not maintaining data 
in the session state. Is this architecturally a good
direction? I am not at all sure - on balance, I would
say: it is not.

One of gnuplot's strong features has always been its
simplicity: data was in unstructured text files, there
was no session state, command files were separate from
data.

We are now breaking this. We are beginning to maintain
state - which means that before long, there will be 
a desire to "reload the session just as it was before",
with various preloaded data sets and all that. 

We are also now mixing data and commands. At what point
will it become habitual to save data always in command
files? There goes the ability to load them into Excel
or pipe them through sort.

At the end of this process stands something like R. If
you know R, you'll see that it has exactly those features:
complicated sessions, complicated internal data models,
the whole shmear. 

But: R is a pain to learn.
And: if gnuplot wants to compete with R, it will lose.
  (In their field, their headstart is too great.)

Do we need to go there, just in order to simplify the
maintenance of the demo/ folder?

Sometimes one should simply say "No" to a new feature.
Simplicity is a virtue. It should not be abandoned
unnecessarily.

I think that heredocs (and some other recent features,
but heredocs in particular) take gnuplot in an unhealthy
direction. I don't think their consequences have been
properly thought through. These features add complexity, 
and they will invariably lead to unforeseen and undesirable
interactions between features - which will lead to even 
more complexity to fix.

Maybe there is a valid need for some of these (and other,
yet unproposed) features. But then we should try to
understand the ramifications, and - quite possibly -
reject some of these ideas, in the name of long-term
health and survival.

(To give you a sense for what I foresee: now that we
have loops, there is a need for an iterable data 
structure. (Currently there is none.) Then there will
be the desire to be able to iterate over the entries
in a heredoc. Now you pretty much have what R calls a
"data frame". Once you have data frames, it makes sense
to operate on them - things like vector addition or the
odd indexing magic that R and matlab delight in. At 
that point, it makes sense to link a matrix library
to do some numerics. And so on.)

I think the Numpy/Scipy universe is suffering from the
inability to say "No". This has led to a humongous,
impenetrable, and totally unstructured API - and
serious quality problems! (Adding "one more feature"
is easy, but working out all the edge cases and 
interactions is not!) 

Gnuplot has always been simple, but reliable. I think
we are at risk of abandoning them - without realizing
it.

> 
> For this type of application, "save" is not a normal command.
> If you want to regenerate a plot you just run the script again.
> 
> 	Ethan  
>