gnuplot-beta

Should heredocs be included in "save"?

[Philipp K. J. <ja...@ie...>]

Just a question/suggestion...

I am trying to understand the proper use of the
new "heredoc" feature.

Ethan pointed out (on gp-info) that it is usually
used as part of a command script. That makes me 
wonder whether the contents of a heredoc should be 
included in the information that is written to file
using "save".

My rationale is this: traditionally, "save" 
persisted the entire session state (excluding
terminal settings). Loading the resulting file
with "load" recreated the entire session, even
if gnuplot had been exited and restarted in the
meantime.

Now I imagine an interactive session, in which
I define a heredoc (to add some points to a graph,
for example). Now doing a "save", exit, and "load"
will NOT recreate the session, in fact the "load"
will fail, since the data block (=heredoc) is no
longer defined!

I think it is undesirable that a command file
generated by "save" will fail to "load". Moreover,
I strongly think that "re-creatibility" of a plot
is an essential feature. But currently, plots
involving interactively defined heredocs are not
recreateable. 

More broadly speaking, I am wondering about the
strategic intent/direction for heredocs. For
instance, I would expect to be able to load a 
data file into a heredoc "variable", so that I 
then don't have to touch the underlying file again.
Currently, their primary purpose seems to be to 
embed  data in command (demo?) files - which is 
not exactly a feature most users need every day.

Best,

		Ph.

Re: Should heredocs be included in "save"?

[Ethan A M. <sf...@us...>]

On Wednesday, 29 October, 2014 11:59:44 Philipp K. Janert wrote:
> 
> Just a question/suggestion...
> 
> I am trying to understand the proper use of the
> new "heredoc" feature.
> 
> Ethan pointed out (on gp-info) that it is usually
> used as part of a command script. That makes me 
> wonder whether the contents of a heredoc should be 
> included in the information that is written to file
> using "save".
> 
> My rationale is this: traditionally, "save" 
> persisted the entire session state (excluding
> terminal settings). Loading the resulting file
> with "load" recreated the entire session, even
> if gnuplot had been exited and restarted in the
> meantime.
> 
> Now I imagine an interactive session, in which
> I define a heredoc (to add some points to a graph,
> for example). Now doing a "save", exit, and "load"
> will NOT recreate the session, in fact the "load"
> will fail, since the data block (=heredoc) is no
> longer defined!

I suppose that's one way of thinking about it, but it's
not the viewpoint that motivated heredoc originally.
I view it as a mechanism to avoid requiring multiple
files associated with a reusable script.  It is an
alternate data source, not part of the session state.

"save" doesn't include the content of the last data file
plotted, and it doesn't include the content of the last
heredoc plotted.  If you had used in-line data via
plot '-' then it wouldn't save that either.

> I think it is undesirable that a command file
> generated by "save" will fail to "load". Moreover,
> I strongly think that "re-creatibility" of a plot
> is an essential feature. But currently, plots
> involving interactively defined heredocs are not
> recreateable. 

That's exactly backwards to the way I see it.  If a script requires
a separate auxilliary data file to run, that introduces an extra
dependency for reproducing any plot created by that script.
By instead placing the data in a heredoc in the script itself,
the plot can be recreated without tracking down an
additional data file.


> More broadly speaking, I am wondering about the
> strategic intent/direction for heredocs. For
> instance, I would expect to be able to load a 
> data file into a heredoc "variable", so that I 
> then don't have to touch the underlying file again.

> Currently, their primary purpose seems to be to 
> embed  data in command (demo?) files - which is 
> not exactly a feature most users need every day.

Think of a data analysis script that takes as input a data file
and produces a plot of the experimental data vs a reference
curve.   If the reference curve is easily generated by an
analytic function then probably you'd use that.  But if the
reference curve itself is defined by observed data then
either you need to provide a separate reference data file or
place the reference data in a heredoc.
Placing it in a heredoc makes the script self-contained.    

For this type of application, "save" is not a normal command.
If you want to regenerate a plot you just run the script again.

	Ethan  

Re: Should heredocs be included in "save"?

[Philipp K. J. <ja...@ie...>]

Comments below. Others please join, I think this
touches a fundamental question. (At the very end
of this email.)

> > Ethan pointed out (on gp-info) that it is usually
> > used as part of a command script. That makes me 
> > wonder whether the contents of a heredoc should be 
> > included in the information that is written to file
> > using "save".
> > 
> > My rationale is this: traditionally, "save" 
> > persisted the entire session state (excluding
> > terminal settings). Loading the resulting file
> > with "load" recreated the entire session, even
> > if gnuplot had been exited and restarted in the
> > meantime.
> > 
> > Now I imagine an interactive session, in which
> > I define a heredoc (to add some points to a graph,
> > for example). Now doing a "save", exit, and "load"
> > will NOT recreate the session, in fact the "load"
> > will fail, since the data block (=heredoc) is no
> > longer defined!
> 
> I suppose that's one way of thinking about it, but it's
> not the viewpoint that motivated heredoc originally.
> I view it as a mechanism to avoid requiring multiple
> files associated with a reusable script.  It is an
> alternate data source, not part of the session state.

Well, IF the data is presented in the command-script,
I agree. But if the data is entered at the command
prompt, then it seems to be the same as a user-defined
variable, and hence part of the session state.

> 
> "save" doesn't include the content of the last data file
> plotted, and it doesn't include the content of the last
> heredoc plotted.  If you had used in-line data via
> plot '-' then it wouldn't save that either.

Yes, but file contents are NEVER part of the session
state - that is (was?) the big distinction in gnuplot:
session state as opposed to data files. 

But heredocs appear to the user as session-variables
that are able to hold more than a single scalar. It
is odd that they are treated differently.

Let me make a symmetry argument: "save" and "load"
are complementary commands. Since heredocs can be
read with "load", it is only natural that they 
should be written with "save". 

(I actually think this symmetry/consistency argument 
is a "killer" argument.)

> 
> > I think it is undesirable that a command file
> > generated by "save" will fail to "load". Moreover,
> > I strongly think that "re-creatibility" of a plot
> > is an essential feature. But currently, plots
> > involving interactively defined heredocs are not
> > recreateable. 
> 
> That's exactly backwards to the way I see it.  If a script requires

I agree! And on some level, that's a compliment to
the feature: apparently, it has uses quite outside
of the original intent.

> a separate auxilliary data file to run, that introduces an extra
> dependency for reproducing any plot created by that script.
> By instead placing the data in a heredoc in the script itself,
> the plot can be recreated without tracking down an
> additional data file.
> 

Yes, I understand that (and I will admit to having
this problem in the past). 

At the same time, the need to create and maintain
demo scripts is not a widespread activity. I am
concerned that a paradigm-breaking feature has been
introduced only for the maintenance of the demo/
folder! 

But now that it's there, it should be made generally
useful. (Or quietly swept under the rug again, and
kept only as "undocumented maintainer feature".)

> 
> > More broadly speaking, I am wondering about the
> > strategic intent/direction for heredocs. For
> > instance, I would expect to be able to load a 
> > data file into a heredoc "variable", so that I 
> > then don't have to touch the underlying file again.
> 
> > Currently, their primary purpose seems to be to 
> > embed  data in command (demo?) files - which is 
> > not exactly a feature most users need every day.
> 
> Think of a data analysis script that takes as input a data file
> and produces a plot of the experimental data vs a reference
> curve.   If the reference curve is easily generated by an
> analytic function then probably you'd use that.  But if the
> reference curve itself is defined by observed data then
> either you need to provide a separate reference data file or
> place the reference data in a heredoc.

I disagree - if your reference model is given through
a data set, then it's a data set and should be treated
as such.

> Placing it in a heredoc makes the script self-contained.    

I totally disagree, and I wish there would be a much
bigger discussion: we are now breaking the distinction
of data and commands. Is this really a good idea? We
are also abandoning the idea of not maintaining data 
in the session state. Is this architecturally a good
direction? I am not at all sure - on balance, I would
say: it is not.

One of gnuplot's strong features has always been its
simplicity: data was in unstructured text files, there
was no session state, command files were separate from
data.

We are now breaking this. We are beginning to maintain
state - which means that before long, there will be 
a desire to "reload the session just as it was before",
with various preloaded data sets and all that. 

We are also now mixing data and commands. At what point
will it become habitual to save data always in command
files? There goes the ability to load them into Excel
or pipe them through sort.

At the end of this process stands something like R. If
you know R, you'll see that it has exactly those features:
complicated sessions, complicated internal data models,
the whole shmear. 

But: R is a pain to learn.
And: if gnuplot wants to compete with R, it will lose.
  (In their field, their headstart is too great.)

Do we need to go there, just in order to simplify the
maintenance of the demo/ folder?

Sometimes one should simply say "No" to a new feature.
Simplicity is a virtue. It should not be abandoned
unnecessarily.

I think that heredocs (and some other recent features,
but heredocs in particular) take gnuplot in an unhealthy
direction. I don't think their consequences have been
properly thought through. These features add complexity, 
and they will invariably lead to unforeseen and undesirable
interactions between features - which will lead to even 
more complexity to fix.

Maybe there is a valid need for some of these (and other,
yet unproposed) features. But then we should try to
understand the ramifications, and - quite possibly -
reject some of these ideas, in the name of long-term
health and survival.

(To give you a sense for what I foresee: now that we
have loops, there is a need for an iterable data 
structure. (Currently there is none.) Then there will
be the desire to be able to iterate over the entries
in a heredoc. Now you pretty much have what R calls a
"data frame". Once you have data frames, it makes sense
to operate on them - things like vector addition or the
odd indexing magic that R and matlab delight in. At 
that point, it makes sense to link a matrix library
to do some numerics. And so on.)

I think the Numpy/Scipy universe is suffering from the
inability to say "No". This has led to a humongous,
impenetrable, and totally unstructured API - and
serious quality problems! (Adding "one more feature"
is easy, but working out all the edge cases and 
interactions is not!) 

Gnuplot has always been simple, but reliable. I think
we are at risk of abandoning them - without realizing
it.

> 
> For this type of application, "save" is not a normal command.
> If you want to regenerate a plot you just run the script again.
> 
> 	Ethan  
> 

Re: Should heredocs be included in "save"?

[Ethan A M. <sf...@us...>]

On Wednesday, 29 October, 2014 15:14:40 Philipp K. Janert wrote:

[big snip to get to a discussion point]

> > Placing it in a heredoc makes the script self-contained.    
> 
> I totally disagree, and I wish there would be a much
> bigger discussion: we are now breaking the distinction
> of data and commands. Is this really a good idea? We
> are also abandoning the idea of not maintaining data 
> in the session state. Is this architecturally a good
> direction? I am not at all sure - on balance, I would
> say: it is not.
> 
> One of gnuplot's strong features has always been its
> simplicity: data was in unstructured text files,

   Binary data files were introduced some time between verions 3.5 and 3.7.
   So - about 20 years ago

> there was no session state,

   I am not sure what you mean here.
   Isn't the session state exactly what you are saving in a "save" command?
   I.e. all the current settings, variable assignments, function defs, etc
  
> command files were separate from data.

   In-line data via plot '-'
   has been supported at least as far back as version 3.7

> We are now breaking this. We are beginning to maintain
> state - which means that before long, there will be 
> a desire to "reload the session just as it was before",
> with various preloaded data sets and all that. 

  Hasn't there all along been a current state that can be saved
  and later reloaded?

> We are also now mixing data and commands. At what point
> will it become habitual to save data always in command
> files? There goes the ability to load them into Excel
> or pipe them through sort.

 Sorry, I'm lost now. 
 What is it that you would be loading into excel?
 While I have taken data _out_ of excel as a *.csv file to plot
 in gnuplot, I am sure I have never wanted to export anything
 back in the other direction.   

 I am honestly confused at this point whether you are arguing
 that a datablock ("heredoc") is part of the current state and
 therefore should be included in save/restore, or whether you
 are arguing that data should never be part of the current state
 and therefore it should not be saved.

	Ethan

> 
> At the end of this process stands something like R. If
> you know R, you'll see that it has exactly those features:
> complicated sessions, complicated internal data models,
> the whole shmear. 
> 
> But: R is a pain to learn.
> And: if gnuplot wants to compete with R, it will lose.
>   (In their field, their headstart is too great.)
> 
> Do we need to go there, just in order to simplify the
> maintenance of the demo/ folder?
> 
> Sometimes one should simply say "No" to a new feature.
> Simplicity is a virtue. It should not be abandoned
> unnecessarily.
> 
> I think that heredocs (and some other recent features,
> but heredocs in particular) take gnuplot in an unhealthy
> direction. I don't think their consequences have been
> properly thought through. These features add complexity, 
> and they will invariably lead to unforeseen and undesirable
> interactions between features - which will lead to even 
> more complexity to fix.
> 
> Maybe there is a valid need for some of these (and other,
> yet unproposed) features. But then we should try to
> understand the ramifications, and - quite possibly -
> reject some of these ideas, in the name of long-term
> health and survival.
> 
> (To give you a sense for what I foresee: now that we
> have loops, there is a need for an iterable data 
> structure. (Currently there is none.) Then there will
> be the desire to be able to iterate over the entries
> in a heredoc. Now you pretty much have what R calls a
> "data frame". Once you have data frames, it makes sense
> to operate on them - things like vector addition or the
> odd indexing magic that R and matlab delight in. At 
> that point, it makes sense to link a matrix library
> to do some numerics. And so on.)
> 
> I think the Numpy/Scipy universe is suffering from the
> inability to say "No". This has led to a humongous,
> impenetrable, and totally unstructured API - and
> serious quality problems! (Adding "one more feature"
> is easy, but working out all the edge cases and 
> interactions is not!) 
> 
> Gnuplot has always been simple, but reliable. I think
> we are at risk of abandoning them - without realizing
> it.
> 
> > 
> > For this type of application, "save" is not a normal command.
> > If you want to regenerate a plot you just run the script again.
> > 
> > 	Ethan  
> > 
> 
> 
> ------------------------------------------------------------------------------
> _______________________________________________
> gnuplot-beta mailing list
> gnu...@li...
> Membership management via: https://lists.sourceforge.net/lists/listinfo/gnuplot-beta

Re: Should heredocs be included in "save"?

[Philipp K. J. <ja...@ie...>]

On Wed, 29 Oct 2014 15:59:21 -0700
Ethan A Merritt <sf...@us...> wrote:

> On Wednesday, 29 October, 2014 15:14:40 Philipp K. Janert wrote:
> 
> [big snip to get to a discussion point]
> 
> > > Placing it in a heredoc makes the script self-contained.    

I am splitting this thread into two.

This one is to heredocs: My argument is
that if heredocs stay, then they should
be thought through and implemented in 
such a way as to present a consistent
user experience.

They can be loaded with "load" and in no
other way. Hence, they should be written
(or at least: be writeable) with "save".

That would support your intended use case,
but also support other uses people will 
find for heredocs.

> > 
> > I totally disagree, and I wish there would be a much
> > bigger discussion: we are now breaking the distinction
> > of data and commands. Is this really a good idea? We
> > are also abandoning the idea of not maintaining data 
> > in the session state. Is this architecturally a good
> > direction? I am not at all sure - on balance, I would
> > say: it is not.
> > 
> > One of gnuplot's strong features has always been its
> > simplicity: data was in unstructured text files,
> 
>    Binary data files were introduced some time between verions 3.5
> and 3.7. So - about 20 years ago
> 
> > there was no session state,
> 
>    I am not sure what you mean here.
>    Isn't the session state exactly what you are saving in a "save"
> command? I.e. all the current settings, variable assignments,
> function defs, etc 
> > command files were separate from data.
> 
>    In-line data via plot '-'
>    has been supported at least as far back as version 3.7
> 
> > We are now breaking this. We are beginning to maintain
> > state - which means that before long, there will be 
> > a desire to "reload the session just as it was before",
> > with various preloaded data sets and all that. 
> 
>   Hasn't there all along been a current state that can be saved
>   and later reloaded?
> 
> > We are also now mixing data and commands. At what point
> > will it become habitual to save data always in command
> > files? There goes the ability to load them into Excel
> > or pipe them through sort.
> 
>  Sorry, I'm lost now. 
>  What is it that you would be loading into excel?
>  While I have taken data _out_ of excel as a *.csv file to plot
>  in gnuplot, I am sure I have never wanted to export anything
>  back in the other direction.   
> 
>  I am honestly confused at this point whether you are arguing
>  that a datablock ("heredoc") is part of the current state and
>  therefore should be included in save/restore, or whether you
>  are arguing that data should never be part of the current state
>  and therefore it should not be saved.
> 
> 	Ethan
> 
> > 
> > At the end of this process stands something like R. If
> > you know R, you'll see that it has exactly those features:
> > complicated sessions, complicated internal data models,
> > the whole shmear. 
> > 
> > But: R is a pain to learn.
> > And: if gnuplot wants to compete with R, it will lose.
> >   (In their field, their headstart is too great.)
> > 
> > Do we need to go there, just in order to simplify the
> > maintenance of the demo/ folder?
> > 
> > Sometimes one should simply say "No" to a new feature.
> > Simplicity is a virtue. It should not be abandoned
> > unnecessarily.
> > 
> > I think that heredocs (and some other recent features,
> > but heredocs in particular) take gnuplot in an unhealthy
> > direction. I don't think their consequences have been
> > properly thought through. These features add complexity, 
> > and they will invariably lead to unforeseen and undesirable
> > interactions between features - which will lead to even 
> > more complexity to fix.
> > 
> > Maybe there is a valid need for some of these (and other,
> > yet unproposed) features. But then we should try to
> > understand the ramifications, and - quite possibly -
> > reject some of these ideas, in the name of long-term
> > health and survival.
> > 
> > (To give you a sense for what I foresee: now that we
> > have loops, there is a need for an iterable data 
> > structure. (Currently there is none.) Then there will
> > be the desire to be able to iterate over the entries
> > in a heredoc. Now you pretty much have what R calls a
> > "data frame". Once you have data frames, it makes sense
> > to operate on them - things like vector addition or the
> > odd indexing magic that R and matlab delight in. At 
> > that point, it makes sense to link a matrix library
> > to do some numerics. And so on.)
> > 
> > I think the Numpy/Scipy universe is suffering from the
> > inability to say "No". This has led to a humongous,
> > impenetrable, and totally unstructured API - and
> > serious quality problems! (Adding "one more feature"
> > is easy, but working out all the edge cases and 
> > interactions is not!) 
> > 
> > Gnuplot has always been simple, but reliable. I think
> > we are at risk of abandoning them - without realizing
> > it.
> > 
> > > 
> > > For this type of application, "save" is not a normal command.
> > > If you want to regenerate a plot you just run the script again.
> > > 
> > > 	Ethan  
> > > 
> > 
> > 
> > ------------------------------------------------------------------------------
> > _______________________________________________
> > gnuplot-beta mailing list
> > gnu...@li...
> > Membership management via:
> > https://lists.sourceforge.net/lists/listinfo/gnuplot-beta

Re: Should heredocs be included in "save"?

[Allin C. <cot...@wf...>]

On Wed, 29 Oct 2014, Philipp K. Janert wrote:

> On Wed, 29 Oct 2014 15:59:21 -0700
> Ethan A Merritt <sf...@us...> wrote:
>
>> On Wednesday, 29 October, 2014 15:14:40 Philipp K. Janert wrote:
>>
>> [big snip to get to a discussion point]
>>
>>>> Placing it in a heredoc makes the script self-contained.
>
> I am splitting this thread into two.
>
> This one is to heredocs: My argument is
> that if heredocs stay, then they should
> be thought through and implemented in
> such a way as to present a consistent
> user experience.
>
> They can be loaded with "load" and in no
> other way. Hence, they should be written
> (or at least: be writeable) with "save".

It's not true that "heredoc" data can be loaded only with "load" 
(and also not true that the only, or even primary, practical use of 
heredoc data is in the gnuplot demo files).

The attraction of this feature is that one can efficiently construct 
an integrated plot file (gnuplot commands plus data) for use now and 
re-use later.

It seems to me you're missing entirely the business of creating 
gnuplot files by programmatic means (whether C programs, perl or 
python scripts or whatever). In that context you can now output a 
gnuplot file with the data inline, so that the data and commands 
will not come "unstuck". The generating program will simply print 
the data into a "heredoc" field, with no "load" required.

You could achieve a similar effect before, but less efficiently, 
since you'd have to use "plot '-'" plus e-terminated "stdin" blocks. 
This meant that if you wanted to plot n variables on the y-axis 
against a single x variable, you'd have to repeat the x data n 
times.

I find it difficult to imagine much use for "heredoc" data blocks 
composed interactively, other than perhaps in trivial cases: would 
you really trust yourself to type large amounts of data in that way? 
Surely you'd compose the script in your favorite editor, then run it 
and revise it as needed.

Allin Cottrell

Re: Should heredocs be included in "save"?

[Philipp K. J. <ja...@ie...>]

[snip]

> It seems to me you're missing entirely the business of creating 
> gnuplot files by programmatic means (whether C programs, perl or 
> python scripts or whatever). In that context you can now output a 
> gnuplot file with the data inline, so that the data and commands 
> will not come "unstuck". The generating program will simply print 
> the data into a "heredoc" field, with no "load" required.

You are absolutely right - this application never
occurred to me.

The way I saw gnuplot heredocs is that they are
a session variables (which they are!), which hold
a complex data set.

The use case that you and Ethan describe is more
akin to Perl's __DATA__ blocks (as opposed to 
heredocs): a block in a command file that does 
not contain further commands, but static data, 
that can be accessed from the commands (but, I 
would add, not outside of this command file!).

(One could even imagine extending the existing
"index" feature to such data blocks, and in this
way allow for multiple data sets to reside in one
file.)

My point is not that I don't understand the 
INTENDED use case - my point is that heredocs,
as currently designed, are not restricted in
any way to that use case. Moreover (as my
reaction proves), they suggest very different
uses for themselves, and will invariably morph
in that direction.

> 
> I find it difficult to imagine much use for "heredoc" data blocks 
> composed interactively, other than perhaps in trivial cases: would 
> you really trust yourself to type large amounts of data in that way? 
> Surely you'd compose the script in your favorite editor, then run it 
> and revise it as needed.

I couldn't either - hence my question on gnuplot-info
the other day.