gnuplot-beta

Unintended consequences

[Philipp K. J. <ja...@ie...>]

My prime argument against the trend towards 
"programming features" (and data structures)
in gnuplot is the worry about unintended
consequences down the road.

I wrote up the little "play" (below) to
demonstrate what I mean by that. Of course
it's a satire, and therefore exaggerates
to make a point.

(I sent this before (30-Oct), but apparently 
it got lost somehow.)


Tongue-in-cheek, please! 

Here we go...


"All: I just implemented heredocs. Now we can bundle data and 
commands into self-contained command files. That should help 
so much managing the demo/ folder."

"I love it - it really cuts down on clutter in my directories."

"Dick - here is the data set we talked about. I know that you 
are a gnuplot user also, so I am not sending you the data as a 
text file, but as a self-contained gnuplot command file. Just 
load it into gnuplot and off you go! It is very convenient."

"John - here is the data set I got from Jim. He sent it as a
self-contained gnuplot command file. I know that you are not
a gnuplot user, but I am sure you can figure out a way to 
extract the data regardless. You got a PhD and everything."

"Help, my name is John. Somebody sent me some data as a 
self-contained gnuplot command file. What the hell am I
supposed to do with that?"

"Hello, John, and welcome to the gnuplot-data mailing list. 
Good news - we just implemented this new terminal "raw-data".
All you need to do is start gnuplot, load the self-contained 
command file that you got with "load", then set the output 
device using "set output filename" (where filename is the 
name of the file that you would like to export the data to), 
then set the terminal to "raw-data", then run "replot" and 
you are good.

"Oh, I see, that's really easy. By the way: why does gnuplot
have this odd notion of "data files"? Isn't that redundant
with the data section in the self-contained command files?"

"Yes, John, you are right. In fact, plain data files are a
deprecated feature in gnuplot 6; starting with gnuplot 6.2
we plan to only support self-contained command files. By that
time, we should also have finished our very simple and
convenient toolkit to manage self-contained command files."

"I have begun work on gptk (gnuplot toolkit) to handle our
convenient self-contained command files but it needs more 
work. I imagine something as simple and intuitive as Unix tar."

"I am almost done with gptk-cut (to extract the command part)
and gptk-extract (to cut out the data part), I also have made
good progress on gptk-paste, to add a data part to an existing
command part, but I am still working on gptk-list, which shows
an inventory of all data sets contained in a file."

"Well, the release date for gnuplot 6.2 has slipped again, 
mostly because I am not done with gptk-cut-binary to extract
the command part from self-contained command files that contain
a data section in binary format. It works for binary data sets
in PNG, but it does not yet work for the arbitrary binary file
formats that have been part of gnuplot since version 3.7 or
so and that we all agreed to allow inside of self-contained
command files when we prepared gnuplot 5.9."

"All - I just discoverd this new feature in the release candidate 
for gnuplot 7. I takes a plain text file (whitespace-separated
columns) and plots it. Of course it is not as powerful as
the self-contained command files that we all know and love,
but I find it amazing that I can now simply write data into
a file and plot it. I didn't know gnuplot could do that."

Re: Unintended consequences - things to think about

[Philipp K. J. <ja...@ie...>]

Here are some things to think about:

1) We now have loops. But we don't have an
iterable data structure. No, wait, we have
one - we fake it as white-space separated 
string. (I admit the ingenuity of the thought,
but - how kludgy is THAT?)

So, before too long, the loops will beget
an array. That seems certain.

2) Once we have arrays, there will be a
desire to load files into them. Otherwise,
why have arrays, right? So, arrays will
beget a "read_file" feature. 

3) Now that we can read file(s) into 
arrays, it's natural that we want to 
operate on them. One could do that with
loops, but (again) it's natural to do 
things like array1 + array2, right? 

4) Whereas items 1-3 lie in the future, the 
following is already an issue today: we have
loops and conditionals, so it seems natural
to write functions. Well - we don't HAVE
functions, but we can fake them as scripts
that we load with "call". Problem is that
these scripts don't define local variables -
any variable I use in my script will become
a variable in the gnuplot session. To prevent
clobbering existing variables, I therefore
need to prepend a prefix to the name of each
variable "MY_SCRIPT_i", "MY_SCRIPT_j", and so
on. Very kludgy. Hence, it is reasonable to
ask for functions with local variables.

(Side question: is that not how matlab 
implements functions - as separate command
files? I seem to remember something like
that. Do we really need to repeat every
mistake that has been made before?)

Anyway...

My point, in case this still isn't clear:

Once you have one programming feature (say:
loops), there is a natural tendency for them 
to require all other programming features 
(data structures, functions, local scope).

Therefore, what starts like a local convenience 
today will - invariably - grow until it has 
acquired all the features that are already 
provided by R, matlab, Octave, Perl, Python, Ruby.

And THAT I think is the wrong direction. If 
somebody wants R, why would they use gnuplot? 

I also think that it is a misdirection of effort: 
do we really need to figure out how to design 
functions with locally scoped variables? This
has already been done, several times. Would it
not be better to leverage other people's work?

If gnuplot needs to acquire programming features
to stay competitive, then it should try to borrow
them from somewhere else, where all these problems
have already been solved - rather than inventing
the wheel (and poorly, at that).

Best,

		Ph.

Re: Unintended consequences - things to think about

[Karl R. <ra...@un...>]

Am 13.11.2014 um 18:20 schrieb Philipp K. Janert:
> 
> Here are some things to think about:
> 
> 1) We now have loops. But we don't have an
> iterable data structure. No, wait, we have
> one - we fake it as white-space separated 
> string. (I admit the ingenuity of the thought,
> but - how kludgy is THAT?)
> 
> So, before too long, the loops will beget
> an array. That seems certain.

Arrays would be very, very useful for plotting. Think of a multiplot,
one plot with five datasets plus fitted curves, and an inset showing a
regression on the fitted parameters. The simplest way to do that in
gnuplot today is have an external script that generates your gnuplot
script. ;-)


> 2) Once we have arrays, there will be a
> desire to load files into them. Otherwise,
> why have arrays, right? So, arrays will
> beget a "read_file" feature. 

Yes. It´ll be a single command, not interfering with anything. With
that, I could e.g. leave all the fitting to another program (although
gnuplot does a very good job at it! Try fitting in Origin.) and hand the
parameters over to gnuplot in a straightforward manner.

> 3) Now that we can read file(s) into 
> arrays, it's natural that we want to 
> operate on them. One could do that with
> loops, but (again) it's natural to do 
> things like array1 + array2, right?

Same, but who would want it, if not to plot the result, which is very
reasonable for a plotting program?

> 
> 4) Whereas items 1-3 lie in the future, the 
> following is already an issue today: we have
> loops and conditionals, so it seems natural
> to write functions. Well - we don't HAVE
> functions, but we can fake them as scripts
> that we load with "call". Problem is that
> these scripts don't define local variables -
> any variable I use in my script will become
> a variable in the gnuplot session. To prevent
> clobbering existing variables, I therefore
> need to prepend a prefix to the name of each
> variable "MY_SCRIPT_i", "MY_SCRIPT_j", and so
> on. Very kludgy. Hence, it is reasonable to
> ask for functions with local variables.

I agree this would be one step too far, but you can import library
functions today, so there´s no real need for this. And i guess it would
be quite hard to find a (backward compatible) syntax for this, let alone
implementing it.

> I also think that it is a misdirection of effort: 
> do we really need to figure out how to design 
> functions with locally scoped variables? This
> has already been done, several times. Would it
> not be better to leverage other people's work?
> 
> If gnuplot needs to acquire programming features
> to stay competitive, then it should try to borrow
> them from somewhere else, where all these problems
> have already been solved - rather than inventing
> the wheel (and poorly, at that).

As long as "keep it backward compatible" stays the most important
paradigm in the development, i don´t quite see how the "programming
environment" features could get out of hand too badly. It makes sure the
learning curve will stay flat in the beginning.

Interfacing gnuplot with other programs is however an issue today, imo.
That would be, for example, much easier with an array variable that can
be read in from a file, see above, and also keep away further need for
new complex features.

  Karl

Re: Unintended consequences - things to think about

[<pl...@pi...>]

On 11/13/14 19:14, Karl Ratzsch wrote:
> Am 13.11.2014 um 18:20 schrieb Philipp K. Janert:
>>
>> Here are some things to think about:
>>
>> 1) We now have loops. But we don't have an
>> iterable data structure. No, wait, we have
>> one - we fake it as white-space separated
>> string. (I admit the ingenuity of the thought,
>> but - how kludgy is THAT?)
>>
>> So, before too long, the loops will beget
>> an array. That seems certain.
>
> Arrays would be very, very useful for plotting. Think of a multiplot,
> one plot with five datasets plus fitted curves, and an inset showing a
> regression on the fitted parameters. The simplest way to do that in
> gnuplot today is have an external script that generates your gnuplot
> script. ;-)
>
>
>> 2) Once we have arrays, there will be a
>> desire to load files into them. Otherwise,
>> why have arrays, right? So, arrays will
>> beget a "read_file" feature.
>
> Yes. It´ll be a single command, not interfering with anything. With
> that, I could e.g. leave all the fitting to another program (although
> gnuplot does a very good job at it! Try fitting in Origin.) and hand the
> parameters over to gnuplot in a straightforward manner.
>
>> 3) Now that we can read file(s) into
>> arrays, it's natural that we want to
>> operate on them. One could do that with
>> loops, but (again) it's natural to do
>> things like array1 + array2, right?
>
> Same, but who would want it, if not to plot the result, which is very
> reasonable for a plotting program?
>
>>
>> 4) Whereas items 1-3 lie in the future, the
>> following is already an issue today: we have
>> loops and conditionals, so it seems natural
>> to write functions. Well - we don't HAVE
>> functions, but we can fake them as scripts
>> that we load with "call". Problem is that
>> these scripts don't define local variables -
>> any variable I use in my script will become
>> a variable in the gnuplot session. To prevent
>> clobbering existing variables, I therefore
>> need to prepend a prefix to the name of each
>> variable "MY_SCRIPT_i", "MY_SCRIPT_j", and so
>> on. Very kludgy. Hence, it is reasonable to
>> ask for functions with local variables.
>
> I agree this would be one step too far, but you can import library
> functions today, so there´s no real need for this. And i guess it would
> be quite hard to find a (backward compatible) syntax for this, let alone
> implementing it.
>
>> I also think that it is a misdirection of effort:
>> do we really need to figure out how to design
>> functions with locally scoped variables? This
>> has already been done, several times. Would it
>> not be better to leverage other people's work?
>>
>> If gnuplot needs to acquire programming features
>> to stay competitive, then it should try to borrow
>> them from somewhere else, where all these problems
>> have already been solved - rather than inventing
>> the wheel (and poorly, at that).
>
> As long as "keep it backward compatible" stays the most important
> paradigm in the development, i don´t quite see how the "programming
> environment" features could get out of hand too badly. It makes sure the
> learning curve will stay flat in the beginning.
>
> Interfacing gnuplot with other programs is however an issue today, imo.
> That would be, for example, much easier with an array variable that can
> be read in from a file, see above, and also keep away further need for
> new complex features.
>
>    Karl
>
>

There have been a number of times when I've regretted the lack of an 
index array structure in gnuplot.

Sometimes I've been able to hack something to do what I want using some 
kind of iteration but this often involves changing something else to 
shoehorn it into a form where it can be done with an iteration.

I do think this would be a useful addition.

Peter.

Re: Unintended consequences - things to think about

[Ethan A M. <sf...@us...>]

On Thursday, 13 November, 2014 09:20:52 Philipp K. Janert wrote:
> 
> Here are some things to think about:
> 
> 1) We now have loops. But we don't have an iterable data structure.
> No, wait, we have one - we fake it as white-space separated 
> string. (I admit the ingenuity of the thought, but - how kludgy is THAT?)

Beauty is in the eye of the beholder.
Gnuplot has 3 types of variables: integer, string, and complex

There is one simple iterator for integer variables:
  for [ i = min : max : step]

There is one simple iterator for string variables:
  for [s in "A B C D"]

Arguably there is room for an interator over complex values,
but so far no one has convinced me that it is needed.
 
> So, before too long, the loops will beget
> an array. That seems certain.

[shrug]
I can see reasons one might want to have arrays,
but the existence of loops is not a compelling one.

I'd be more inclined to think about adding support
for lists or unordered sets, with a corresponding 
iterator
  for [element in set]


> 2) Once we have arrays, there will be a
> desire to load files into them. Otherwise,
> why have arrays, right? So, arrays will
> beget a "read_file" feature. 
> 
> 3) Now that we can read file(s) into 
> arrays, it's natural that we want to 
> operate on them. One could do that with
> loops, but (again) it's natural to do 
> things like array1 + array2, right? 
> 
> 4) Whereas items 1-3 lie in the future, the 
> following is already an issue today: we have
> loops and conditionals, so it seems natural
> to write functions. Well - we don't HAVE
> functions, but we can fake them as scripts
> that we load with "call". 

"call" is a prime example of a gnuplot feature that I have ignored
for decades.  We continue to support the "call" mechanism
(there's that backwards compatibility promise)
but I honestly don't see a good use for it.
Version 5 provides a slightly more standard/familiar variant that
treats the call parameters like shell variables, with save+restore
of the values. But I doubt I'll ever want to use the new variant either.

> Problem is that
> these scripts don't define local variables -
> any variable I use in my script will become
> a variable in the gnuplot session. To prevent
> clobbering existing variables, I therefore
> need to prepend a prefix to the name of each
> variable "MY_SCRIPT_i", "MY_SCRIPT_j", and so
> on. Very kludgy. Hence, it is reasonable to
> ask for functions with local variables.

Now it seems you are not so much arguing for discussing
new features as you are implicitly arguing that we should
get rid of a feature that has been there for 25 years.
Isn't this the Python mistake?   To break old scripts as a 
consequence of removing an old feature just because
you thought of a better way to do something similar?

If you don't like "call", don't use it.

If you want to propose a syntax for defining fully-scoped function 
blocks, go right ahead (although maybe you are advocating against
this rather than for this; it's hard to tell).

> My point, in case this still isn't clear:
> 
> Once you have one programming feature (say:
> loops), there is a natural tendency for them 
> to require all other programming features 
> (data structures, functions, local scope).
> 
> Therefore, what starts like a local convenience 
> today will - invariably - grow until it has 
> acquired all the features that are already 
> provided by R, matlab, Octave, Perl, Python, Ruby.
> 
> And THAT I think is the wrong direction. If 
> somebody wants R, why would they use gnuplot? 

Because they find it easier to make nice plots?

Let's flip that question around.  If someone is happy with
gnuplot except for some hypothetical missing feature
provided by R, should we not consider adding that missing
feature to gnuplot?

> I also think that it is a misdirection of effort: 
> do we really need to figure out how to design 
> functions with locally scoped variables? This
> has already been done, several times. Would it
> not be better to leverage other people's work?
> 
> If gnuplot needs to acquire programming features
> to stay competitive, then it should try to borrow
> them from somewhere else, where all these problems
> have already been solved - rather than inventing
> the wheel (and poorly, at that).

So how about instead of dragging in lots of bad examples,
you suggest what nice feature from some other language
or program we should consider adopting.

	Ethan (not actually finding this discussion very useful)

Re: Unintended consequences - things to think about

[<pl...@pi...>]

On 11/13/14 20:38, Ethan A Merritt wrote:
>> And THAT I think is the wrong direction. If
>> >somebody wants R, why would they use gnuplot?

> Because they find it easier to make nice plots?

I did try using R at one stage and had to dump the result out to a text 
file and use gnuplot to get what I wanted. So I agree with Ethan on that 
one.

Then I found that R has a habit of changing your data to what it 
"thinks" you wanted it be. Bizarrely, I ended up with more data points 
than I started with.

I did not bother working out exactly where it got the extra data from to 
insert because I prefer to spend time working on my own code rather 
than double checking my data at every step to ensure that the "language" 
I'm using has not decided change it for me.

I don't want a programming language screwing around any more that I want 
my girlfriend screwing around. In fact, I may have more tolerance for 
the latter.

Plus the documentation is terse and incomplete half the time so I never 
really know even what  it is supposed to be doing.



YMMV.