gnuplot development

Ethan Merritt wrote:
[...]

> For one thing, we don't currently have any easy way to undefine all
> these variables.  "show var GPSTAT" would show all of them, but
> "undefine GPSTAT" doesn't get rid of them.  That's fixable, I suppose,
> but I don't like the variables for another reason...

> 4) I suggest adding a mechanism for explicitly clearing out the set of
> stats calculations.  One obvious syntax is
>     reset stats

This one deserves generalization.  Unless I missed something, we 
currently don't have a command to get rid of any variable (short of 
'exit' and starting a new session), whereas the number of variables we 
have gnuplot create by itself seems to be increasing all the time (first 
"fit" results, then GPVAL_*, now possibly GPSTAT_*).

I think a generic

	unset variable {<name>| pattern <regex> | fit | stats}

command would be in order.  I would prefer 'unset' over 'reset' here 
because it matches 'show variables' a bit better than 'reset'.  It 
should probably be extended to user-defined functions, too.  And maybe 
we should even allow

	set variable <var>=<expr>
and
	set function <name>(<arguments>)=<expression>

as an optional syntax instead of the usual <var>=<expr> etc., too.

Philipp K. Janert <janert <at> ieee.org> writes:

> 
> On Monday 09 November 2009 07:29:10 am Jonathan Thornburg wrote:
> > On Mon, 9 Nov 2009, Philipp K. Janert wrote:
> > > There was a question whether the stats command
> > > works on "a sample or a population". Neither!
> > > It works on a a DATA FILE, like the rest of gnuplot.
> > > (And therefore there is no ambiguity.)
> >
> > I'm sorry, but I still don't know whether this means it uses N or N-1
> > weighting.  I would find it useful to have *both* printed out -- each is
> > valuable in different circumstances.
> 
> I see. That makes more sense.
> 
> Currently, the stats command divides by N.

I don't really see the difference: since the sum of whatever quantity you want
is reported, as is the number of records, you can define either sum_x / records
or sum_x / (records-1). Similar argument applies to the standard deviations and
the like. I don't want to add new variables for quantities that can easily be
calculated from existing ones. 
Best,
Zoltán

Tait <gnuplot-devel <at> t41t.com> writes:

> 
> Mean as used here seems to be the arithmetic mean. What about the geometric
> mean? (Or harmonic mean, or any of the other types of averages?)

Harmonic mean is easily done with the present stats command as 
stats 'foo' u 1:(1.0/$2) noout var
harmonic = records / sum_y

> I have always addressed these sorts of issues by using a tool that's
> designed for manipulating arbitrary data sets (Perl in my case, but there
> are others). This has the advantage of providing infinite flexibility,

That is sort of a platform-dependent solution for a problem that comes up too
often, but Philipp has already discussed this issue at length, I believe.

> I wonder, rather than providing a restricted set of pre-defined functions,
> is there a way to allow the user to provide a formula or expression that
> will be applied across multiple rows? Then the user could calculate the
> mean (whatever that means to their application) or standard deviation or
> some other arbitrary metric on their own.

stats 'foo' u ($2*$3+cos($4)) should work as it is, if that is what you meant. A
fairly large set of quantities can be calculated using the variables that are
produced by stats, if the proper function is applied to the columns beforehand. 
Best,
Zoltán

On Tuesday 10 November 2009 13:00:16 Hans-Bernhard Bröker wrote:

> This one deserves generalization.  Unless I missed something, we 
> currently don't have a command to get rid of any variable (short of 
> 'exit' and starting a new session), 

Yeah, we do.  You can
   undefine VARNAME
But it doesn't take wildcards.

Perhaps it should do the same as the "show var PREFIX" command,
and treat the string as a leading prefix to the full variable name.

> whereas the number of variables we  
> have gnuplot create by itself seems to be increasing all the time (first 
> "fit" results, then GPVAL_*, now possibly GPSTAT_*).
> 
> I think a generic
> 
> 	unset variable {<name>| pattern <regex> | fit | stats}
> 
> command would be in order.  I would prefer 'unset' over 'reset' here 
> because it matches 'show variables' a bit better than 'reset'.  

Hmm.  But it isn't  "set var foo",  it's "var = foo".
So "unset" is misleading.

> It should probably be extended to user-defined functions, too.

Yes.  Good point.

But there is a possible "gotcha".  If you undefine a function that is
called by another previously-defined function, bad things could happen.


> And maybe  
> we should even allow
> 
> 	set variable <var>=<expr>
> and
> 	set function <name>(<arguments>)=<expression>
> 
> as an optional syntax instead of the usual <var>=<expr> etc., too.

Yes, that would be the other way to justify use of "unset" :-)
But it seems a more drastic change than extending "undefine".

-- 
Ethan A Merritt

I was out all day so I am joining this thread a little later.

>
> I had occasion to use your patch in the real world yesterday.
> I was making figures from data stored in a csv file, with 16 columns
> of data each corresponding to one experiment.  I wanted to normalize
> the curves so that each one filled the range [0:1] when superimposed.

Thanks for giving it a "real-world whirl".

>
> set datafile separator ','
> data = "My Data File"
>
> stats data using 5 variable="col5"
> stats data using 6 variable="col6"
> ...
> stats data using 20 variable="col20"
>
> plot ....
>
> All great.
> But having to type 16 separate commands and then hunt for the
> results to construct a single complicated plotting command was tedious.
> So I have several specific suggestions for improvement.
>
> 1) The syntax
>     stats data using 20 variable="col20"
> did not do at all what I expected.  I expected to get variables
>     col20_max_x  and so on,
> but what I actually got was variables with embedded quote marks:
>   "col20"max_x
> Trying to embed this in a plot command was a pain.

The quotes are your addition. We don't expect them,
but we don't actively remove them. Maybe we should -
I admit that there is a user expectation that "there should
be quotes". (I know, because I found myself adding them.)
On the other hand, I don't want to have a command that is
too smart (removing quotes silently, but leaving everything
else alone...)

>
> I suggest that the syntax should be
>     stats data using <foo> name <string>
> and should produce variables named
>     GPSTAT_string_max_x
>     GPSTAT_string_npoints  (NB: not ...nrecords)
> Except that I really don't like this mechanism very much.

Zortan and I discussed this. Our idea was to keep the
variable names short and user-friendly. The GPSTAT_
prefix really does not help very much. If users want to
avoid polluting their own namespace, we give them the
ability to choose their own prefixes.

>
> For one thing, we don't currently have any easy way to undefine all
> these variables.  "show var GPSTAT" would show all of them, but
> "undefine GPSTAT" doesn't get rid of them.  That's fixable, I suppose,
> but I don't like the variables for another reason...
>
> 2)  Here's the command I want to issue in the end:
>
>    plot for [col=5:20] data using (column(4)) :  (column(col) /
> statmax(col))  \ title label(col)
>
> I doesn't work, because I can't figure out how to define a function
> statmax(col) that retrieves the desired value.  We don't have a user-level
> command in gnuplot that will retrieve the value of a gnuplot variable by
> its string name. Yesterday I had to forego the iterator and type in a
> 16-line plot command instead.

If I see this correctly, mostly you would like to add 
support for iteration into the stats command? This
is certainly something we can think about.

>
> So I want to request a different mechanism for storing and retrieving the
> stats values.  You've seen this before, but here it comes again:
> I don't want dozens of variables to be created by every stats command,
> because they are too hard to retrieve inside a script.  Instead I want each
> stats command to load a structure, and I want a set of functions that
> retrieve the previously calculated stats values, indexed by name.  If you
> want to load a named variable from one of the stats values, fine.  Just say
>    Run5_xmin = statmin("Run5")
> That will persist across a save/load sequence, for instance, even though
> the internal stats structures will not.

Why is that goodness? I don't understand the 
motivation here. Why do you want to go the
roundabout way (and force the user through 
this detour) of accessing variables through 
functions, rather than as variables? 

The "prefix" that we offer for variable names 
serves exactly the same purpose as the data
structure that you refer to: a logical grouping.

>
> 3)  For convenience, the stats command should accept an iterator.
> My plots yesterday could then have been created in two commands:
>
>    stats for [col=5:20] data using col name "Run".col
>    plot  for [col=5:20] data using (column(4)) : \
>          (column(col) / statmax("Run".col)) \
>          title sprintf("Run%d",col)
>
> Note that "Run".col  is the same as sprintf("Run%d",col).
>
> 4) I suggest adding a mechanism for explicitly clearing out the set of
> stats calculations.  One obvious syntax is
>     reset stats

Yes, and in fact Zoltan and I have implemented a
function to do that, but not hooked it up.

On Tuesday 10 November 2009 01:00:16 pm Hans-Bernhard Bröker wrote:
> Ethan Merritt wrote:
> [...]
>
> > For one thing, we don't currently have any easy way to undefine all
> > these variables.  "show var GPSTAT" would show all of them, but
> > "undefine GPSTAT" doesn't get rid of them.  That's fixable, I suppose,
> > but I don't like the variables for another reason...
> >
> > 4) I suggest adding a mechanism for explicitly clearing out the set of
> > stats calculations.  One obvious syntax is
> >     reset stats
>
> This one deserves generalization.  Unless I missed something, we
> currently don't have a command to get rid of any variable (short of
> 'exit' and starting a new session), whereas the number of variables we
> have gnuplot create by itself seems to be increasing all the time (first
> "fit" results, then GPVAL_*, now possibly GPSTAT_*).
>
> I think a generic
>
> 	unset variable {<name>| pattern <regex> | fit | stats}
>

I fully agree. Zoltan and I discussed this and even
implemented a function that will do that. We just
don't currently expose it as a command - partially
because we did not want to add yet another user-level
command. 

But the idea of overloading unset for this purpose is
a good one.

> command would be in order.  I would prefer 'unset' over 'reset' here
> because it matches 'show variables' a bit better than 'reset'.  It
> should probably be extended to user-defined functions, too.  And maybe
> we should even allow
>
> 	set variable <var>=<expr>
> and
> 	set function <name>(<arguments>)=<expression>
>
> as an optional syntax instead of the usual <var>=<expr> etc., too.
>
>
> ---------------------------------------------------------------------------
>--- Let Crystal Reports handle the reporting - Free Crystal Reports 2008
> 30-Day trial. Simplify your report design, integration and deployment - and
> focus on what you do best, core application coding. Discover what's new
> with Crystal Reports now.  http://p.sf.net/sfu/bobj-july
> _______________________________________________
> gnuplot-beta mailing list
> gnuplot-beta@...
> https://lists.sourceforge.net/lists/listinfo/gnuplot-beta

On Tuesday 10 November 2009 01:32:06 pm Ethan Merritt wrote:
> On Tuesday 10 November 2009 13:00:16 Hans-Bernhard Bröker wrote:
> > This one deserves generalization.  Unless I missed something, we
> > currently don't have a command to get rid of any variable (short of
> > 'exit' and starting a new session),
>
> Yeah, we do.  You can
>    undefine VARNAME
> But it doesn't take wildcards.

Side question: is that behavior documented
somewhere? I did not know about it, either.

>
> Perhaps it should do the same as the "show var PREFIX" command,
> and treat the string as a leading prefix to the full variable name.

That's what we were thinking. (I will admit that
I am reluctant to implement a regular expression
parser. It seems like overkill.)

I tend to prefer "unset" over "reset" - simply 
because I am already used to "unset" taking
arguments, whereas "reset" does not.

>
> > whereas the number of variables we
> > have gnuplot create by itself seems to be increasing all the time (first
> > "fit" results, then GPVAL_*, now possibly GPSTAT_*).
> >
> > I think a generic
> >
> > 	unset variable {<name>| pattern <regex> | fit | stats}
> >
> > command would be in order.  I would prefer 'unset' over 'reset' here
> > because it matches 'show variables' a bit better than 'reset'.
>
> Hmm.  But it isn't  "set var foo",  it's "var = foo".
> So "unset" is misleading.
>
> > It should probably be extended to user-defined functions, too.
>
> Yes.  Good point.
>
> But there is a possible "gotcha".  If you undefine a function that is
> called by another previously-defined function, bad things could happen.
>
> > And maybe
> > we should even allow
> >
> > 	set variable <var>=<expr>
> > and
> > 	set function <name>(<arguments>)=<expression>
> >
> > as an optional syntax instead of the usual <var>=<expr> etc., too.
>
> Yes, that would be the other way to justify use of "unset" :-)
> But it seems a more drastic change than extending "undefine".

> >
> > Yeah, we do.  You can
> >    undefine VARNAME
> > But it doesn't take wildcards.
>
>
> I tend to prefer "unset" over "reset" - simply
> because I am already used to "unset" taking
> arguments, whereas "reset" does not.
>

I will correct myself. If we already have the 
ability to undefine a variable, then we should
use that, rather than introducing an additional
method.

How about:
	undefine foo*
	undefine *foo
	undefine foo*bar
with the obvious meaning of the wildcard?

On Tuesday 10 November 2009, Philipp K. Janert wrote:

> > 1) The syntax
> >     stats data using 20 variable="col20"
> > did not do at all what I expected.  I expected to get variables
> >     col20_max_x  and so on,
> > but what I actually got was variables with embedded quote marks:
> >   "col20"max_x
> > Trying to embed this in a plot command was a pain.
> 
> The quotes are your addition. We don't expect them,
> but we don't actively remove them. Maybe we should -
> I admit that there is a user expectation that "there should
> be quotes".

Since 4.0 we support string variables. That means wherever a string
is required in the input command, it is acceptable to provide
a string constant, a string variable, or a string-valued function.

So 
  A = "mydata"
  stats "file.dat" using 1 variable=A

must expand A to find "mydata", not use it as an unmarked constant.
All strings should be parsed using the routine try_to_get_string(),
which handles the three cases.
Also, normal commands do not use = signs.
So the source code should be something like:

   char *prefix= NULL;
   if (equals(c_token,"variable")) {
        c_token++;
        prefix = try_to_get_string();
   }

> > 2)  Here's the command I want to issue in the end:
> >
> >    plot for [col=5:20] data using (column(4)) :  \
> >             column(col) / statmax(col)) 
> >
> > It doesn't work, because I can't figure out how to define a function
> > statmax(col) that retrieves the desired value.  We don't have a user-level
> > command in gnuplot that will retrieve the value of a gnuplot variable by
> > its string name. Yesterday I had to forego the iterator and type in a
> > 16-line plot command instead.
> 
> If I see this correctly, mostly you would like to add 
> support for iteration into the stats command? This
> is certainly something we can think about.

Iteration is a major reason.  But the same problem arises whenever you
have constructed the variable name via a script.  How do you insert the
value of that variable back into another command?

> >
> > So I want to request a different mechanism for storing and retrieving the
> > stats values.  You've seen this before, but here it comes again:
> > I don't want dozens of variables to be created by every stats command,
> > because they are too hard to retrieve inside a script.  Instead I want each
> > stats command to load a structure, and I want a set of functions that
> > retrieve the previously calculated stats values, indexed by name.  If you
> > want to load a named variable from one of the stats values, fine.  Just say
> >    Run5_xmin = statmin("Run5")
> > That will persist across a save/load sequence, for instance, even though
> > the internal stats structures will not.
> 
> Why is that goodness? I don't understand the 
> motivation here. 

I thought this was something you wanted.  You said gnuplot should not
become statefull, which I take to mean that save/load should get you 
back to where you were without having to replay the whole history of
commands.

> Why do you want to go the
> roundabout way (and force the user through 
> this detour) of accessing variables through 
> functions, rather than as variables? 

Because, as I noted above, you cannot currently access their value
by name in a script.  I agree there are other possible solutions to
the problem.

> > The quotes are your addition. We don't expect them,
> > but we don't actively remove them. Maybe we should -
> > I admit that there is a user expectation that "there should
> > be quotes".
>
> Since 4.0 we support string variables. That means wherever a string
> is required in the input command, it is acceptable to provide
> a string constant, a string variable, or a string-valued function.
>
> So
>   A = "mydata"
>   stats "file.dat" using 1 variable=A
>
> must expand A to find "mydata", not use it as an unmarked constant.
> All strings should be parsed using the routine try_to_get_string(),
> which handles the three cases.
> Also, normal commands do not use = signs.

Good point on the string variable issue - I did not
think of that. 

There is a reason for the equality sign, though:
it indicates that the next token is the prefix - 
because we have chosen to make the prefix
optional. The equality sign is a way of telling 
gnuplot that the next token is a prefix, not the
next keyword.

(I admit that the equality sign is unusual and I did
hesitate a little. But it does provide a simple solution
to this particular problem.)

I don't want to make the prefix mandatory. For
convenience, it seems that in many cases it won't
be needed.

On Tuesday 10 November 2009 06:22:17 pm Philipp K. Janert wrote:
> > > The quotes are your addition. We don't expect them,
> > > but we don't actively remove them. Maybe we should -
> > > I admit that there is a user expectation that "there should
> > > be quotes".
> >
> > Since 4.0 we support string variables. That means wherever a string
> > is required in the input command, it is acceptable to provide
> > a string constant, a string variable, or a string-valued function.

Although there is still a confusion: you write
	"where a string is required"

In this example, no string is required. A bareword
is required. It is your assumption that it should be
a string. (And it is now a discussion item whether 
it should be a string - in which case the string needs
to be handled properly, admittedly.)

> >
> > So
> >   A = "mydata"
> >   stats "file.dat" using 1 variable=A
> >
> > must expand A to find "mydata", not use it as an unmarked constant.
> > All strings should be parsed using the routine try_to_get_string(),
> > which handles the three cases.
> > Also, normal commands do not use = signs.
>
> Good point on the string variable issue - I did not
> think of that.
>
> There is a reason for the equality sign, though:
> it indicates that the next token is the prefix -
> because we have chosen to make the prefix
> optional. The equality sign is a way of telling
> gnuplot that the next token is a prefix, not the
> next keyword.
>
> (I admit that the equality sign is unusual and I did
> hesitate a little. But it does provide a simple solution
> to this particular problem.)
>
> I don't want to make the prefix mandatory. For
> convenience, it seems that in many cases it won't
> be needed.
>
> ---------------------------------------------------------------------------
>--- Let Crystal Reports handle the reporting - Free Crystal Reports 2008
> 30-Day trial. Simplify your report design, integration and deployment - and
> focus on what you do best, core application coding. Discover what's new
> with Crystal Reports now.  http://p.sf.net/sfu/bobj-july
> _______________________________________________
> gnuplot-beta mailing list
> gnuplot-beta@...
> https://lists.sourceforge.net/lists/listinfo/gnuplot-beta

On Tuesday 10 November 2009, Philipp K. Janert wrote:
> > >
> > > Since 4.0 we support string variables. That means wherever a string
> > > is required in the input command, it is acceptable to provide
> > > a string constant, a string variable, or a string-valued function.
> 
> Although there is still a confusion: you write
> 	"where a string is required"
> 
> In this example, no string is required. 

Of course a string is required.  It is going to become the
first N characters of a longer string.  How could it be anything
other than a string itself?

> A bareword
> is required. It is your assumption that it should be
> a string. (And it is now a discussion item whether 
> it should be a string - in which case the string needs
> to be handled properly, admittedly.)
> 
> > >
> > > So
> > >   A = "mydata"
> > >   stats "file.dat" using 1 variable=A
> > >
> > > must expand A to find "mydata", not use it as an unmarked constant.
> > > All strings should be parsed using the routine try_to_get_string(),
> > > which handles the three cases.
> > > Also, normal commands do not use = signs.
> >
> > Good point on the string variable issue - I did not
> > think of that.
> >
> > There is a reason for the equality sign, though:
> > it indicates that the next token is the prefix -
> > because we have chosen to make the prefix
> > optional. The equality sign is a way of telling
> > gnuplot that the next token is a prefix, not the
> > next keyword.

Not following you here.
The keyword itself can be optional - you don't have to provide a prefix.
But if you include the keyword in your command, then the next token
must be a string.  Where does the = sign come in?

[maybe "prefix" is a better keyword than "variable"]


> > (I admit that the equality sign is unusual and I did
> > hesitate a little. But it does provide a simple solution
> > to this particular problem.)
> >
> > I don't want to make the prefix mandatory. For
> > convenience, it seems that in many cases it won't
> > be needed.

So let it default to an empty string.

On Tuesday 10 November 2009 07:10:51 pm you wrote:
> On Tuesday 10 November 2009, Philipp K. Janert wrote:
> > > > Since 4.0 we support string variables. That means wherever a string
> > > > is required in the input command, it is acceptable to provide
> > > > a string constant, a string variable, or a string-valued function.
> >
> > Although there is still a confusion: you write
> > 	"where a string is required"
> >
> > In this example, no string is required.
>
> Of course a string is required.  It is going to become the
> first N characters of a longer string.  How could it be anything
> other than a string itself?

It's not a string as far as the gnuplot session is
concerned. Here:
	plot "file" using 1:2
In this example, "file" is a string within the gnuplot
session. But using is not a string in the same way -
it's a keyword.

Similarly:
	a = 1
Here, a is not a string within the gnuplot session.
It's a variable name.

In the same way, our prefix is not a string within 
the gnuplot session. It's an (unquoted) bareword.
The best way to see this is that there is no need 
to quote it and any quotes are not stripped out.

(This is a totally different question whether any of
this is IMPLEMENTED as a C string. Of course it is.
But that's not what I am talking about.)

>
> > A bareword
> > is required. It is your assumption that it should be
> > a string. (And it is now a discussion item whether
> > it should be a string - in which case the string needs
> > to be handled properly, admittedly.)
> >
> > > > So
> > > >   A = "mydata"
> > > >   stats "file.dat" using 1 variable=A
> > > >
> > > > must expand A to find "mydata", not use it as an unmarked constant.
> > > > All strings should be parsed using the routine try_to_get_string(),
> > > > which handles the three cases.
> > > > Also, normal commands do not use = signs.
> > >
> > > Good point on the string variable issue - I did not
> > > think of that.
> > >
> > > There is a reason for the equality sign, though:
> > > it indicates that the next token is the prefix -
> > > because we have chosen to make the prefix
> > > optional. The equality sign is a way of telling
> > > gnuplot that the next token is a prefix, not the
> > > next keyword.
>
> Not following you here.
> The keyword itself can be optional - you don't have to provide a prefix.

Ha! But :
	stats "file"
and
	stats "file" var
have different behavior!

In the first case, no assignment to variables is made.
Only in the second do we assign to variables. (Without
a prefix.)

So, how do I distinguish without the equality sign (or
another keyword) between:
	stats "file" var noout
and 
	stats "file" var foo

Here, the first is supposed to assign to variables and 
not print to screen (keyword "noout"), whereas the former
assigns to variables with prefix foo, but does not print to 
screen? To distinguish these cases, I need to tell gnuplot
that the foo in the second cases "belongs to" var. The simplest
way I could think of was to use the equality sign.

> But if you include the keyword in your command, then the next token
> must be a string.  Where does the = sign come in?
>
> [maybe "prefix" is a better keyword than "variable"]
>
> > > (I admit that the equality sign is unusual and I did
> > > hesitate a little. But it does provide a simple solution
> > > to this particular problem.)
> > >
> > > I don't want to make the prefix mandatory. For
> > > convenience, it seems that in many cases it won't
> > > be needed.
>
> So let it default to an empty string.

On Tuesday 10 November 2009, Philipp K. Janert wrote:
> On Tuesday 10 November 2009 07:10:51 pm you wrote:
> > On Tuesday 10 November 2009, Philipp K. Janert wrote:
> > > > > Since 4.0 we support string variables. That means wherever a string
> > > > > is required in the input command, it is acceptable to provide
> > > > > a string constant, a string variable, or a string-valued function.
> > >
> > > Although there is still a confusion: you write
> > > 	"where a string is required"
> > >
> > > In this example, no string is required.
> >
> > Of course a string is required.  It is going to become the
> > first N characters of a longer string.  How could it be anything
> > other than a string itself?


> >
> > > A bareword
> > > is required. It is your assumption that it should be
> > > a string. (And it is now a discussion item whether
> > > it should be a string - in which case the string needs
> > > to be handled properly, admittedly.)
> > >
> > > > > So
> > > > >   A = "mydata"
> > > > >   stats "file.dat" using 1 variable=A
> > > > >
> > > > > must expand A to find "mydata", not use it as an unmarked constant.
> > > > > All strings should be parsed using the routine try_to_get_string(),
> > > > > which handles the three cases.
> > > > > Also, normal commands do not use = signs.
> > > >
> > > > Good point on the string variable issue - I did not
> > > > think of that.
> > > >
> > > > There is a reason for the equality sign, though:
> > > > it indicates that the next token is the prefix -
> > > > because we have chosen to make the prefix
> > > > optional. The equality sign is a way of telling
> > > > gnuplot that the next token is a prefix, not the
> > > > next keyword.
> >
> > Not following you here.
> > The keyword itself can be optional - you don't have to provide a prefix.
> 
> Ha! But :
> 	stats "file"
> and
> 	stats "file" var
> have different behavior!
> 
> In the first case, no assignment to variables is made.
> Only in the second do we assign to variables. (Without
> a prefix.)
> 
> So, how do I distinguish without the equality sign (or
> another keyword) between:
> 	stats "file" var noout
> and 
> 	stats "file" var foo

try_to_get_string() will return NULL if noout is not a currently
defined string variable.  If you define a string variable that is
the same as a keyword then yes, you could create a problem.
If that bothers you, you could explicitly test whether the next
token is "noout".


> Here, the first is supposed to assign to variables and 
> not print to screen (keyword "noout"), whereas the former
> assigns to variables with prefix foo, but does not print to 
> screen? To distinguish these cases, I need to tell gnuplot
> that the foo in the second cases "belongs to" var. The simplest
> way I could think of was to use the equality sign.
> 
> > But if you include the keyword in your command, then the next token
> > must be a string.  Where does the = sign come in?
> >
> > [maybe "prefix" is a better keyword than "variable"]
> >
> > > > (I admit that the equality sign is unusual and I did
> > > > hesitate a little. But it does provide a simple solution
> > > > to this particular problem.)
> > > >
> > > > I don't want to make the prefix mandatory. For
> > > > convenience, it seems that in many cases it won't
> > > > be needed.
> >
> > So let it default to an empty string.

Same answer as before, with an explicit default case:

   char *prefix= NULL;
   if (equals(c_token,"variable")) {
        c_token++;
        prefix = try_to_get_string();
   }
   if (!prefix)
	prefix = gp_strdup("");


...

   free(prefix);

Ethan Merritt wrote:
> 2)  Here's the command I want to issue in the end:
> 
>    plot for [col=5:20] data using (column(4)) :  (column(col) / statmax(col))  \
>         title label(col)
> 
> I doesn't work, because I can't figure out how to define a function statmax(col)
> that retrieves the desired value.  We don't have a user-level command in gnuplot
> that will retrieve the value of a gnuplot variable by its string name.

Maybe more to the point, we lack array variables, which are exactly what 
this really would call for.  Collections you loop over to retrieve 
individual elements by numbers are just that: arrays.  Whether they be 
emulated by fancy variable name construction from fragments, or a 
function taking the index as an argument, they're still just arrays.

OTOH, arrays (a.k.a. vectors, and eventually matrices) would be one more 
step towards mimicking MatLab.  Which we used to say we weren't going to do.

Maybe all those quirks popping up are to warn us that this is not a 
direction we should continue walking in.

Ethan Merritt wrote:

> Hmm.  But it isn't  "set var foo",  it's "var = foo".
> So "unset" is misleading.

But we _do_ "show var foo", so "{set|unset} var foo" would match the 
usual relation between show/set/unset rather more nicely than a new 
command of its own.

> But there is a possible "gotcha".  If you undefine a function that is
> called by another previously-defined function, bad things could happen.

None of those would be worse than the bad things already happening by

*) never defining a variable used by some function in the first place
*) never defining a function used by anther function in the first place
*) undefining a variable that was referenced by some function
*) never defining a variable used to set the value of another variable
*) never defining a function called to set the value of a variable

> > Mean as used here seems to be the arithmetic mean. What about the geometric
> > mean? (Or harmonic mean, or any of the other types of averages?)
>
> Harmonic mean is easily done with the present stats command as
> stats 'foo' u 1:(1.0/$2) noout var
> harmonic = records / sum_y

I was really trying to make two points here. One, that the choice of name
is ambiguous, and perhaps a more specific name would serve better. Second,
the choice of an arithmetic mean (out of all the possible formulas one
could use for expected value) seems arbitrary. I can almost convince myself
that arithmetic mean is possibly the most common, so maybe that justifies
its selection. But arithmetic mean, as you've pointed out with the harmonic
mean, can be calculated separately via sum_y/records. Others like the
commonly-used geometric mean can't be calculated from other values exposed
by stats.

> > I wonder, rather than providing a restricted set of pre-defined functions,
> > is there a way to allow the user to provide a formula or expression that
> > will be applied across multiple rows? Then the user could calculate the
> > mean (whatever that means to their application) or standard deviation or
> > some other arbitrary metric on their own.
>
> stats 'foo' u ($2*$3+cos($4)) should work as it is, if that is what you meant. A
> fairly large set of quantities can be calculated using the variables that are
> produced by stats, if the proper function is applied to the columns beforehand.

That is not at all what I meant. The stats command (and plot, and others) 
provide access to other data values on the same row, via $column-number. 
There is, however, no way to access values on _other_ rows. I can't plot 
the delta between the current $2 and the $2 of the previous row, for 
example. A general way to provide formulas or expressions that operate 
across multiple rows would be more flexible and also make "stats" 
unnecessary.

The tangent about Perl was intended to be in the same vein as Ethan's
comment that gnuplot is not MatLab or Mathematica or MathCad or R, and 
we shouldn't try to be. Maybe stats is trying to make gnuplot do too
much. The danger of using an 80% tool is that it will be abused and
expanded to try and do 100% of jobs, when the user should have 
switched to a more appropriate tool long ago. 

Maybe an alternate (and more useful?) way to provide the stats 
functionality is to add a contrib directory to the gnuplot 
distribution in which are placed stand-alone utilities like stats
that perform useful transformations on and summaries of 
gnuplot-looking data files, using gnuplot-looking syntaxes. 
Someone wanting to know the record count of a data file could 
(in gnuplot) do records=`contrib/countrows -using 3 -every ::2::2`.
This avoids syntactic complexity in gnuplot, is more flexible
while (I think) solving the same problems. It can be easily
expanded to include new and improved functionality without even
needing to recompile gnuplot itself.

Tait

On Wednesday 11 November 2009, Tait wrote:
>
> > stats 'foo' u ($2*$3+cos($4)) should work as it is, if that is what you meant. A
> > fairly large set of quantities can be calculated using the variables that are
> > produced by stats, if the proper function is applied to the columns beforehand.
> 
> That is not at all what I meant. The stats command (and plot, and others) 
> provide access to other data values on the same row, via $column-number. 
> There is, however, no way to access values on _other_ rows. I can't plot 
> the delta between the current $2 and the $2 of the previous row, for 
> example. A general way to provide formulas or expressions that operate 
> across multiple rows would be more flexible and also make "stats" 
> unnecessary.

You can do this while plotting. See "running_avg.dem".
Whether it's worth making the syntax of the stats command more like the
plot command, that's another question.  Or for that matter, whether
it's better to have a separate stats command or to incorporate it into
the existing plot command.

> The tangent about Perl was intended to be in the same vein as Ethan's
> comment that gnuplot is not MatLab or Mathematica or MathCad or R, and 
> we shouldn't try to be.

The wasn't me. I think it's silly to rule out a useful addition on 
that basis. I think we should seriously consider any addition
that strengthens gnuplot's ability to create plots.  I do want to hear a
strong case, however, that the stats command needs to be an internal part
of gnuplot rather than an external script.  I currently do this sort of
thing in an external perl script (or in R). But using R to calculate the
min/mean/max is ridiculous overkill, and I'd still have to get the
information back into gnuplot in order to generate the plots.
 
> Maybe an alternate (and more useful?) way to provide the stats 
> functionality is to add a contrib directory to the gnuplot 
> distribution in which are placed stand-alone utilities like stats
> that perform useful transformations on and summaries of 
> gnuplot-looking data files, using gnuplot-looking syntaxes. 

Exactly.  I'm not 100% convinced that the initial set of capabilities
in the "stats" command justifies including it in the core code rather
than just running an external script.  But if further integration with
the plotting code gives additional benefits over running an external 
script, so be it.

> > The tangent about Perl was intended to be in the same vein as Ethan's
> > comment that gnuplot is not MatLab or Mathematica or MathCad or R, and
> > we shouldn't try to be.
>
> The wasn't me. I think it's silly to rule out a useful addition on
> that basis. I think we should seriously consider any addition
> that strengthens gnuplot's ability to create plots.  I do want to hear a
> strong case, however, that the stats command needs to be an internal part
> of gnuplot rather than an external script.  I currently do this sort of
> thing in an external perl script (or in R). But using R to calculate the
> min/mean/max is ridiculous overkill, and I'd still have to get the
> information back into gnuplot in order to generate the plots.

Strong case is always a little in the eye of the beholder.
But here are my arguments:

1) Convenience
I do this stuff w/ external scripts, too, and I always find it
annoying that I have to pop up a different window, run my
little script, copy and paste the results back into gnuplot...

2) Multiplatform
Exactly how do you do any of this if you are NOT on Linux?
(I admit this is a pretty weak argument, but it is not entirely
baseless. People are much less likely to have Perl/Python
installed on their Win box, compared to a standalone gnuplot
binary.)

3) Stand-alone Scripting
Running a separate script from gnuplot is a pain. Cutting
and pasting values manually is not an option for scripts.
Having stuff assigned to variables w/in the gnuplot session
is therefore desirable.

4) Input Parsing
Gnuplot's input parsing is very robust and flexible. It can
eat a lot. My problem with external scripts is that it is more
difficult to make them as robust, which means that I don't
have any "pre-canned" - I write them ad-hoc for each file
format that I am dealing with. But that makes it inconvenient
in the long run. On the other hand, I don't want to own and
maintain my own stats script, when I might as well have this
functionality included in gnuplot (where it also benefits 
everybody else).

I think some of the reluctance comes from the fact that the
set of capabilities of the current stats command is fixed. 

There are two answers to this:

- Should we extend the set of values calculated? Personally,
	the values that we have included cover approximately
	92.5% of what I need. I think that's a pretty good ratio!
	I would also argue that it covers those values that are
	most important for PLOTTING. But I am willing to take
	suggestions for additional quantities. (We can also look
	at what other packages like R do in their "summary"
	functions.)

- Should gnuplot (deep breath) develop a "plugin" architecture?
	So that you could run an external script and assign the
	returns to gnuplot vars in a transparent and convenient
	fashion? 

I think the latter idea is worth a thought, but it is clearly a much
bigger project (not the coding, but designing a good user 
interface). But it seems a little like overkill for what we are trying
to accomplish here.

There is no claim that the stats command will save the world.
But I do claim that it provides enough of a convenience (see
above) to have it included. 

Counter-argument: I'd like to hear a strong argument why it
should NOT be included. ;-)

Best,

		Ph.

On Wednesday 11 November 2009, Philipp K. Janert wrote:
> 
> > > The tangent about Perl was intended to be in the same vein as Ethan's
> > > comment that gnuplot is not MatLab or Mathematica or MathCad or R, and
> > > we shouldn't try to be.
> >
> > That wasn't me. I think it's silly to rule out a useful addition on
> > that basis. I think we should seriously consider any addition
> > that strengthens gnuplot's ability to create plots.  I do want to hear a
> > strong case, however, that the stats command needs to be an internal part
> > of gnuplot rather than an external script.  I currently do this sort of
> > thing in an external perl script (or in R). But using R to calculate the
> > min/mean/max is ridiculous overkill, and I'd still have to get the
> > information back into gnuplot in order to generate the plots.
> 
> Strong case is always a little in the eye of the beholder.
> But here are my arguments:
> 
> 1) Convenience
> I do this stuff w/ external scripts, too, and I always find it
> annoying that I have to pop up a different window, run my
> little script, copy and paste the results back into gnuplot...

But that's what I had to do inside gnuplot itself when I was
making plots on Monday, even _with_ the stats command!  

I had to run the stats command, then copy and paste the min/max
values back from 16 separate invocations back into my plot command.
To make this easier, I ran two copies of gnuplot, using one
to run the stats command and the other to issue plot commands.

If we can make it so that the whole process is scriptable, then
fine.  Otherwise I see zero difference between opening a 
second window to run an external script and opening a second
window to run another copy of gnuplot.

> 2) Multiplatform
> Exactly how do you do any of this if you are NOT on Linux?

I take it you mean "how do you do any of this under Windows?".
Frankly, I don't care.

> 3) Stand-alone Scripting
> Running a separate script from gnuplot is a pain. Cutting
> and pasting values manually is not an option for scripts.
> Having stuff assigned to variables w/in the gnuplot session
> is therefore desirable.

Yes, so let's get that working. Right now it isn't.

> 4) Input Parsing
> 
> - Should we extend the set of values calculated? Personally,
> 	the values that we have included cover approximately
> 	92.5% of what I need. I think that's a pretty good ratio!

I did more plotting yesterday, of the same sort as on Monday.
Again I was creating plots for publication based on multi-column
data in a csv file.  It's still cumbersome.  Not because the
wrong values are calculated, but because they are not easily
tracked and made available in a subsequent plot command.
Given the tools I have on hand, it's actually easier to use an
external script because I can say:
    min_Run3 = system(" some command ")
whereas if I say instead
    stats "datafile" using 3
then I have to manually cut-and-paste to do the assignment.
Again, let's get the variable assigment working properly
so that this annoyance goes away.

> - Should gnuplot (deep breath) develop a "plugin" architecture?
> 	So that you could run an external script and assign the
> 	returns to gnuplot vars in a transparent and convenient
> 	fashion? 

You can already do that for external scripts.
A true plugin architecture using compiled functions would be possible.
In fact there is a patchset on SourceForge that implements it.
But how could we support it cross-platform? 

> Counter-argument: I'd like to hear a strong argument why it
> should NOT be included. ;-)

I think it is a potentially valuable addition.

But until it works in a scripted enviroment, it's strictly worse than
using the system() command to run external tools, or for that 
matter having the script run an external tool first and pass the
values to gnuplot.

Ethan Merritt wrote:

> Of course a string is required.  It is going to become the
> first N characters of a longer string.  

Erm, no.  It's going to become the first N characters of a variable name.

> How could it be anything other than a string itself?

By being a (partial) variable name.  And I don't we ever quote a 
variable name in gnuplot.

>>> There is a reason for the equality sign, though:
>>> it indicates that the next token is the prefix -
>>> because we have chosen to make the prefix
>>> optional. 

So find a different way of signalling that.  It could be as simple as 
changing the default from "no variables created" to "GPSTAT_* variables 
created, so that part of the syntax would become
	
	{ | novariables | prefix <name> }

(wherein leaving out the argument would turn on variables, and if you 
don't want them, you have to turn them off explicitly}

Or leave the default at "novar", and make the syntax

	{ | variables | prefix <name> }

Or have no default at all, and make it

	{ {no}variables | prefix <name> }

On Wednesday 11 November 2009, Hans-Bernhard Bröker wrote:
> Ethan Merritt wrote:
> 
> > Of course a string is required.  It is going to become the
> > first N characters of a longer string.  
> 
> Erm, no.  It's going to become the first N characters of a variable name.

You are missing the point that in order for it to be useful in a script,
I have to be able to construct the prefix dynamically, which means 
I need to handle the prefix itself as a variable.  Hence: a string variable.

> > How could it be anything other than a string itself?
> 
> By being a (partial) variable name.  And I don't [think] we ever quote a 
> variable name in gnuplot.

Yep. It's a partial name of something.  In other words, it is a substring
of the full name.  To specify or manipulate this substring, I want to put
it in a string variable.

If these names can not be manipulated as variables, and it does seem that
nasty issues are poking their heads out of holes in the woodwork,  then
I don't see any point to having them.  Just pick a fixed set of names and
be done with it.  But then we'll have to come up with some other mechanism
for dealing with multi-column data files, and so far I haven't heard any
other suggestions.   

I am talking about real-world cases here.  To plot multiple curves from a
data file I need to be able to iterate over the columns in the data file.  
I can provide sample data and the desired output if you like.  

If the stats functionality were embedded in the 'plot' command itself,
then I could intersperse reloading of the stats output values with the
individual clauses of the plot command:
   plot \
	'datafile' stats ($5), '' using ($5/stats_max), \
      'datafile' stats ($6), '' using ($6/stats_max), \
	...

Note that each time "stats_max" is encountered during evaluation of the 
plot command, it has been reloaded with a new value by the preceding
stats clause. That works fine for iteration:

   plot for [i=5:20] 'datafile' stats (column(i)), '' using (column(i)/stats_max)

But if the stats are collected via a separate command then, as I
discovered the hard way, it is very cumbersome to collect them into a single
multi-column plot command.  This is true even aside from the question of
using iteration, which anyhow isn't possible with the prototype stats
implementation.

Philipp K. Janert wrote:
> 1) Convenience
> I do this stuff w/ external scripts, too, and I always find it
> annoying that I have to pop up a different window, run my
> little script, copy and paste the results back into gnuplot...

Then don't.  At least the wgnuplot_pipes binary can do `backtics` 
command substitution and other piping tricks just fine.  For fully 
command-line style usage, there's always the "console gnuplot for 
windows" build option.

> 2) Multiplatform
> Exactly how do you do any of this if you are NOT on Linux?

The same way you do it on Linux, for just about every gnuplot except the 
traditional, non-pipe wgnuplot.exe on MS Windows.

> (I admit this is a pretty weak argument, but it is not entirely
> baseless. People are much less likely to have Perl/Python
> installed on their Win box, compared to a standalone gnuplot
> binary.)

People who are likely to even arrive at the idea of trying to use them, 
will.

> - Should gnuplot (deep breath) develop a "plugin" architecture?

If anybody can figure out one that is anywhere near as portable as 
gnuplot itself, sure.  And let's not forget that the internals of the 
program would need a _major_ overhaul before they could safely be 
exported to a (possibly malicious) plugin...

> If the stats functionality were embedded in the 'plot' command itself,
> then I could intersperse reloading of the stats output values with the
> individual clauses of the plot command:
>    plot \
> 	'datafile' stats ($5), '' using ($5/stats_max), \
>       'datafile' stats ($6), '' using ($6/stats_max), \
> 	...
> 
>    plot for [i=5:20] 'datafile' stats (column(i)), '' using (column(i)/stats_max)

stat 'datafile' using columns {2 | 2:6 | 2:2:6 | all}

(It can provide GPVAL_DATAFILE_COLUMNS=8.)

BTW, the "stats" command could provide "number of points", which can be 
useful.

---
PM