gnuplot-beta

Re: Problem and solution: Patch for new "stats" command

[Philipp K. J. <ja...@ie...>]

On Tuesday 10 November 2009 07:10:51 pm you wrote:
> On Tuesday 10 November 2009, Philipp K. Janert wrote:
> > > > Since 4.0 we support string variables. That means wherever a string
> > > > is required in the input command, it is acceptable to provide
> > > > a string constant, a string variable, or a string-valued function.
> >
> > Although there is still a confusion: you write
> > 	"where a string is required"
> >
> > In this example, no string is required.
>
> Of course a string is required.  It is going to become the
> first N characters of a longer string.  How could it be anything
> other than a string itself?

It's not a string as far as the gnuplot session is
concerned. Here:
	plot "file" using 1:2
In this example, "file" is a string within the gnuplot
session. But using is not a string in the same way -
it's a keyword.

Similarly:
	a = 1
Here, a is not a string within the gnuplot session.
It's a variable name.

In the same way, our prefix is not a string within 
the gnuplot session. It's an (unquoted) bareword.
The best way to see this is that there is no need 
to quote it and any quotes are not stripped out.

(This is a totally different question whether any of
this is IMPLEMENTED as a C string. Of course it is.
But that's not what I am talking about.)

>
> > A bareword
> > is required. It is your assumption that it should be
> > a string. (And it is now a discussion item whether
> > it should be a string - in which case the string needs
> > to be handled properly, admittedly.)
> >
> > > > So
> > > >   A = "mydata"
> > > >   stats "file.dat" using 1 variable=A
> > > >
> > > > must expand A to find "mydata", not use it as an unmarked constant.
> > > > All strings should be parsed using the routine try_to_get_string(),
> > > > which handles the three cases.
> > > > Also, normal commands do not use = signs.
> > >
> > > Good point on the string variable issue - I did not
> > > think of that.
> > >
> > > There is a reason for the equality sign, though:
> > > it indicates that the next token is the prefix -
> > > because we have chosen to make the prefix
> > > optional. The equality sign is a way of telling
> > > gnuplot that the next token is a prefix, not the
> > > next keyword.
>
> Not following you here.
> The keyword itself can be optional - you don't have to provide a prefix.

Ha! But :
	stats "file"
and
	stats "file" var
have different behavior!

In the first case, no assignment to variables is made.
Only in the second do we assign to variables. (Without
a prefix.)

So, how do I distinguish without the equality sign (or
another keyword) between:
	stats "file" var noout
and 
	stats "file" var foo

Here, the first is supposed to assign to variables and 
not print to screen (keyword "noout"), whereas the former
assigns to variables with prefix foo, but does not print to 
screen? To distinguish these cases, I need to tell gnuplot
that the foo in the second cases "belongs to" var. The simplest
way I could think of was to use the equality sign.

> But if you include the keyword in your command, then the next token
> must be a string.  Where does the = sign come in?
>
> [maybe "prefix" is a better keyword than "variable"]
>
> > > (I admit that the equality sign is unusual and I did
> > > hesitate a little. But it does provide a simple solution
> > > to this particular problem.)
> > >
> > > I don't want to make the prefix mandatory. For
> > > convenience, it seems that in many cases it won't
> > > be needed.
>
> So let it default to an empty string.

Re: Problem and solution: Patch for new "stats" command

[Ethan M. <merritt@u.washington.edu>]

On Tuesday 10 November 2009, Philipp K. Janert wrote:
> On Tuesday 10 November 2009 07:10:51 pm you wrote:
> > On Tuesday 10 November 2009, Philipp K. Janert wrote:
> > > > > Since 4.0 we support string variables. That means wherever a string
> > > > > is required in the input command, it is acceptable to provide
> > > > > a string constant, a string variable, or a string-valued function.
> > >
> > > Although there is still a confusion: you write
> > > 	"where a string is required"
> > >
> > > In this example, no string is required.
> >
> > Of course a string is required.  It is going to become the
> > first N characters of a longer string.  How could it be anything
> > other than a string itself?


> >
> > > A bareword
> > > is required. It is your assumption that it should be
> > > a string. (And it is now a discussion item whether
> > > it should be a string - in which case the string needs
> > > to be handled properly, admittedly.)
> > >
> > > > > So
> > > > >   A = "mydata"
> > > > >   stats "file.dat" using 1 variable=A
> > > > >
> > > > > must expand A to find "mydata", not use it as an unmarked constant.
> > > > > All strings should be parsed using the routine try_to_get_string(),
> > > > > which handles the three cases.
> > > > > Also, normal commands do not use = signs.
> > > >
> > > > Good point on the string variable issue - I did not
> > > > think of that.
> > > >
> > > > There is a reason for the equality sign, though:
> > > > it indicates that the next token is the prefix -
> > > > because we have chosen to make the prefix
> > > > optional. The equality sign is a way of telling
> > > > gnuplot that the next token is a prefix, not the
> > > > next keyword.
> >
> > Not following you here.
> > The keyword itself can be optional - you don't have to provide a prefix.
> 
> Ha! But :
> 	stats "file"
> and
> 	stats "file" var
> have different behavior!
> 
> In the first case, no assignment to variables is made.
> Only in the second do we assign to variables. (Without
> a prefix.)
> 
> So, how do I distinguish without the equality sign (or
> another keyword) between:
> 	stats "file" var noout
> and 
> 	stats "file" var foo

try_to_get_string() will return NULL if noout is not a currently
defined string variable.  If you define a string variable that is
the same as a keyword then yes, you could create a problem.
If that bothers you, you could explicitly test whether the next
token is "noout".


> Here, the first is supposed to assign to variables and 
> not print to screen (keyword "noout"), whereas the former
> assigns to variables with prefix foo, but does not print to 
> screen? To distinguish these cases, I need to tell gnuplot
> that the foo in the second cases "belongs to" var. The simplest
> way I could think of was to use the equality sign.
> 
> > But if you include the keyword in your command, then the next token
> > must be a string.  Where does the = sign come in?
> >
> > [maybe "prefix" is a better keyword than "variable"]
> >
> > > > (I admit that the equality sign is unusual and I did
> > > > hesitate a little. But it does provide a simple solution
> > > > to this particular problem.)
> > > >
> > > > I don't want to make the prefix mandatory. For
> > > > convenience, it seems that in many cases it won't
> > > > be needed.
> >
> > So let it default to an empty string.

Same answer as before, with an explicit default case:

   char *prefix= NULL;
   if (equals(c_token,"variable")) {
        c_token++;
        prefix = try_to_get_string();
   }
   if (!prefix)
	prefix = gp_strdup("");


...

   free(prefix);

Re: Problem and solution: Patch for new "stats" command

[Tait <gnu...@t4...>]

> > Mean as used here seems to be the arithmetic mean. What about the geometric
> > mean? (Or harmonic mean, or any of the other types of averages?)
>
> Harmonic mean is easily done with the present stats command as
> stats 'foo' u 1:(1.0/$2) noout var
> harmonic = records / sum_y

I was really trying to make two points here. One, that the choice of name
is ambiguous, and perhaps a more specific name would serve better. Second,
the choice of an arithmetic mean (out of all the possible formulas one
could use for expected value) seems arbitrary. I can almost convince myself
that arithmetic mean is possibly the most common, so maybe that justifies
its selection. But arithmetic mean, as you've pointed out with the harmonic
mean, can be calculated separately via sum_y/records. Others like the
commonly-used geometric mean can't be calculated from other values exposed
by stats.

> > I wonder, rather than providing a restricted set of pre-defined functions,
> > is there a way to allow the user to provide a formula or expression that
> > will be applied across multiple rows? Then the user could calculate the
> > mean (whatever that means to their application) or standard deviation or
> > some other arbitrary metric on their own.
>
> stats 'foo' u ($2*$3+cos($4)) should work as it is, if that is what you meant. A
> fairly large set of quantities can be calculated using the variables that are
> produced by stats, if the proper function is applied to the columns beforehand.

That is not at all what I meant. The stats command (and plot, and others) 
provide access to other data values on the same row, via $column-number. 
There is, however, no way to access values on _other_ rows. I can't plot 
the delta between the current $2 and the $2 of the previous row, for 
example. A general way to provide formulas or expressions that operate 
across multiple rows would be more flexible and also make "stats" 
unnecessary.

The tangent about Perl was intended to be in the same vein as Ethan's
comment that gnuplot is not MatLab or Mathematica or MathCad or R, and 
we shouldn't try to be. Maybe stats is trying to make gnuplot do too
much. The danger of using an 80% tool is that it will be abused and
expanded to try and do 100% of jobs, when the user should have 
switched to a more appropriate tool long ago. 

Maybe an alternate (and more useful?) way to provide the stats 
functionality is to add a contrib directory to the gnuplot 
distribution in which are placed stand-alone utilities like stats
that perform useful transformations on and summaries of 
gnuplot-looking data files, using gnuplot-looking syntaxes. 
Someone wanting to know the record count of a data file could 
(in gnuplot) do records=`contrib/countrows -using 3 -every ::2::2`.
This avoids syntactic complexity in gnuplot, is more flexible
while (I think) solving the same problems. It can be easily
expanded to include new and improved functionality without even
needing to recompile gnuplot itself.

Tait

Re: Problem and solution: Patch for new "stats" command

[Ethan M. <merritt@u.washington.edu>]

On Wednesday 11 November 2009, Tait wrote:
>
> > stats 'foo' u ($2*$3+cos($4)) should work as it is, if that is what you meant. A
> > fairly large set of quantities can be calculated using the variables that are
> > produced by stats, if the proper function is applied to the columns beforehand.
> 
> That is not at all what I meant. The stats command (and plot, and others) 
> provide access to other data values on the same row, via $column-number. 
> There is, however, no way to access values on _other_ rows. I can't plot 
> the delta between the current $2 and the $2 of the previous row, for 
> example. A general way to provide formulas or expressions that operate 
> across multiple rows would be more flexible and also make "stats" 
> unnecessary.

You can do this while plotting. See "running_avg.dem".
Whether it's worth making the syntax of the stats command more like the
plot command, that's another question.  Or for that matter, whether
it's better to have a separate stats command or to incorporate it into
the existing plot command.

> The tangent about Perl was intended to be in the same vein as Ethan's
> comment that gnuplot is not MatLab or Mathematica or MathCad or R, and 
> we shouldn't try to be.

The wasn't me. I think it's silly to rule out a useful addition on 
that basis. I think we should seriously consider any addition
that strengthens gnuplot's ability to create plots.  I do want to hear a
strong case, however, that the stats command needs to be an internal part
of gnuplot rather than an external script.  I currently do this sort of
thing in an external perl script (or in R). But using R to calculate the
min/mean/max is ridiculous overkill, and I'd still have to get the
information back into gnuplot in order to generate the plots.
 
> Maybe an alternate (and more useful?) way to provide the stats 
> functionality is to add a contrib directory to the gnuplot 
> distribution in which are placed stand-alone utilities like stats
> that perform useful transformations on and summaries of 
> gnuplot-looking data files, using gnuplot-looking syntaxes. 

Exactly.  I'm not 100% convinced that the initial set of capabilities
in the "stats" command justifies including it in the core code rather
than just running an external script.  But if further integration with
the plotting code gives additional benefits over running an external 
script, so be it.  

Re: Problem and solution: Patch for new "stats" command

[Philipp K. J. <ja...@ie...>]

> > The tangent about Perl was intended to be in the same vein as Ethan's
> > comment that gnuplot is not MatLab or Mathematica or MathCad or R, and
> > we shouldn't try to be.
>
> The wasn't me. I think it's silly to rule out a useful addition on
> that basis. I think we should seriously consider any addition
> that strengthens gnuplot's ability to create plots.  I do want to hear a
> strong case, however, that the stats command needs to be an internal part
> of gnuplot rather than an external script.  I currently do this sort of
> thing in an external perl script (or in R). But using R to calculate the
> min/mean/max is ridiculous overkill, and I'd still have to get the
> information back into gnuplot in order to generate the plots.

Strong case is always a little in the eye of the beholder.
But here are my arguments:

1) Convenience
I do this stuff w/ external scripts, too, and I always find it
annoying that I have to pop up a different window, run my
little script, copy and paste the results back into gnuplot...

2) Multiplatform
Exactly how do you do any of this if you are NOT on Linux?
(I admit this is a pretty weak argument, but it is not entirely
baseless. People are much less likely to have Perl/Python
installed on their Win box, compared to a standalone gnuplot
binary.)

3) Stand-alone Scripting
Running a separate script from gnuplot is a pain. Cutting
and pasting values manually is not an option for scripts.
Having stuff assigned to variables w/in the gnuplot session
is therefore desirable.

4) Input Parsing
Gnuplot's input parsing is very robust and flexible. It can
eat a lot. My problem with external scripts is that it is more
difficult to make them as robust, which means that I don't
have any "pre-canned" - I write them ad-hoc for each file
format that I am dealing with. But that makes it inconvenient
in the long run. On the other hand, I don't want to own and
maintain my own stats script, when I might as well have this
functionality included in gnuplot (where it also benefits 
everybody else).

I think some of the reluctance comes from the fact that the
set of capabilities of the current stats command is fixed. 

There are two answers to this:

- Should we extend the set of values calculated? Personally,
	the values that we have included cover approximately
	92.5% of what I need. I think that's a pretty good ratio!
	I would also argue that it covers those values that are
	most important for PLOTTING. But I am willing to take
	suggestions for additional quantities. (We can also look
	at what other packages like R do in their "summary"
	functions.)

- Should gnuplot (deep breath) develop a "plugin" architecture?
	So that you could run an external script and assign the
	returns to gnuplot vars in a transparent and convenient
	fashion? 

I think the latter idea is worth a thought, but it is clearly a much
bigger project (not the coding, but designing a good user 
interface). But it seems a little like overkill for what we are trying
to accomplish here.

There is no claim that the stats command will save the world.
But I do claim that it provides enough of a convenience (see
above) to have it included. 

Counter-argument: I'd like to hear a strong argument why it
should NOT be included. ;-)

Best,

		Ph.

Re: Problem and solution: Patch for new "stats" command

[Ethan M. <merritt@u.washington.edu>]

On Wednesday 11 November 2009, Philipp K. Janert wrote:
> 
> > > The tangent about Perl was intended to be in the same vein as Ethan's
> > > comment that gnuplot is not MatLab or Mathematica or MathCad or R, and
> > > we shouldn't try to be.
> >
> > That wasn't me. I think it's silly to rule out a useful addition on
> > that basis. I think we should seriously consider any addition
> > that strengthens gnuplot's ability to create plots.  I do want to hear a
> > strong case, however, that the stats command needs to be an internal part
> > of gnuplot rather than an external script.  I currently do this sort of
> > thing in an external perl script (or in R). But using R to calculate the
> > min/mean/max is ridiculous overkill, and I'd still have to get the
> > information back into gnuplot in order to generate the plots.
> 
> Strong case is always a little in the eye of the beholder.
> But here are my arguments:
> 
> 1) Convenience
> I do this stuff w/ external scripts, too, and I always find it
> annoying that I have to pop up a different window, run my
> little script, copy and paste the results back into gnuplot...

But that's what I had to do inside gnuplot itself when I was
making plots on Monday, even _with_ the stats command!  

I had to run the stats command, then copy and paste the min/max
values back from 16 separate invocations back into my plot command.
To make this easier, I ran two copies of gnuplot, using one
to run the stats command and the other to issue plot commands.

If we can make it so that the whole process is scriptable, then
fine.  Otherwise I see zero difference between opening a 
second window to run an external script and opening a second
window to run another copy of gnuplot.

> 2) Multiplatform
> Exactly how do you do any of this if you are NOT on Linux?

I take it you mean "how do you do any of this under Windows?".
Frankly, I don't care.

> 3) Stand-alone Scripting
> Running a separate script from gnuplot is a pain. Cutting
> and pasting values manually is not an option for scripts.
> Having stuff assigned to variables w/in the gnuplot session
> is therefore desirable.

Yes, so let's get that working. Right now it isn't.

> 4) Input Parsing
> 
> - Should we extend the set of values calculated? Personally,
> 	the values that we have included cover approximately
> 	92.5% of what I need. I think that's a pretty good ratio!

I did more plotting yesterday, of the same sort as on Monday.
Again I was creating plots for publication based on multi-column
data in a csv file.  It's still cumbersome.  Not because the
wrong values are calculated, but because they are not easily
tracked and made available in a subsequent plot command.
Given the tools I have on hand, it's actually easier to use an
external script because I can say:
    min_Run3 = system(" some command ")
whereas if I say instead
    stats "datafile" using 3
then I have to manually cut-and-paste to do the assignment.
Again, let's get the variable assigment working properly
so that this annoyance goes away.

> - Should gnuplot (deep breath) develop a "plugin" architecture?
> 	So that you could run an external script and assign the
> 	returns to gnuplot vars in a transparent and convenient
> 	fashion? 

You can already do that for external scripts.
A true plugin architecture using compiled functions would be possible.
In fact there is a patchset on SourceForge that implements it.
But how could we support it cross-platform? 

> Counter-argument: I'd like to hear a strong argument why it
> should NOT be included. ;-)

I think it is a potentially valuable addition.

But until it works in a scripted enviroment, it's strictly worse than
using the system() command to run external tools, or for that 
matter having the script run an external tool first and pass the
values to gnuplot.  

Re: Problem and solution: Patch for new "stats" command

[Hans-Bernhard B. <HBB...@t-...>]

Philipp K. Janert wrote:
> 1) Convenience
> I do this stuff w/ external scripts, too, and I always find it
> annoying that I have to pop up a different window, run my
> little script, copy and paste the results back into gnuplot...

Then don't.  At least the wgnuplot_pipes binary can do `backtics` 
command substitution and other piping tricks just fine.  For fully 
command-line style usage, there's always the "console gnuplot for 
windows" build option.

> 2) Multiplatform
> Exactly how do you do any of this if you are NOT on Linux?

The same way you do it on Linux, for just about every gnuplot except the 
traditional, non-pipe wgnuplot.exe on MS Windows.

> (I admit this is a pretty weak argument, but it is not entirely
> baseless. People are much less likely to have Perl/Python
> installed on their Win box, compared to a standalone gnuplot
> binary.)

People who are likely to even arrive at the idea of trying to use them, 
will.

> - Should gnuplot (deep breath) develop a "plugin" architecture?

If anybody can figure out one that is anywhere near as portable as 
gnuplot itself, sure.  And let's not forget that the internals of the 
program would need a _major_ overhaul before they could safely be 
exported to a (possibly malicious) plugin...

Re: Problem and solution: Patch for new "stats" command

[Hans-Bernhard B. <HBB...@t-...>]

Ethan Merritt wrote:

> Of course a string is required.  It is going to become the
> first N characters of a longer string.  

Erm, no.  It's going to become the first N characters of a variable name.

> How could it be anything other than a string itself?

By being a (partial) variable name.  And I don't we ever quote a 
variable name in gnuplot.

>>> There is a reason for the equality sign, though:
>>> it indicates that the next token is the prefix -
>>> because we have chosen to make the prefix
>>> optional. 

So find a different way of signalling that.  It could be as simple as 
changing the default from "no variables created" to "GPSTAT_* variables 
created, so that part of the syntax would become
	
	{ | novariables | prefix <name> }

(wherein leaving out the argument would turn on variables, and if you 
don't want them, you have to turn them off explicitly}

Or leave the default at "novar", and make the syntax

	{ | variables | prefix <name> }

Or have no default at all, and make it

	{ {no}variables | prefix <name> }

Re: Problem and solution: Patch for new "stats" command

[Ethan M. <merritt@u.washington.edu>]

On Wednesday 11 November 2009, Hans-Bernhard Bröker wrote:
> Ethan Merritt wrote:
> 
> > Of course a string is required.  It is going to become the
> > first N characters of a longer string.  
> 
> Erm, no.  It's going to become the first N characters of a variable name.

You are missing the point that in order for it to be useful in a script,
I have to be able to construct the prefix dynamically, which means 
I need to handle the prefix itself as a variable.  Hence: a string variable.

> > How could it be anything other than a string itself?
> 
> By being a (partial) variable name.  And I don't [think] we ever quote a 
> variable name in gnuplot.

Yep. It's a partial name of something.  In other words, it is a substring
of the full name.  To specify or manipulate this substring, I want to put
it in a string variable.

If these names can not be manipulated as variables, and it does seem that
nasty issues are poking their heads out of holes in the woodwork,  then
I don't see any point to having them.  Just pick a fixed set of names and
be done with it.  But then we'll have to come up with some other mechanism
for dealing with multi-column data files, and so far I haven't heard any
other suggestions.   

I am talking about real-world cases here.  To plot multiple curves from a
data file I need to be able to iterate over the columns in the data file.  
I can provide sample data and the desired output if you like.  

If the stats functionality were embedded in the 'plot' command itself,
then I could intersperse reloading of the stats output values with the
individual clauses of the plot command:
   plot \
	'datafile' stats ($5), '' using ($5/stats_max), \
      'datafile' stats ($6), '' using ($6/stats_max), \
	...

Note that each time "stats_max" is encountered during evaluation of the 
plot command, it has been reloaded with a new value by the preceding
stats clause. That works fine for iteration:

   plot for [i=5:20] 'datafile' stats (column(i)), '' using (column(i)/stats_max)

But if the stats are collected via a separate command then, as I
discovered the hard way, it is very cumbersome to collect them into a single
multi-column plot command.  This is true even aside from the question of
using iteration, which anyhow isn't possible with the prototype stats
implementation.

Re: Problem and solution: Patch for new "stats" command

[Petr M. <mi...@ph...>]

> If the stats functionality were embedded in the 'plot' command itself,
> then I could intersperse reloading of the stats output values with the
> individual clauses of the plot command:
>    plot \
> 	'datafile' stats ($5), '' using ($5/stats_max), \
>       'datafile' stats ($6), '' using ($6/stats_max), \
> 	...
> 
>    plot for [i=5:20] 'datafile' stats (column(i)), '' using (column(i)/stats_max)

stat 'datafile' using columns {2 | 2:6 | 2:2:6 | all}

(It can provide GPVAL_DATAFILE_COLUMNS=8.)

BTW, the "stats" command could provide "number of points", which can be 
useful.

---
PM