Re: Problem and solution: Patch for new "stats" command

[Tait <gnu...@t4...>]

> > Mean as used here seems to be the arithmetic mean. What about the geometric
> > mean? (Or harmonic mean, or any of the other types of averages?)
>
> Harmonic mean is easily done with the present stats command as
> stats 'foo' u 1:(1.0/$2) noout var
> harmonic = records / sum_y

I was really trying to make two points here. One, that the choice of name
is ambiguous, and perhaps a more specific name would serve better. Second,
the choice of an arithmetic mean (out of all the possible formulas one
could use for expected value) seems arbitrary. I can almost convince myself
that arithmetic mean is possibly the most common, so maybe that justifies
its selection. But arithmetic mean, as you've pointed out with the harmonic
mean, can be calculated separately via sum_y/records. Others like the
commonly-used geometric mean can't be calculated from other values exposed
by stats.

> > I wonder, rather than providing a restricted set of pre-defined functions,
> > is there a way to allow the user to provide a formula or expression that
> > will be applied across multiple rows? Then the user could calculate the
> > mean (whatever that means to their application) or standard deviation or
> > some other arbitrary metric on their own.
>
> stats 'foo' u ($2*$3+cos($4)) should work as it is, if that is what you meant. A
> fairly large set of quantities can be calculated using the variables that are
> produced by stats, if the proper function is applied to the columns beforehand.

That is not at all what I meant. The stats command (and plot, and others) 
provide access to other data values on the same row, via $column-number. 
There is, however, no way to access values on _other_ rows. I can't plot 
the delta between the current $2 and the $2 of the previous row, for 
example. A general way to provide formulas or expressions that operate 
across multiple rows would be more flexible and also make "stats" 
unnecessary.

The tangent about Perl was intended to be in the same vein as Ethan's
comment that gnuplot is not MatLab or Mathematica or MathCad or R, and 
we shouldn't try to be. Maybe stats is trying to make gnuplot do too
much. The danger of using an 80% tool is that it will be abused and
expanded to try and do 100% of jobs, when the user should have 
switched to a more appropriate tool long ago. 

Maybe an alternate (and more useful?) way to provide the stats 
functionality is to add a contrib directory to the gnuplot 
distribution in which are placed stand-alone utilities like stats
that perform useful transformations on and summaries of 
gnuplot-looking data files, using gnuplot-looking syntaxes. 
Someone wanting to know the record count of a data file could 
(in gnuplot) do records=`contrib/countrows -using 3 -every ::2::2`.
This avoids syntactic complexity in gnuplot, is more flexible
while (I think) solving the same problems. It can be easily
expanded to include new and improved functionality without even
needing to recompile gnuplot itself.

Tait