Menu
▾
▴
gnuplot-beta
|
From: Petr M. <mi...@ph...> - 2014-08-29 23:45:24
|
In gnuplot, the "stat" command calculates "Std Dev:" and "STATS_stddev_x" as
sqrt( SumOfSquares / n )
In GNU R, OpenOffice et al, the function stddev() calculates
sqrt( SumOfSquares / (n-1) )
while function stddevp() returns
sqrt( SumOfSquares / n )
That's confusing. Why has gnuplot chosen the other possibility (this smaller
stddev estimate means that the average value of the statistical ensemble
is known).
What to do with that?
- It would be nice if gnuplot gets compatible and has both
"Std Dev:" and "STATS_stddev_x" and "Std Dev P:" and "STATS_stddevp_x".
But then scripts running on different gnuplot versions would give
different results :-(
- Let gnuplot write both variants of stddev so that the user sees which
values is which type of stddev (one is smaller and the other is larger). The
other could be called "Std Dev (n-1):" and "STATS_stddev1_x".
- Document stddev used in gnuplot by writing its formula in the help.
- I would also like if gnuplot writes the deviation of the mean
as used in (mean +- err) where err=sqrt(SumOfSquares/(n*(n-1))).
How to call this value?
Well, I figured this out when preparing some hints for students...
---
Petr Mikulik
|
|
From: Allin C. <cot...@wf...> - 2014-08-30 00:39:47
|
On Sat, 30 Aug 2014, Petr Mikulik wrote: > In gnuplot, the "stat" command calculates "Std Dev:" and "STATS_stddev_x" as > sqrt( SumOfSquares / n ) That's the maximum likelihood estimator (MLE) of the population standard deviation. It happens to be biased, although it's consistent (converges in probability on the population standard deviation in large samples). > In GNU R, OpenOffice et al, the function stddev() calculates > sqrt( SumOfSquares / (n-1) ) That's the "sample standard deviation", an unbiased estimator of the population standard deviation if the relevant degrees of freedom are indeed n - 1. > while function stddevp() returns > sqrt( SumOfSquares / n ) > > That's confusing. Why has gnuplot chosen the other possibility (this smaller > stddev estimate means that the average value of the statistical ensemble > is known). > > What to do with that? If what you're saying is correct as a description of what gnuplot does then it's arguably idiosyncratic of gnuplot to give the MLE when most statistical software gives the sample standard deviation. It woud be sufficient either to state in the documentation that gnuplot gives the MLE or to switch to the sample standard deviation. I don't see much to be said for more complicated options. Allin Cottrell |
|
From: <pl...@pi...> - 2014-08-30 12:12:45
|
On 08/30/14 02:10, Allin Cottrell wrote: > It woud be sufficient either to state in the documentation that > gnuplot gives the MLE or to switch to the sample standard deviation. > I don't see much to be said for more complicated options. > > Allin Cottrell This should be clearly stated up front without the need for digging the doc. If gnuplot is calculating the "sample standard deviation" ( which would seem to be the most appropriate when given a sample of data ) then that is what it should label it as when outputting the result, rather than just 'std dev'. The note about degrees of freedom assumption could go in the doc. Uneducated used of these kind of statistics is pretty common, perhaps more so in population that are going to depend on a plotting program to provide them, rather than having existing knowledge and software to provide the stats. It would be good to be as clear as possible about what is being output with an assumption that user probably has little knowledge of the subject. A link to some starting point to further understanding as a note in help may be valuable. Peter. |
|
From: sfeam <sf...@us...> - 2014-08-31 00:24:10
|
On Saturday, 30 August 2014 07:39:57 AM pl...@pi... wrote: > On 08/30/14 02:10, Allin Cottrell wrote: > > It woud be sufficient either to state in the documentation that > > gnuplot gives the MLE or to switch to the sample standard deviation. > > I don't see much to be said for more complicated options. > > > > Allin Cottrell > > > This should be clearly stated up front without the need for digging the doc. > > If gnuplot is calculating the "sample standard deviation" ( which would > seem to be the most appropriate when given a sample of data ) then that > is what it should label it as when outputting the result, rather than > just 'std dev'. The note about degrees of freedom assumption could go > in the doc. Gnuplot "stats" is not calculating the sample standard deviation. That is exactly the point being raised. The standard deviation, skew and kurtosis values all use a full-population assumption rather than being corrected for finite sampling. I definitely agree this should be clearly stated in the documentation, and I'll work on editing gnuplot.doc to include the relevant equation for each variable STATS_*. However that does not resolve the question of whether gnuplot should be reporting the finite-sampling assumption values instead of, or in addition to, what it reports now. By the way, this same point came up a while back with regard to the calculation of the bandwidth in "smooth kdensity". The code currently uses the population standard deviation, but arguably should use the sample-based standard deviation instead. The question did not generate any discussion at that time. Ethan > Uneducated used of these kind of statistics is pretty common, perhaps > more so in population that are going to depend on a plotting program to > provide them, rather than having existing knowledge and software to > provide the stats. It would be good to be as clear as possible about > what is being output with an assumption that user probably has little > knowledge of the subject. > > A link to some starting point to further understanding as a note in help > may be valuable. > > Peter. |
|
From: <pl...@pi...> - 2014-08-31 06:28:00
|
On 08/31/14 02:20, sfeam wrote: > Gnuplot "stats" is not calculating the sample standard deviation. > > That is exactly the point being raised. > > The standard deviation, skew and kurtosis values all use a full-population > > assumption rather than being corrected for finite sampling. > > I definitely agree this should be clearly stated in the documentation, > > and I'll work on editing gnuplot.doc to include the relevant equation > > for each variable STATS_*. > > However that does not resolve the question of whether gnuplot should > > be reporting the finite-sampling assumption values instead of, or in > > addition to, what it reports now. > > By the way, this same point came up a while back with regard to the > > calculation of the bandwidth in "smooth kdensity". > > The code currently uses the population standard deviation, but arguably > > should use the sample-based standard deviation instead. > > The question did not generate any discussion at that time. > > Ethan > Is there any difficulty in simply providing both, clearly labelled sample SD and population SD. This forces anyone who does not know what that means to find out ( or if the difference is small carry on regardless ). Relevant assumptions made about the population, degrees of freedom etc explained in help. I don't recall the kdensity discussion. I usually use a suitable low-pass filter if I want "smooth" data, though I do on occasion use the bezier spline. Peter. |
|
From: Jonathan T. <jt...@as...> - 2014-08-31 15:26:53
|
On Sun, Aug 31, 2014 at 08:21:21AM +0200, pl...@pi... wrote:
> Is there any difficulty in simply providing both, clearly labelled
> sample SD and population SD. This forces anyone who does not know what
> that means to find out ( or if the difference is small carry on
> regardless ). Relevant assumptions made about the population, degrees
> of freedom etc explained in help.
I strongly agree -- this option provides the most useful information
for users who *do* know what's going on, and avoids misleading the unwary.
--
-- "Jonathan Thornburg [remove -animal to reply]" <jt...@as...>
Dept of Astronomy & IUCSS, Indiana University, Bloomington, Indiana, USA
currently on the west coast of Canada
"There was of course no way of knowing whether you were being watched
at any given moment. How often, or on what system, the Thought Police
plugged in on any individual wire was guesswork. It was even conceivable
that they watched everybody all the time." -- George Orwell, "1984"
|
|
From: sfeam <sf...@us...> - 2014-08-31 21:36:08
|
On Sunday, 31 August 2014 11:12:26 AM Jonathan Thornburg wrote: > On Sun, Aug 31, 2014 at 08:21:21AM +0200, pl...@pi... wrote: > > Is there any difficulty in simply providing both, clearly labelled > > sample SD and population SD. This forces anyone who does not know what > > that means to find out ( or if the difference is small carry on > > regardless ). Relevant assumptions made about the population, degrees > > of freedom etc explained in help. > > I strongly agree -- this option provides the most useful information > for users who *do* know what's going on, and avoids misleading the unwary. Stats now calculates and returns the sample standard deviation STATS_ssd (in cvs for 5.0 and 5.1). The formulae used to calculate all single-column stats values are now given in the TeX-derived documentation. The difference between _stddev and _ssd in particular is noted in all documentation formats. Ethan |
