gnuplot-beta

interpretation of fit errors

[Bastian M. <bma...@we...>]

Does somebody have a better name for the proposed setting on how to 
interpret data errors? The currently proposed

  set fit relativeerrors|absoluteerrors

sounds a bit odd to me. See bug #2956524

    Bastian

INSTALL and CVS

[<pl...@pi...>]

Hi,

I just got CVS instructions from sourceforge and grabbed the source 
code. I referred to INSTALL for instructions and it tells me to use a 
non existent configure.

Installation from sources
=========================

For the impatient
-----------------

    Configuration options are in the Makefile and in src/term.h, which
selects the set of terminal drivers to be compiled in.

    The recommended way to configure both of these is the GNU-style
"./configure" script described below, and also in INSTALL.gnu.




Some checking , re-reading and head scratching later I spot a file 
called prepare. That does the trick.

I seem to recall hitting this last time I used CVS and was told CVS was 
a special case or some such. Well if it is , why is it not documented?

For the cost of adding one line to the install instructions it seems odd 
that I have to waste time working out why the install instructions don't 
work and find out what I need to do myself.

Surely a sentence like "if you downloaded from CVS repository you will 
need the additional step of running ./prepare before configure/make" 
would mean it was properly documented.

Any reason that is not in the install instructions in INSTALL?

regards. Peter.

Re: INSTALL and CVS

[Allin C. <cot...@wf...>]

On Thu, 5 May 2011 pl...@pi... wrote:

> I just got CVS instructions from sourceforge and grabbed the source
> code. I referred to INSTALL for instructions and it tells me to use a
> non existent configure.

You need to read README.1ST ;-)

Allin Cottrell

Re: INSTALL and CVS

[sfeam (E. Merritt) <eam...@gm...>]

On Thursday, 05 May 2011, pl...@pi... wrote:
> Any reason that is not in the install instructions in INSTALL?

The CVS source tree contains a file INSTALL for use with the
eventual distributed release package.   The instructions in
INSTALL are correct for the release package.   But if you are 
building directly from the CVS source tree you need to first run
the files through it through autoconf.
There is a script  "./prepare" that does this for you.

	Ethan

Re: INSTALL and CVS

[<pl...@pi...>]

On 05/05/11 17:40, sfeam (Ethan Merritt) wrote:
> On Thursday, 05 May 2011, pl...@pi... wrote:
>> Any reason that is not in the install instructions in INSTALL?
>
> The CVS source tree contains a file INSTALL for use with the
> eventual distributed release package.   The instructions in
> INSTALL are correct for the release package.   But if you are
> building directly from the CVS source tree you need to first run
> the files through it through autoconf.
> There is a script  "./prepare" that does this for you.
>
> 	Ethan
>

Allin Cottrell wrote:

You need to read README.1ST ;-)


Hi,

Allin is of course correct. Thanks for pointing to that file:

    If your source is from the CVS repository rather from a release package,
    the order of commands is  ./prepare; ./configure; make



So the point is that INSTALL, despite it's length, does not tell you how 
to install in all cases. There are two files giving install notes with a 
paragraph headed "Installation from sources
", the two with differing information.

Is there any pressing reason for this simple phrase about how to install 
not being in the file called INSTALL?

Why is there information in README.1ST: 'Installation from sources' that 
is not in INSTALL: 'Installation from sources' ?

Surely the two should be syncronised or the para from README.1ST should 
be merged into INSTALL.

thanks for the replies.


regards, Peter.

Re: interpretation of fit errors

[Daniel J S. <dan...@ie...>]

On 05/05/2011 02:01 AM, Bastian Märkisch wrote:

> Does somebody have a better name for the proposed setting on how to
> interpret data errors? The currently proposed
>
>    set fit relativeerrors|absoluteerrors
>
> sounds a bit odd to me. See bug #2956524
>
>      Bastian

Bastian,

Please fill us in about what these are supposed to mean.  Maybe that 
will lead to better syntax.  Browsing the documentation for "fit" and 
reading the bug report is a bit to digest.  Here is the comment in the 
bug report:

**
Dear gnuplotters,

I may be missing something, but I understood that with least squares 
with given standard deviations, the error in the fit parameters is 
simply the square root of the according covariance matrix diagonal 
element, opposed to the case of unknown standard deviations, where the 
standard deviations are assumed to be equal and are estimated a 
posteriori by chi^2 / ndf. If this is the case, I think lines 766--768 
in fit.c of gnuplot version 4.4 rc1 are a bug, because in the case of 
known data errors, this code should not be exercised.

Thanks for your comments,

Paul
**

The case of "known data errors", what is that?  Does Paul mean known 
statistics for the data errors (i.e., the case where an extra column is 
supplied to the input)?

I see in the documentation something about "set fit errorvariables" and 
variables created with "_err" tagged onto the string.  Is this what 
"relativeerrors" and "absoluteerrors" refers to?  Is it something having 
to do with the way in which "_err" error variables are derived?  If so, 
what then if the user wants both for comparison purposes?  Maybe it 
should become two sets of error variables "_errabs" and "_errrel".  Am I 
on the right track?

As for the original bug report, unless this is something obvious, 
perhaps there is a way to illustrate the error with a test case, to 
ensure the fit is solved correctly.  In fact, a tutorial demo or two 
illustrating settings would be nice.

Dan



>
> ------------------------------------------------------------------------------
> WhatsUp Gold - Download Free Network Management Software
> The most intuitive, comprehensive, and cost-effective network
> management toolset available today.  Delivers lowest initial
> acquisition cost and overall TCO of any competing solution.
> http://p.sf.net/sfu/whatsupgold-sd
> _______________________________________________
> gnuplot-beta mailing list
> gnu...@li...
> https://lists.sourceforge.net/lists/listinfo/gnuplot-beta
>


-- 

Dan Sebald
email: daniel(DOT)sebald(AT)ieee(DOT)org
URL: http://www(DOT)dansebald(DOT)com

Re: interpretation of fit errors

[Bastian M. <bma...@we...>]

 > Please fill us in about what these are supposed to mean.  Maybe that
 > will lead to better syntax.  Browsing the documentation for "fit" and
 > reading the bug report is a bit to digest.
 > (snip)

The point of the original report is the following: After the actual fit 
the calculated errors of free variables are currently scaled by 
FIT_STDFIT. This is correct if there was no error column given for the 
dependent variable, or the errors are in fact relative weights, ie. they 
only give the relative "credibility" of the data points. This would be 
the behaviour of "set fit relativeerrors".

If the error column actually contains (absolute) data errors, e.g. 
statistical errors, this scaling is undesirable. "set fit 
absoluteerrors" would allow the user to switch it off and therefore 
obtain the same errors as e.g. CERN Minuit does.

This issue has been discussed at lengths on this mailing list (and 
elsewhere) several times already, see e.g.:

http://thread.gmane.org/gmane.comp.graphics.gnuplot.devel/3737/focus=3740
and
http://thread.gmane.org/gmane.comp.graphics.gnuplot.devel/6205

The proposed solution leaves the decision on how to interpret data 
errors to the user. (My gnuplot.ini will certainly have "set fit abs" in 
it.) For the sake of compatibility and since "weigths" are standard in 
some fields, the default stays like it is.

> **
> Dear gnuplotters,
>
> I may be missing something, but I understood that with least squares
> with given standard deviations, the error in the fit parameters is
> simply the square root of the according covariance matrix diagonal
> element, opposed to the case of unknown standard deviations, where the
> standard deviations are assumed to be equal and are estimated a
> posteriori by chi^2 / ndf. If this is the case, I think lines 766--768
> in fit.c of gnuplot version 4.4 rc1 are a bug, because in the case of
> known data errors, this code should not be exercised.
>
> Thanks for your comments,
>
> Paul
> **
>
> The case of "known data errors", what is that?  Does Paul mean known
> statistics for the data errors (i.e., the case where an extra column is
> supplied to the input)?

Yes. Supplied errors, but with an "absolute" meaning.

> I see in the documentation something about "set fit errorvariables" and
> variables created with "_err" tagged onto the string.  Is this what
> "relativeerrors" and "absoluteerrors" refers to?  Is it something having
> to do with the way in which "_err" error variables are derived?  If so,
> what then if the user wants both for comparison purposes?  Maybe it
> should become two sets of error variables "_errabs" and "_errrel".  Am I
> on the right track?

Right track, see above. I don't think that having two variables is a 
good solution, though: the data errors (input) are either relative 
weights or absolute errors. If the user want's both, he could still 
multiply/divide by FIT_STDFIT when needed.

   Bastian

> As for the original bug report, unless this is something obvious,
> perhaps there is a way to illustrate the error with a test case, to
> ensure the fit is solved correctly.  In fact, a tutorial demo or two
> illustrating settings would be nice.
>
> Dan

Re: interpretation of fit errors

[Daniel J S. <dan...@ie...>]

On 05/05/2011 05:33 AM, Bastian Märkisch wrote:
>
>  > Please fill us in about what these are supposed to mean. Maybe that
>  > will lead to better syntax. Browsing the documentation for "fit" and
>  > reading the bug report is a bit to digest.
>  > (snip)
>
> The point of the original report is the following: After the actual fit
> the calculated errors of free variables are currently scaled by
> FIT_STDFIT. This is correct if there was no error column given for the
> dependent variable, or the errors are in fact relative weights, ie. they
> only give the relative "credibility" of the data points. This would be
> the behaviour of "set fit relativeerrors".

When you say "or" here, are you giving alternate explanation for what it 
means when there is no error column given?  Or do you mean an alternate 
case?  I'm somewhat perplexed by the term "relative".  Scaling by a 
standard deviation to me seems like a normalization process.  Is this a 
well-known technique in the fitting field?  If so, maybe a name related 
to that would help the user understand.

Credibility of the data points is more an interpretation of the 
application.  That's a measurement error sort of thing, isn't it?  (As 
opposed to actual randomness in the quantity itself.)


> If the error column actually contains (absolute) data errors, e.g.
> statistical errors, this scaling is undesirable. "set fit
> absoluteerrors" would allow the user to switch it off and therefore
> obtain the same errors as e.g. CERN Minuit does.
>
> This issue has been discussed at lengths on this mailing list (and
> elsewhere) several times already, see e.g.:
>
> http://thread.gmane.org/gmane.comp.graphics.gnuplot.devel/3737/focus=3740

I read a bit, but didn't get much smarter.  I didn't spend too much time 
on it though.


> and
> http://thread.gmane.org/gmane.comp.graphics.gnuplot.devel/6205
>
> The proposed solution leaves the decision on how to interpret data
> errors to the user. (My gnuplot.ini will certainly have "set fit abs" in
> it.) For the sake of compatibility and since "weigths" are standard in
> some fields, the default stays like it is.

Is this change significant?  Is this an iterative process where the 
resulting errors are fed back into the computations to improve the fit? 
  If so, then I'd say backward compatibility is important.

But if it is some nuanced detailed that initially could be seen as a 
mistake in coding, then I'd say backward compatibility isn't so much an 
issue.

My fear with this is that a user could run the fit, get the results and 
significantly misinterpret what they mean by assuming errors were 
expressed as scaled or unscaled.  That's especially troublesome if the 
standard deviation FIT_STDFIT is near 1.0 because its effect might not 
be so apparent to the user.  You are saying "(My gnuplot.ini will 
certainly have "set fit abs" in it.)", so you think of the errors in one 
way.  Others may think another way, apparently.  So, if using the 
expression "errors" in fitting is in any way ambiguous, it might be best 
to always refer to "absolute errors" or "relative (normalized?) errors". 
  I.e., gnuplot shouldn't input or report something as just "errors".

That probably didn't help any; I'm just trying to brainstorm how to 
clear this up.

Dan

Re: interpretation of fit errors

[Bastian M. <bma...@we...>]

> But if it is some nuanced detailed that initially could be seen as a
> mistake in coding, then I'd say backward compatibility isn't so much an
> issue.

I am pretty sure that this was a deliberate choice. The reasoning being 
that as long as the fit is good, FIT_STDFIT is somehow close to 1. So it 
wouldn't hurt too much if "real" errors were given. See
http://article.gmane.org/gmane.comp.graphics.gnuplot.devel/3737

> My fear with this is that a user could run the fit, get the results and
> significantly misinterpret what they mean by assuming errors were
> expressed as scaled or unscaled.

That is already the case. Most physicists I know incorrectly assume that 
gnuplot reports "unscaled" errors. Why not give them means to get what 
they expect?

   Bastian

Re: interpretation of fit errors

[Daniel J S. <dan...@ie...>]

On 05/05/2011 04:10 PM, Bastian Märkisch wrote:
>> But if it is some nuanced detailed that initially could be seen as a
>> mistake in coding, then I'd say backward compatibility isn't so much an
>> issue.
>
> I am pretty sure that this was a deliberate choice. The reasoning being
> that as long as the fit is good, FIT_STDFIT is somehow close to 1. So it
> wouldn't hurt too much if "real" errors were given. See
> http://article.gmane.org/gmane.comp.graphics.gnuplot.devel/3737

I don't know about the argument that once the fit isn't so good it 
doesn't matter whether the errors are normalized or not.

One thing I notice is that Hans-Bernhard uses the term "residuals". 
Now, "residuals" is a fairly common definition in fitting.  Maybe it 
would have been better in the first place to use "_res" extensions to 
variable names for the unscaled errors and "_err" for the scaled errors 
(or maybe "_ferr" for fitting error with the inherent meaning that 
fitting errors are always scaled).

The argument is made in the post that one is derived from the other with 
a simple scaling; a scaling which is made available to the user so 
redundant information is given.  True, but I don't know if minimal 
representation is that important.  If we're talking minimal basis or 
some linear algebra concept, sure.  But my main point in all this is to 
avoid ambiguity.  If "error" has some ambiguity in the field, whereas 
"residual" is much less ambiguous, then go with the latter.


>> My fear with this is that a user could run the fit, get the results and
>> significantly misinterpret what they mean by assuming errors were
>> expressed as scaled or unscaled.
>
> That is already the case. Most physicists I know incorrectly assume that
> gnuplot reports "unscaled" errors.

That's not good.


> Why not give them means to get what
> they expect?

Yes, naturally.  But the most coherent way to do that is the question, 
right?  We want gnuplot to be easy to use, not arcane.

Dan

Re: interpretation of fit errors

[Hans-Bernhard B. <HBB...@t-...>]

On 06.05.2011 05:08, Daniel J Sebald wrote:
> On 05/05/2011 04:10 PM, Bastian Märkisch wrote:

> One thing I notice is that Hans-Bernhard uses the term "residuals".

Yes, but only for what's always called that: the difference between 
fitted model and data.

> Now, "residuals" is a fairly common definition in fitting.  Maybe it
> would have been better in the first place to use "_res" extensions to
> variable names for the unscaled errors and "_err" for the scaled errors

I rather much doubt that.  The term residual is not applicable to 
parameter errors in any meaningful way.

> The argument is made in the post that one is derived from the other with
> a simple scaling;

That argument would be wrong.  Parameter errors scale in parallel with 
the data errors (or weights, if you prefer), not with the residiuals.

> avoid ambiguity.  If "error" has some ambiguity in the field, whereas
> "residual" is much less ambiguous, then go with the latter.

"Residual" is unambiguous primarily because it is use for exactly one 
purpose.  Calling something else by the same name would only break that 
unambiguity.  That wouldn't be particularly helpful.

Re: interpretation of fit errors

[Hans-Bernhard B. <HBB...@t-...>]

On 05.05.2011 10:52, Daniel J Sebald wrote:

> The case of "known data errors", what is that?  Does Paul mean known
> statistics for the data errors (i.e., the case where an extra column is
> supplied to the input)?

Not quite.  The question is not whether data errors are present.  To 
some extent they always are --- if none were supplied explicitly, 'fit' 
defaults to a constant 1.0 for all of them.

The question is what those values _mean_, and what to do if the 
assumption about their meaning breaks down.

The meaning alluded to by people complaing about gnuplot's behaviour in 
this regard is that the input errors are actual, precise standard 
deviations of (presumably Gaussian distributed) input variables.

Now if this meaning were strictly true, and the fit generally valid, the 
chisq should end up being about equal to the number of degrees of 
freedom.  I.e. STDFIT should end up so close to 1.0 the the division 
performed by gnuplot would not make any notable difference.

The problems start when this plan fails, i.e. you're facing a fit that 
yielded a STDFIT far away from 1.0.  In effect this means that either 
the input errors were wrong, or the model function doesn't actually 
describe the given data at all.  For lack of omniscience, gnuplot has no 
choice but to assume the former, i.e. it decides that those data errors 
are not as reliable as they're made out to be.

Let's say you end up with a STDFIT of about 10.  That means the actual 
deviations between the fitted model and the data are on average 10 times 
as big as the data errors said they should be.  That fit has, in other 
words, missed its goal by a factor of 10 --- you've not even come close 
to threading that function through those error bars.

So what gnuplot does to resolve this conflict is to re-scale the input 
errors by the same factor of 10 they're apparently wrong by.  This 
factor ends up as a factor of 10 increase of the fitted parameters' errors.

In the end effect this means gnuplot treats the data errors as 
_weights_, not as strictly reliable errors.

gnuplot has been working like that since effectively forever.

> As for the original bug report, unless this is something obvious,
> perhaps there is a way to illustrate the error with a test case, to
> ensure the fit is solved correctly.

The demo is dead simple.  Pick any fit from the demos or wherever, and 
repeat it with the data errors multiplied by a fixed factor, i.e. replace

	fit f(x) 'foo.dat' u 1:2:3 via ...

by

	fit f(x) 'foo.dat' u 1:2:($3*20) via ...

'fit' will report the same data errors, both in the printed output and 
in the saved *_err variables.  Only the chisq and STDFIT will have 
shrinked by a factor of 20.

People thinking I made a bad decision here say that the errors on the 
parameters should become 20 times as large in the second case.

Re: interpretation of fit errors

[Bastian M. <bma...@we...>]

For what it's worth, here's the result of a little survey on data 
analysis packages:

* Minuit does not scale
* Origin offers both options, the default being version dependent
* SAS and Mathematica default to scaling of errors

The default choice of how to treat errors is somewhat arbitrary and 
depends on the problem at hand - I am not arguing about that. For a 
given problem only one choice is correct, though.

The default for gnuplot has been chosen long ago. All I am proposing is 
to make this default user changeable. Actual name suggestions for that 
setting are more than welcome!

IMHO there's no point in printing both numbers and even creating another 
error variable. As pointed out earlier, the other value can still be 
easily obtained by multiplying/dividing by the variable FIT_STDFIT.

Finally, here's an example of why _not_ to scale fit errors in 
combination with "real" data errors is important:

Consider NDF = 10. According to the ChiSq distribution, ChiSq = 6.74 
corresponds to P = 0.75, with STDFIT = 0.82, whereas and ChiSq = 12.55 
corresponds to P = 0.25, with STDFIT = 1.12. It is equally probable to 
obtain either ChiSq value, but scaling would tells us errors should 
differ by 30%!

In physics a probability P of the fit below 0.05 or above 0.95 is 
typically considered as an indication of "something being wrong", ie. 
there's some problem with data, errors or model. But within that range 
the fit is accepted. For NDF=10, Bernhard's example of STDFIT=10 would 
correspond to P < 10^-200.

   Bastian

Re: interpretation of fit errors

[Daniel J S. <dan...@ie...>]

On 05/05/2011 06:30 PM, Hans-Bernhard Bröker wrote:
> On 05.05.2011 10:52, Daniel J Sebald wrote:
[snip]
>> As for the original bug report, unless this is something obvious,
>> perhaps there is a way to illustrate the error with a test case, to
>> ensure the fit is solved correctly.
>
> The demo is dead simple.  Pick any fit from the demos or wherever, and
> repeat it with the data errors multiplied by a fixed factor, i.e. replace
>
> 	fit f(x) 'foo.dat' u 1:2:3 via ...
>
> by
>
> 	fit f(x) 'foo.dat' u 1:2:($3*20) via ...
>
> 'fit' will report the same data errors, both in the printed output and
> in the saved *_err variables.  Only the chisq and STDFIT will have
> shrinked by a factor of 20.
>
> People thinking I made a bad decision here say that the errors on the
> parameters should become 20 times as large in the second case.

Well, this is certainly a valid approach.  Often some statistical or 
algebraic quantity is independent of scale to reflect is quality or 
fundamental nature.  Could we create an additional set of variables 
"_res" that is scaled as others might want (i.e., residuals)?

Dan

Re: interpretation of fit errors

[Ethan A M. <sf...@us...>]

On Thursday, May 05, 2011 12:01:53 am Bastian Märkisch wrote:
> Does somebody have a better name for the proposed setting on how to 
> interpret data errors? The currently proposed
> 
>   set fit relativeerrors|absoluteerrors
> 
> sounds a bit odd to me. See bug #2956524

I may be missing something, but why not just report both
absolute and relative errors?   The user is free to pick whichever
is relevant to the particular case at hand.   No extra options needed.

	Ethan

Re: interpretation of fit errors

[Bastian M. <bma...@we...>]

Am 05.05.2011 19:58, schrieb Ethan A Merritt:
> On Thursday, May 05, 2011 12:01:53 am Bastian Märkisch wrote:
>> Does somebody have a better name for the proposed setting on how to
>> interpret data errors? The currently proposed
>>
>>    set fit relativeerrors|absoluteerrors
>>
>> sounds a bit odd to me. See bug #2956524
>
> I may be missing something, but why not just report both
> absolute and relative errors?   The user is free to pick whichever
> is relevant to the particular case at hand.   No extra options needed.
>
> 	Ethan

I think you just proved that the name may be misleading ;). This is 
about the interpretation of data errors (input), and the resulting 
scaling with FIT_STDFIT of reported variable errors (output). This is 
not about reporting relative errors as in a_err/a.

Not only is scaling the errors wrong in certain cases, but these error 
values are also saved to variables for further processing, e.g. in 
labels. Since, the user should be able to select. With `set fit abs` 
gnuplot reports the same errors as e.g. CERN Minuit.

Any other ideas about the names? weights|errors? scaling|noscaling? 
weights|real?

   Bastian