Re: interpretation of fit errors

[Thomas M. <mat...@ph...>]

On 2011-05-06, at 3:56 PM, Daniel J Sebald wrote:

> On 05/06/2011 04:10 PM, Thomas Mattison wrote:
>> 
>> 2.  Gnuplot will indeed refuse to fit a constant function to a single
>> data point, or a line to two points.  Yes, I hadn't done the
>> experiment.  Sorry.
>> 
>> However I regard item 2 as a bug not a feature.  It is perfectly well
>> defined to ask the question, what is the chisquare for agreement
>> between data and model, as a function of the parameter values, for
>> these cases.  There is a perfectly well defined answer to that
>> question.  There is a perfectly well defined point of minimum
>> chisquare, which gives the best fit parameters.  And the curvature of
>> the chisquare curve or surface gives the fit errors.  Nothing
>> anomalous happens in the case where the number of data points equals
>> the number of parameters.
> 
> It seems like it should.  

But it doesn't, from a mathematical point of view.

> From H.B.B.'s last post, if there are only two points and the curve to fit is a line (two parameters, first order equation**) then the fit is exact in the sense that the residuals are zero...if no other information is given about the data.  But that doesn't say anything about the measurement "error" in the fit.  (I sort of prefer the word "uncertainty", but that's just me.)  We can only glean information about the uncertainty of the fit (i.e., the statistics of the original data) if there are additional data points beyond the number of parameters.

You are right that there need to be more measurements than parameters in order to use the chisquare/degree-of-freedom to infer something about the errors of the fit INPUTS from the quality of the fit.

But in many cases, the user ALREADY KNOWS what the input data measurement errors are.  He knows the precision of the markings on his ruler, or how many decimal places his voltmeter has, or square-root-of-N for the bin contents of a histogram, etc [I am not discussing "systematic" errors here, like is the calibration of your voltmeter correct].

In those cases, the "raw" fit errors are the appropriate ones, and the "rescaled" errors are less appropriate.

> 
> OK, so I'm beginning to see there is this vital piece of information about the quality of fit, reflecting the underlying statistics or the uncertainty in the data (i.e., how noisy it is).  That's what is meant by error, right?

The way I would phrase it is, the noise in the input data propagates to noise in the fit parameters.  The "standard errors" of the fit parameters are the standard deviation of the input data (assumed to be known) propagated to the standard deviation of the fit parameters.

> 
>> I hope we can all agree that when fitting a line to two data points
>> that have errors, the slope and intercept DO have errors.
> 
> Yes, but I don't see how one can estimate the size of that error without additional data beyond two data points, unless some information is known apriori.
> 

Sure, but when the user has supplied meaningful y-errors on his data, we have precisely the a-priori informaton that your intuition is telling you is needed.


> But I'm surprised that no well defined, commonly understood terminology has developed for what seems like an important idea in the numerical analysis community.  Bastian points out that:
> 
> * Minuit does not scale
> * Origin offers both options, the default being version dependent
> * SAS and Mathematica default to scaling of errors
> 
> That inconsistency is the problem.

To me, that looks like most programs give you a choice.  Mathematica may default to scaling, but it looks like it's optional.  I don't have much knowledge about SAS, but I would expect a big package like that to have an option buried somewhere.

Minuit comes from the physics community, nuclear and particle physics in particular.  In physics, we tend to have pretty good a-priori estimates for the errors of the measurements going into a fit.  We use the chisquare/DOF as a measure of the (statistical) fit quality.  Physicists would seldom rescale fit parameter errors by sqrt(chisq/DOF).  And since Minuit is normally used in an expert-programming context instead of an ignorant-GUI-user context, the authors would expect the user to be able multiply by sqrt(chisq/DOF) himself.

In bioscience, and even more so in medical and social science, there's often next to nothing known about the errors of the input measurements a-priori.  In such cases, there's not much the user can do except set all the errors equal (or not tell gnuplot any errors, which will have the same result).  Then the "raw" errors from the fit aren't very meaningful, but the "rescaled" errors have at least some utility.  So defaulting to rescaled makes some sence.

My guess would be that there are more gnuplot fit users who don't know much about their input data errors than those who know a great deal about their input data errors.  So if I were forced to make a choice of only one error to present, I'd probably present the rescaled error.  It will be the "right thing" for users who don't give errors or give uniform and arbitrary errors.  And for the people who supply accurate errors, if the model really reflects the data, then chisq/DOF will be close to 1, and the "rescaled" errors will not be very different, very often, from the "raw" errors.


But why not give both?





                                      Cheers

Prof. Thomas Mattison           Hennings 276
University of British Columbia  Dept. of Physics and Astronomy
6224 Agricultural Road          Vancouver  BC        V6T 1Z1  CANADA
mat...@ph...         phone: 604-822-9690  fax:604-822-5324