Re: [Gnuplot-info] Help request - linear regression giving screwy results

[Thomas S. <t.s...@fz...>]

let's look at your data:

Dec2009 5.2
...
Mar2011 34.7

dates in gnuplot are calculated as 'seconds since beginning
of the gnuplot epoch' which is '2000-01-01 00:00:00'. so internally
your data are:

gnuplot> print strptime("%b%Y","Dec2009")
312940800.0
gnuplot> print strptime("%b%Y","Mar2011")
352252800.0

312940800.0 5.2
...
352252800.0 34.7

this means the slope 'm' of your linear regression function
f(x) = m*x + b
will be around
gnuplot> print (34.7-5.2)/(352252800.0-312940800.0)
7.50407000407001e-07

the y-axis intercept 'b' will be around
gnuplot> print 34.7 - 7.50407000407001e-07*352252800.0
-229.632967032967

thus the fit algorithm should make small steps in the 1.e-8 range
when varying 'm', but large steps in the 1.e-1 range when varying
'b'. this difference in step size makes it (nearly) impossible for the
fit algorithm to converge properly, so it will end up with a regression
function which crosses your data points but with a wrong slope.

a small step (1.e-8) which has a strong effect on 'm' has 'nearly'
no effect on 'b'.
and a step which has an effect on 'b' will - when applied to 'm' -
result in a regression function which misses the data points totally.
this behavior confuses the fit algorithm.

to make the fit algorithm happy, apply a factor of e.g. 1.e-7 to 'm'

f(x) = 1.e-7 * m *x + b

and the fit algorithm will converge after a few iterations because
the effect of a step in 'm' or 'b' on the deviation of the regression
function from the data is now nearly equal.

not necessary, but also helpful: shift the origin of the regression
function into the x-range defined by the data points (=subtract
10 years from 'x'), this makes the fit algorithm faster.


Eric Raymond-3 wrote:
> 
> Thomas Sefzick <t.s...@fz...>:
>> 
>> on the x-axis you have seconds since 2000-01-01, so your x-values
>> are around 315360000.
>> your y-values are around 20.
>> 
>> this leads to parameters which are different by at least 7 orders of
>> magnitude.
>> 
>> to bring the parameters into the same range, subtract 10 years and scale
>> the
>> resulting x-values:
>> 
>> offset=10*365*24*60*60
>> f(x)=1.e-7*m*(x-offset)+b
>> m=7.
>> b=6.
>> fit f(x) 'comscore.dat' using 1:2 via m,b
> 
> Thank you, that does solve the problem.  I take it the failure of the
> fit to converge properly was a result of some internal problem due
> to floating-point limitations?
> 
> It seems as though this might be a general issue with plots using
> time-valued x data. Might I suggest we do a FAQ entry on this topic?
> If you were to post a draft here I would read and try to polish it it
> for comprehensibility from a user point of view.
> -- 
> 		 http://www.catb.org/~esr/ Eric S. Raymond 
> 
> ------------------------------------------------------------------------------
> Achieve unprecedented app performance and reliability
> What every C/C++ and Fortran developer should know.
> Learn how Intel has extended the reach of its next-generation tools
> to help boost performance applications - inlcuding clusters.
> http://p.sf.net/sfu/intel-dev2devmay
> _______________________________________________
> gnuplot-info mailing list
> gnu...@li...
> https://lists.sourceforge.net/lists/listinfo/gnuplot-info
> 
> 

-- 
View this message in context: http://old.nabble.com/Help-request---linear-regression-giving-screwy-results-tp31627742p31635555.html
Sent from the Gnuplot - User mailing list archive at Nabble.com.