gnuplot-beta

fit ignores yrange

[Karl R. <ra...@un...>]

Hi,

as Ethan has started preparing the release of 5.0pl1, I wanted to draw
attention to this bug

https://sourceforge.net/p/gnuplot/bugs/1594/

. Since gp5.0, "fit" ignores any set y range, both from "set yrange" or
inline.

It's a rather bad regression against previous versions, and one that's
possibly hard to spot for some people. It'd be great if the next release
contained a fix.

Many thanks in advance, and of course to all developers!

Best regards

   Karl

Re: fit ignores yrange

[Ethan A M. <sf...@us...>]

On Friday, 05 June, 2015 16:36:12 Karl Ratzsch wrote:
> Hi,
> 
> as Ethan has started preparing the release of 5.0pl1, I wanted to draw
> attention to this bug
> 
> https://sourceforge.net/p/gnuplot/bugs/1594/
> 
> . Since gp5.0, "fit" ignores any set y range, both from "set yrange" or
> inline.

As I understand it, all fits now use "z" as the function range variable.
So in order to restrict the function range, use

  set zrange [min:max]

I don't know what the intent was with regard to adding in-line
range restrictions in the "fit" command.

	Ethan




> It's a rather bad regression against previous versions, and one that's
> possibly hard to spot for some people. It'd be great if the next release
> contained a fix.
> 
> Many thanks in advance, and of course to all developers!
> 
> Best regards
> 
>    Karl
> 
> 
> ------------------------------------------------------------------------------
> _______________________________________________
> gnuplot-beta mailing list
> gnu...@li...
> Membership management via: https://lists.sourceforge.net/lists/listinfo/gnuplot-beta

Re: fit ignores yrange

[Hans-Bernhard B. <HBB...@t-...>]

Am 05.06.2015 um 22:49 schrieb Ethan A Merritt:

> I don't know what the intent was with regard to adding in-line
> range restrictions in the "fit" command.

The intent should be obvious: 'fit' should accept the same data input 
options as 'plot'.  That's what the documentation always said, and it's 
what the code did until two years ago, and almost did even after that 
change.

Overriding 'set' options on-the-fly has been possible in gnuplot since 
day one, and 'fit' was no exception to that, nor should it be.

'fit' actually still did honor a yrange ... but only in a fit with more 
than one independent variable.  The yrange was ignored only if there was 
one variable, but two range specifications.

Re: fit ignores yrange

[Ethan A M. <sf...@us...>]

On Friday, 05 June, 2015 23:07:19 Hans-Bernhard Bröker wrote:
> Am 05.06.2015 um 22:49 schrieb Ethan A Merritt:
> 
> > I don't know what the intent was with regard to adding in-line
> > range restrictions in the "fit" command.
> 
> The intent should be obvious: 'fit' should accept the same data input 
> options as 'plot'.  That's what the documentation always said, and it's 
> what the code did until two years ago, and almost did even after that 
> change.
> 
> Overriding 'set' options on-the-fly has been possible in gnuplot since 
> day one, and 'fit' was no exception to that, nor should it be.
> 
> 'fit' actually still did honor a yrange ... but only in a fit with more 
> than one independent variable.  The yrange was ignored only if there was 
> one variable, but two range specifications.

Exactly.  

'fit' is treating y as the name of an independent variable, not
as the name of the result f(x).  The question is whether the
intent was to allow
   fit [xmin:xmax][zmin:zmax]  f(x) 'data'
even though the plot command, the documentation, and the
current fit command all treat the second bracketed range
as a restriction on y rather than on z.

Alternatively the intent of 
  "f(x, y, ...) = z  even for the case f(x) = z" 
might have been to allow
   fit [xmin:xmax][][zmin:zmax] f(x) 'data'

I.e. a range restriction on z requires 3 bracketed pairs
even if there is no 'y'.

Neither of these works at the moment, but which would
cause the least confusion?

	Ethan

Re: fit ignores yrange

[<pl...@pi...>]

On 05/06/15 22:49, Ethan A Merritt wrote:
> On Friday, 05 June, 2015 16:36:12 Karl Ratzsch wrote:
>> Hi,
>>
>> as Ethan has started preparing the release of 5.0pl1, I wanted to draw
>> attention to this bug
>>
>> https://sourceforge.net/p/gnuplot/bugs/1594/
>>
>> . Since gp5.0, "fit" ignores any set y range, both from "set yrange" or
>> inline.
>
> As I understand it, all fits now use "z" as the function range variable.
> So in order to restrict the function range, use
>
>    set zrange [min:max]
>
> I don't know what the intent was with regard to adding in-line
> range restrictions in the "fit" command.
>
> 	Ethan
>
>
>
>
>> It's a rather bad regression against previous versions, and one that's
>> possibly hard to spot for some people. It'd be great if the next release
>> contained a fix.
>>
>> Many thanks in advance, and of course to all developers!
>>
>> Best regards
>>
>>     Karl
>>
>>

setting xrange, either explicitly or via a range parameter to fit 
command, works as expected.

  m=c=0.1;fit [1975:2000] lin(x) datafile u 1:2 via m,c;

since x is by convention the independent variable and is REQUIRED to 
have negligible  error  and negligible non linear variability to 
regression to give an accurate estimation of the supposed linear 
relationship, this seems appropriate.

I don't understand where z comes into the discussion in x,y data. :?

Trying to set the range of the fitting process by defining a range on 
the dependent variable suggests it may be being applied correctly. I can 
understand that the algorithm doing the fitting is based on the 
assumption that x is the independent variable and that is it xrange that 
defines any subset used for a particular fit.

Maybe this needs to be stated explicitly in the documentation.

Peter.

Re: fit ignores yrange

[sfeam <sf...@us...>]

On Saturday, 06 June 2015 08:48:38 AM pl...@pi... wrote:
> 
> I don't understand where z comes into the discussion in x,y data. :?
> 

Digging into the history of "fit" documentation, here are extracts from
gnuplot.doc for earlier gnuplot versions. (irrelevant lines trimmed)

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
Version 4.2.5 (Feb 2009)
 Syntax:
       fit {[xrange] {[yrange]}} <function> '<datafile>'

 Ranges may be specified to temporarily limit the data which is to be fitted;
 any out-of-range data points are ignored. The syntax is
       [{dummy_variable=}{<min>}{:<max>}],
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
Version 4.4.0 (Feb 2010)
 The `fit` command can fit a user-supplied expression to a set of data points
 (x,z) or (x,y,z), using an implementation of the nonlinear least-squares
 (NLLS) Marquardt-Levenberg algorithm.

 Syntax:
       fit {<ranges>} <expression>

 Ranges may be specified to temporarily limit the data which is to be fitted;
 any out-of-range data points are ignored. The syntax is
       [{dummy_variable=}{<min>}{:<max>}],

 The default data formats for fitting functions with a single
 independent variable, z=f(x), are z or x:z.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

So the change from treating the function range as "y" to treating it as "z" 
happened in the 4.3 development period and first appeared in the 4.4 release.

I have confirmed that the 4.4.0 executable applied "set zrange"
rather than "set yrange" as a filter on data read by 
    fix f(x) 'data'

So really this discussion is about 5 years too late.
We do try to maintain backwards compatibility where possible,
but ...
 
Given the ambiguity of whether the second pair of inline brackets
in a fit command refers to y or z, isn't it better not to allow this at all,
or issue a warning:
   gnuplot> fit [][ymin:ymax] f(x) 'data' via a,b
   Warning:  bracketed range on "y" is ignored

Note that the documentation has always [incorrectly!] implied that it would
work to say
   gnuplot> fit [][z=ymin:ymax] f(x) 'data' via a,b

Maybe we should fix this and *require* the z= form of the command?
   gnuplot> fit [][ymin:ymax] f(x) 'data' via a,b
   Error:   Second bracketed range in a 1-parameter fit must begin with z=


			Ethan

> >> It's a rather bad regression against previous versions, and one that's
> >> possibly hard to spot for some people. It'd be great if the next release
> >> contained a fix.
> >>
> >> Many thanks in advance, and of course to all developers!
> >>
> >> Best regards
> >>
> >>     Karl
> >>
> >>
> 
> setting xrange, either explicitly or via a range parameter to fit 
> command, works as expected.
> 
>   m=c=0.1;fit [1975:2000] lin(x) datafile u 1:2 via m,c;
> 
> since x is by convention the independent variable and is REQUIRED to 
> have negligible  error  and negligible non linear variability to 
> regression to give an accurate estimation of the supposed linear 
> relationship, this seems appropriate.
> 
> I don't understand where z comes into the discussion in x,y data. :?
> 
> Trying to set the range of the fitting process by defining a range on 
> the dependent variable suggests it may be being applied correctly. I can 
> understand that the algorithm doing the fitting is based on the 
> assumption that x is the independent variable and that is it xrange that 
> defines any subset used for a particular fit.
> 
> Maybe this needs to be stated explicitly in the documentation.
> 
> Peter.
> 
> 
> 
> 
> 
> 
> ------------------------------------------------------------------------------
> _______________________________________________
> gnuplot-beta mailing list
> gnu...@li...
> Membership management via: https://lists.sourceforge.net/lists/listinfo/gnuplot-beta

Re: fit ignores yrange

[<pl...@pi...>]

On 06/06/15 18:08, sfeam wrote:
> On Saturday, 06 June 2015 08:48:38 AM pl...@pi... wrote:
>
>  >
>
>  > I don't understand where z comes into the discussion in x,y data. :?
>
>  >
>
> Digging into the history of "fit" documentation, here are extracts from
>
> gnuplot.doc for earlier gnuplot versions. (irrelevant lines trimmed)
>
> %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
>
> Version 4.2.5 (Feb 2009)
>
> Syntax:
>
> fit {[xrange] {[yrange]}} <function> '<datafile>'
>
> Ranges may be specified to temporarily limit the data which is to be fitted;
>
> any out-of-range data points are ignored. The syntax is
>
> [{dummy_variable=}{<min>}{:<max>}],
>
> %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
>
> Version 4.4.0 (Feb 2010)
>
> The `fit` command can fit a user-supplied expression to a set of data points
>
> (x,z) or (x,y,z), using an implementation of the nonlinear least-squares
>
> (NLLS) Marquardt-Levenberg algorithm.
>
> Syntax:
>
> fit {<ranges>} <expression>
>
> Ranges may be specified to temporarily limit the data which is to be fitted;
>
> any out-of-range data points are ignored. The syntax is
>
> [{dummy_variable=}{<min>}{:<max>}],
>
> The default data formats for fitting functions with a single
>
> independent variable, z=f(x), are z or x:z.
>
> %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
>
> So the change from treating the function range as "y" to treating it as "z"
>
> happened in the 4.3 development period and first appeared in the 4.4
> release.
>
> I have confirmed that the 4.4.0 executable applied "set zrange"
>
> rather than "set yrange" as a filter on data read by
>
> fix f(x) 'data'
>
> So really this discussion is about 5 years too late.
>
> We do try to maintain backwards compatibility where possible,
>
> but ...
>
> Given the ambiguity of whether the second pair of inline brackets
>
> in a fit command refers to y or z, isn't it better not to allow this at all,
>
> or issue a warning:
>
> gnuplot> fit [][ymin:ymax] f(x) 'data' via a,b
>
> Warning: bracketed range on "y" is ignored
>
> Note that the documentation has always [incorrectly!] implied that it would
>
> work to say
>
> gnuplot> fit [][z=ymin:ymax] f(x) 'data' via a,b
>
> Maybe we should fix this and *require* the z= form of the command?
>
> gnuplot> fit [][ymin:ymax] f(x) 'data' via a,b
>
> Error: Second bracketed range in a 1-parameter fit must begin with z=
>
> Ethan

Thanks Ethan.

that may be a start but I think the issue of whether this is legitimate 
application of least squares regression needs to be addressed.

x is the independant variable thus any limits on the data range should 
be expressed in x coordinates.

Since y ( or z ) values are assumed by the fitting algo to be error 
laden using them to select a range is not valid.  If there is a 
need/desire to do this maybe the regression should be being done the 
other way around.

There are huge problems in many fields of science from the lack of 
appreciation of the assumptions and pre-conditions required for 
least-squares to provide a result that is a valid estimator of the 
regressed function.

Perhaps someone could suggest why this feature is there at all and in 
which circumstances it can be considered a legitimate application of LSQ.

Peter.



>
>  > >> It's a rather bad regression against previous versions, and one that's
>
>  > >> possibly hard to spot for some people. It'd be great if the next
> release
>
>  > >> contained a fix.
>
>  > >>
>
>  > >> Many thanks in advance, and of course to all developers!
>
>  > >>
>
>  > >> Best regards
>
>  > >>
>
>  > >> Karl
>
>  > >>
>
>  > >>
>
>  >
>
>  > setting xrange, either explicitly or via a range parameter to fit
>
>  > command, works as expected.
>
>  >
>
>  > m=c=0.1;fit [1975:2000] lin(x) datafile u 1:2 via m,c;
>
>  >
>
>  > since x is by convention the independent variable and is REQUIRED to
>
>  > have negligible error and negligible non linear variability to
>
>  > regression to give an accurate estimation of the supposed linear
>
>  > relationship, this seems appropriate.
>
>  >
>
>  > I don't understand where z comes into the discussion in x,y data. :?
>
>  >
>
>  > Trying to set the range of the fitting process by defining a range on
>
>  > the dependent variable suggests it may be being applied correctly. I can
>
>  > understand that the algorithm doing the fitting is based on the
>
>  > assumption that x is the independent variable and that is it xrange that
>
>  > defines any subset used for a particular fit.
>
>  >
>
>  > Maybe this needs to be stated explicitly in the documentation.
>
>  >
>
>  > Peter.
>
>  >
>
>  >
>
>  >
>
>  >
>
>  >
>
>  >
>
>  >
> ------------------------------------------------------------------------------
>
>  > _______________________________________________
>
>  > gnuplot-beta mailing list
>
>  > gnu...@li...
>
>  > Membership management via:
> https://lists.sourceforge.net/lists/listinfo/gnuplot-beta
>
>
>
> ------------------------------------------------------------------------------
>
>
>
> _______________________________________________
> gnuplot-beta mailing list
> gnu...@li...
> Membership management via: https://lists.sourceforge.net/lists/listinfo/gnuplot-beta
>

Re: fit ignores yrange

[sfeam <sf...@us...>]

On Saturday, 06 June 2015 06:59:45 PM pl...@pi... wrote:
> >
> > gnuplot> fit [][ymin:ymax] f(x) 'data' via a,b
> > Warning: bracketed range on "y" is ignored
> >
> > or
> >
> > gnuplot> fit [][ymin:ymax] f(x) 'data' via a,b
> > Error: Second bracketed range in a 1-parameter fit must begin with z=
> >

> Thanks Ethan.
> 
> that may be a start but I think the issue of whether this is legitimate 
> application of least squares regression needs to be addressed.
> 
> x is the independant variable thus any limits on the data range should 
> be expressed in x coordinates.

>From a data-centric viewpoint, I don't think I buy that argument.
When both x and y are experimentally determined values, why should
you be able to limit the acceptable range of one but not the other?

> Since y ( or z ) values are assumed by the fitting algo to be error 
> laden using them to select a range is not valid.  

I do not follow your line of thinking there, unless you are
suggesting that the limit should be applied to (y +/- yerror)
rather than to y.

Anyhow, there can also be errors associated with x.

I realize the default form of the fit command does not handle errors on x,
but version 5 introduced an option "fit xyerror" that does.

> If there is a 
> need/desire to do this maybe the regression should be being done the 
> other way around.
> 
> There are huge problems in many fields of science from the lack of 
> appreciation of the assumptions and pre-conditions required for 
> least-squares to provide a result that is a valid estimator of the 
> regressed function.
> 
> Perhaps someone could suggest why this feature is there at all and in 
> which circumstances it can be considered a legitimate application of LSQ.

If you mean "why do we allow bracketed ranges in the fit command?"
the only justification I see is backwards compatibility.
I recommend against using bracketed ranges in any of the commands,
'plot', 'fit', or 'stats'.  

If you mean "why do we allow filtering on z range?", I think that
is a case of providing a tool and leaving it up to the user to make
responsible use of it.   I don't use gnuplot's fit routine that much,
and when I do I usually pre-filter the data in an external tool before
giving it to gnuplot.  But there are times when it is convenient
to filter on the range of either x or y in 2D experimental data,
or on the range of all observed values in higher dimensional data.

	Ethan