gnuplot-beta

fitting functions with more independent variables

[James R. V. Z. <jr...@co...>]

Currently gnuplot has these data formats for fitting functions with
one or two independent variables:

      y
      x:y
      x:y:s
      x:y:z:s

I propose to implement these additional data formats for fitting
functions with 3-5 independent variables:

      v:x:y:z:s
      u:v:x:y:z:s
      t:u:v:x:y:z:s

For example, if there are five columns:

  The first column inherits its data range and time/date flag from the
  V axis unless a specific data range is given, and has the dummy
  variable 'v' unless a different name is given in a range or "set
  dummy v" statement.  The first range spec in a fit command would
  affect this variable.

  The second column inherits its data range and time/date flag from
  the first X axis unless a specific data range is given, and has the
  dummy variable 'x' unless a different name is given in a range or
  "set dummy x" statement.  The second range spec in a fit command
  would affect this variable.

  The third column inherits its data range and time/date flag from the
  first Y axis unless a specific data range is given, and has the
  dummy variable 'y' unless a different name is given in a range or
  "set dummy y" statement.

  The fourth column has the dependent variable, and inherits from the
  Z axis.

  The last column gives the error standard deviation.

Linear regression would look like this:

  h(v,x,y) = a*v + b*x + c*y
  fit h(v,x,y) 'foo.dat' using 1:2:3:4:(1) via a,b,c

  fit [q=-8:8] [r=*:*] [s=*:*] [t=0:10] a*q+b*r+c*s+d*t \
      'foo.dat' using 1:2:3:4:5:(1) via a,  b,  c,  d

I think I see how to do this.  

Comments?

         - Jim Van Zandt

Re: fitting functions with more independent variables

[Ethan M. <merritt@u.washington.edu>]

On Saturday 31 January 2009 17:37:51 James R. Van Zandt wrote:
> 
> Currently gnuplot has these data formats for fitting functions with
> one or two independent variables:
> 
>       y
>       x:y
>       x:y:s
>       x:y:z:s
> 
> I propose to implement these additional data formats for fitting
> functions with 3-5 independent variables:
> 
>       v:x:y:z:s
>       u:v:x:y:z:s
>       t:u:v:x:y:z:s

I am not familiar with the "fit" subsystem, so I may be missing
some essential point....
but the above looks very strange  to me.  Why stick your additional
dummy variables at the start rather than the end?  Surely it is
more natural to have
	x
	x:y
	x:y:z
	...
	x:y:z:t:u:v

> For example, if there are five columns:
> 
>   The first column inherits its data range and time/date flag from the
>   V axis unless a specific data range is given, and has the dummy
>   variable 'v' unless a different name is given in a range or "set
>   dummy v" statement.  The first range spec in a fit command would
>   affect this variable.
> 
>   The second column inherits its data range and time/date flag from
>   the first X axis unless a specific data range is given, and has the
>   dummy variable 'x' unless a different name is given in a range or
>   "set dummy x" statement.  The second range spec in a fit command
>   would affect this variable.
> 
>   The third column inherits its data range and time/date flag from the
>   first Y axis unless a specific data range is given, and has the
>   dummy variable 'y' unless a different name is given in a range or
>   "set dummy y" statement.
> 
>   The fourth column has the dependent variable, and inherits from the
>   Z axis.
> 
>   The last column gives the error standard deviation.
> 
> Linear regression would look like this:
> 
>   h(v,x,y) = a*v + b*x + c*y
>   fit h(v,x,y) 'foo.dat' using 1:2:3:4:(1) via a,b,c

Wouldn't it be far more natural to say
    h(x,y,z) = a*x + b*y + c*z
    fit h(x,y,z) 'foo.dat' using 1:2:3:4 via a,b,c

As I say, I am not familiar with the 'fit' subsystem, but that's
what I would expect it to behave like, and I suspect other naive
users would expect the same.

	Ethan

>   fit [q=-8:8] [r=*:*] [s=*:*] [t=0:10] a*q+b*r+c*s+d*t \
>       'foo.dat' using 1:2:3:4:5:(1) via a,  b,  c,  d
> 
> I think I see how to do this.  
> 
> Comments?
> 
>          - Jim Van Zandt

-- 
Ethan A Merritt

Re: fitting functions with more independent variables

[James R. V. Z. <jr...@co...>]

 Ethan Merritt <merritt@u.washington.edu> writes:
> On Saturday 31 January 2009 17:37:51 James R. Van Zandt wrote:
> > 
> > Currently gnuplot has these data formats for fitting functions with
> > one or two independent variables:
> > 
> >       y
> >       x:y
> >       x:y:s
> >       x:y:z:s
> > 
> > I propose to implement these additional data formats for fitting
> > functions with 3-5 independent variables:
> > 
> >       v:x:y:z:s
> >       u:v:x:y:z:s
> >       t:u:v:x:y:z:s

> I am not familiar with the "fit" subsystem, so I may be missing
> some essential point....
> but the above looks very strange  to me.  Why stick your additional
> dummy variables at the start rather than the end?  Surely it is
> more natural to have
>	   x
>	   x:y
>	   x:y:z
           x:y:z:t
           x:y:z:t:u
>	   x:y:z:t:u:v

Currently, the number of using specs determines the number of
independent variables.  If there are two independent variables, there
must be four using specs - one for each of those independent
variables, one for the dependent variable, and one for the error.

> > Linear regression would look like this:
> > 
> >   h(v,x,y) = a*v + b*x + c*y
> >   fit h(v,x,y) 'foo.dat' using 1:2:3:4:(1) via a,b,c
>
> Wouldn't it be far more natural to say
>     h(x,y,z) = a*x + b*y + c*z
>     fit h(x,y,z) 'foo.dat' using 1:2:3:4 via a,b,c

There are several issues here:

First, you have four specs, which would mean there are two independent
variables.  I would rather not not add some other mechanism to specify
the number of independent variables, so for three independent variables
we need to add a spec for the error.  E.g.:

    fit  <expression>  'foo.dat' using 1:2:3:4:(1) via a,b,c

Next we need to decide what to call the colums.  At present, we are
using Z for the dependent variable, S for the error, and X and Y for
independent variables.  It turns out that each column of data read
from a file must be associated with an axis (that's how the minimum
and maximum values are stored).  So for the rest of the independent
variables I chose the axes with names nearest X.

Third, we need to decide order of the columns read from the data file.
I am assuming the independent variables are first, then the dependent
variable, then the error.

Fourth, how to allocate dummy variable names to the columns.  I chose
alphabetical order.  We could instead say that the first two dummy
variables are always X and Y, and we add others in reverse
alphabetical order:

   x:z
   x:z:s
   x:y:z:s
   x:y:v:z:s
   x:y:v:u:z:s
   x:y:v:u:t:z:s

Or we could start with T and add in alphabetical order:

   x:z:s
   x:y:z:s
   x:y:t:z:s
   x:y:t:u:z:s
   x:y:t:u:v:z:s

I think these would be more awkward.

On the other hand, we could require the user to supply a range spec
with a dummy variable name for each independent variable.  That way he
could use whatever names he likes.  The only rules would be that the
first range spec corresponds to the first using spec, etc., and the
last two using specs would still be for the dependent variable and the
error.  E.g.:

  fit  [lat=*:*] [lon=0:pi] [alt=0:4000] a*sin(lat)+b*cos(lon)+c*alt  \
  'foo.dat' using 1:2:3:4:(1) via a,b,c

The data file might start like this:

  # lat   lon     alt   temp
  2.34    48.86   211   13
  5.37    43.31   820   22
  4.83    45.76   443   8
  1.45    43.62   411   23
  7.27    43.70   331   32
  -1.57   47.23   282   18

I think I would still need to store the min and max values in
axis_array entries.   I could add some extra entries, so I don't
interfere with the current axes.

BTW I'm still debugging.  So far the new code only works with one
independent variable...

         - Jim Van Zandt

Re: fitting functions with more independent variables

[James R. V. Z. <jr...@co...>]

  
My new fitting code is working.

I had proposed these formats:

>>>       v:x:y:z:s
>>>       u:v:x:y:z:s
>>>       t:u:v:x:y:z:s

Peter and Ethan objected to the ordering.  I eventually changed my
mind, deciding the following set of formats was easier to implement,
to explain, and to remember:

       z
       x:z
       x:z:s
       x:y:z:s
       x:y:t:z:s
       x:y:t:u:z:s
       x:y:t:u:v:z:s

If you have three or more using specs, then the last one is for
standard deviation, the next to last is for Z values (dependent
variable), and all the preceding ones are for independent variables.
And the first two dummy variables are always 'x' and 'y'.

Here are some examples:

  fit a*x + b*y + c*t 'foo.dat' using 1:2:3:4:(1) via a,b,c
  h(x,y,t,u,v) = a*x + b*y + c*t + d*u + e*v
  fit h(x,y,t,u,v) 'foo.dat' using 1:2:3:4:5:6:(1) via a,b,c,d,e

You can also use ranges, including those that change the dummy
variable names:

  g(x,yaks,tulips) = a*x+b*yaks+c*tulips
  fit [*:9][yaks=*:*][tulips=*:*] g(x,yaks,tulips) 'surface.dat' u 1:2:3:(1) via a,b,c


I have uploaded a patch and a set of data files and demo scripts here:

       http://jrv.oddones.org/fit-patch
       http://jrv.oddones.org/fit-demo.tar.gz

The only code I touched was in fit.c.  However, it involved a lot of
things I have not worked with before, so I would rather other
developers would check out the patch before I commit the changes.

For example:
  
  I'm using the T, U, and V axes to store range information.  Are there
  side effects?  Would it be better to set those entries in df_axis to
  NO_AXIS and store range information separately?
  
  You can still do a one- or two-variable fit with a function using
  "t" as a regular user-defined variable.  However, if your fit
  function includes "t" but you haven't defined it, you don't get an
  "undefined variable" error.  That's because while gnuplot is parsing
  the expression, it assumes "t" is an independent variable, and
  doesn't discover differently until later.  How can this be fixed?
  
  I don't think there are any new memory leaks.  Is there a way to
  automatically check for leaks?

  I think any of the new columns can have time formats.  However, I
  don't use time data myself, and haven't tested it.
  
  Though this code allows up to five independent variables, I would
  really rather there was no fixed limit.  There are at least a couple
  of challenges in removing the limit:

  - df_readascii is set up to read a maximum of seven columns.
  - For an arbitrary number of variables, we would also need a set of
    dummy variable names that can be extended.  For example:
  
         z
         x:z
         x:z:s
         x:y:z:s
         x:y:t1:z:s
         x:y:t1:t2:z:s
         x:y:t1:t2:t3:z:s
         x:y:t1:t2:t3:t4:z:s
         ...
  
  This would be incompatible with the names I have used. 
  
  Or, we could decide that any user who wants more than five independent
  variables would have to supply their own dummy variable names in range
  specs.

I noticed Z ranges were implemented in the code, but were not
documented or reported in the log file.  The log file reported "using"
specs and the fit function.  Also, if some range spec changed a dummy
variable name, the new name was not reported, making it hard to
interpret the reported range and fit function.  I have fixed those
bugs.

Every now and then while fitting, I have gotten an error message "No
data to fit" which brings me up short.  So while I was working in the
area, I expanded the report like this:

  fit [-4:4][yaks=-2:2][4:4][0:22]  a*x+b*yaks+c*t 'lin3.dat' u 1:2:3:4:5 via a,b,c
       Read 64 points
       Skipped 32 points outside range [x=-4:4]
       Skipped 24 points outside range [yaks=-2:2]
       Skipped 6 points outside range [t=4:4]
       Skipped 2 points outside range [z=0:22]
       No data to fit 

(If there are points to fit, nothing new is printed.)

So, let me know what you think.

           - Jim Van Zandt

Re: fitting functions with more independent variables

[Ethan A M. <merritt@u.washington.edu>]

On Friday 13 February 2009, James R. Van Zandt wrote:

>   Though this code allows up to five independent variables, I would
>   really rather there was no fixed limit.  There are at least a couple
>   of challenges in removing the limit:
> 
>   - df_readascii is set up to read a maximum of seven columns.

That isn't quite true.  The limit comes from the number of slots in each
point's data structure.  df_readascii itself is quite happy to read from
as many columns as you like.  It just cannot store all of them into the
data structure belonging to a single point.  There are already several 
plot styles or input modes that involve reading more data than fits into
7 slots.  Plot "with vectors", for example, allocates two data structures
for each point so that it can store values for each end of the vector.
And the 'matrix' keyword allows reading an unlimited number of values
per line (at least so far as I know).  But I don't know if either of these
is a useful model for what you want to do.

	Ethan


>   - For an arbitrary number of variables, we would also need a set of
>     dummy variable names that can be extended.  For example:
>   
>          z
>          x:z
>          x:z:s
>          x:y:z:s
>          x:y:t1:z:s
>          x:y:t1:t2:z:s
>          x:y:t1:t2:t3:z:s
>          x:y:t1:t2:t3:t4:z:s
>          ...
>   
>   This would be incompatible with the names I have used. 
>   
>   Or, we could decide that any user who wants more than five independent
>   variables would have to supply their own dummy variable names in range
>   specs.
> 
> I noticed Z ranges were implemented in the code, but were not
> documented or reported in the log file.  The log file reported "using"
> specs and the fit function.  Also, if some range spec changed a dummy
> variable name, the new name was not reported, making it hard to
> interpret the reported range and fit function.  I have fixed those
> bugs.
> 
> Every now and then while fitting, I have gotten an error message "No
> data to fit" which brings me up short.  So while I was working in the
> area, I expanded the report like this:
> 
>   fit [-4:4][yaks=-2:2][4:4][0:22]  a*x+b*yaks+c*t 'lin3.dat' u 1:2:3:4:5 via a,b,c
>        Read 64 points
>        Skipped 32 points outside range [x=-4:4]
>        Skipped 24 points outside range [yaks=-2:2]
>        Skipped 6 points outside range [t=4:4]
>        Skipped 2 points outside range [z=0:22]
>        No data to fit 
> 
> (If there are points to fit, nothing new is printed.)
> 
> So, let me know what you think.
> 
>            - Jim Van Zandt

-- 
Ethan A Merritt

Re: fitting functions with more independent variables

[James R. V. Z. <jr...@co...>]

Hearing no objection, I've committed my revised fitting code.  I added
a section to demo/fit.dem to demonstrate fitting with three
independent variables and revised docs/gnuplot.doc.

              - Jim Van Zandt