#191 Simple extension to >3D fitting, syntax aspects
Hi,
I've got a feature request as well as a preliminary implementation. I hope that this place is okay to get this started.
Currently gnuplot supports fitting to z=f(x) and z=f(x,y), where x or x,y are dummy variables for data file columns, and subjected to x/yrange.
I need to fit functions based on more than two columns, e.g. in a simple "5-D" case I'd like to do
f(x1,x2,x3,x4) = a * x1 + b * x2 + c * x3 + d * x4
fit f(x1,x2,x3,x4) "data" using 1:2:3:4:5:(1) via a,b,c,d
(Note that this is not about the number of "via" variables.)
The extensions of fit.c are straightforward and involve only minor modifications, but there are some design issues needing discussion.
1) the name of the dummy variables:
Obviously, we already have the naming f(x) and f(x,y) and z fixed. A general extensible approach would be f(x1,x2,...).
At the moment, I didn't manage to keep the old names for compatibility, so it's f(x,y,x2,x3,x4,...)
2) range checking:
Currently, I do not limit the additional dummy variables x2,x3... to a range.
Since there are t/u/v/x/y/x2/y2/z ranges, one could imagine to include x3,x4... ranges too. I've no idea of the consequences (in terms of coding effort).
3) number of dummy variables:
The current limit is MAXDATACOL == 7, limiting the number of using specs. This leaves us with 5 independent variables f(x,y,x2,x3,x4).
I have not yet looked into increasing this limit. Would this have undesired side effects? - If not:
The next limit is MAX_NUM_VAR == 12 (except for DOS16), which seems sufficient for most purposes, and would be consistent with user expectations: if you can define functions with MAX_NUM_VAR variables, you'd except to be able to fit to them, too.
My current implementation adapts to the minimum of both limits, thus it should not cause trouble if any can be incresed later on.
4) less extensible alternative:
Would be to use the 5 existing dummy names x,y,t,u,v,
hardcode for 5 variables, and use the existing ranges.
Drawback: harder to change (in a compatible and consistent way) for more variables later on.
Comments are very welcome. If needed at this stage, I'm willing to submit a patch to fit.c, and later to provide example data (demos) and to edit the documentation accordingly.
While I'm just beginning to run tests, I thought it worthwhile to present the current issues for discussion (so that I won't need to change all my fine new gnuplot fitting scripts later on :-)
All the best,
Hanno
Logged In: YES
user_id=235620
Originator: NO
I will now display my ignorance about the fit commands...
Are you really limited by MAXDATACOL?
Does it not work to define your function
f(x,y, column(3), column(4), column(5), ...)
instead of
f(x,y,x2,x3,x4,...)?
There is no limit to the number of columns read while evaluating an input function. The return value is a single number, and only takes up 1 entry in v[MAXDATACOL]. Certainly for a "plot" command you could say
plot 'data' using 1:($2+$3+$4+...+$99)
Does the 'fit' command not work the same way?
Logged In: YES
user_id=1863852
Originator: YES
Alas, no...: gnuplot complains that "column() called from invalid context"
The 'fit' command is limited to exactly the two function argument lists '(x)' and '(x,y)', verbatim.
Yes, you can use column calculations in 'using' expressions just as with 'plot'. But this doesn't help once the columns are independent variables in the fitting function. I'll try to explain:
My example was about fitting coefficients a, b, c, d to express some data as a linear combination of basis functions, which are not given analytically, but only tabulated. The data file contains columns
# x f1(x) f2(x) f3(x) f4(x) data1 data2 ...
Now the model is model(x) = a * f1(x) + b * f2(x) + c * f3(x) + d * f4(x)
and all I want to do is to fit data1 to the model, like
fit model(x) "datafile" using 1:6 via a,b,c,d
which would work if f1(x), f2(x), f3(x), f4(x) were user-defined functions.
'fit' would use some starting values for a,b,c,d. It would calculate for each data point model(x) and the difference model(x) - data1(x), the latter being taken from the file "using 1:6".
Then a,b,c,d are iteratively adjusted and model(x) is calculated again, until the difference is not getting better any more.
The problem is only about calculating the model function, which can be written as
model(f1,f2,f3,f4) = a * f1 + b * f2 + c * f3 + d * f4.
since it does not explicitly depend on x. So I have four independent variables f1,f2,f3,f4 in my data file, and the data1 column which I want to fit against. To calculate the model function I have to take the f1...f4 values from columns "using 2:3:4:5:...".
A "using" expression with a,b,c,d appearing inside it, like
using 1:(a * $2 + b * $3 + c * $4 + d * $5)
is not useful, because this gets evaluated only once, when reading the file, and thus depends on the values of a,b,c,d at that time, but not on the necessary variations of a,b,c,d during the fit. This means the fit doesn't, ehm, fit.
So I really need to say something like
fit model(f1,f2,f3,f4) "datafile" using 2:3:4:5:6 via a,b,c,d
where '2:3:4:5' read the four independent variables f1,f2,f3,f4, and ':6' reads the data to be fitted.
This means that during iteration, all the columns must be available so that the model function can be calculated again and again with changing values of a,b,c,d. THAT IS THE POINT. Sorry for this long post, I found it surprisingly hard to explain...
For historical reasons, gnuplot expects fits to be two-dimensional, i.e. of the form y=f(x), when the using spec has one, two, or three columns, similar to 2-D plots (where the three-column case corresponds to 'with yerrorbars'). For x and y, the xrange and yrange are used to clip the data set.
A three-dimensional fit must be indicated by use of f(x,y) and a using spec with four columns. So there must always be an error estimate, which you can set to 1 if no real error information is available. Now z=f(x,y), and the three x/y/z-range are used to clip.
To continue this tradition, we should use n+2 'using' specs for fits of functions with n parameters. Since the function value is a scalar, the plot of it would be n+1-dimensional in gnuplot-speak.
And - finally - this is the reason why I've hit the MAXDATACOLS limit...to answer the first question last.
Did this make some sense to you?
Hanno
Logged In: YES
user_id=27517
Originator: NO
No,
fit f(x,y, column(3), column(4), column(5), ...) "data"
can't work, because column() is only allowed in an extended using specification. The fitting function isn't one. 'fit' is different from plot. Very roughly said, it's like a datafile plot of the input file, followed by repeated function plots of the target function, for different parameter sets.
> The 'fit' command is limited to exactly the two function argument lists
> '(x)' and '(x,y)', verbatim.
Not really. You don't even have to use functions and argument lists at all, nor do they have to be named 'x' and 'y'.
The limitation is in the number of independent variables, and 'fit' inherited that from the main body of gnuplot. gnuplot can't plot a function of three arguments, so it won't fit it either. Extending this would be possible, but probably harder than you expect. 'fit' relies on the gnuplot core for function evaluation and other things, including specifications of ranges and names for the independent variables. If there are more than 2, central data structures (basically the axis_array[]) have to be modified.
Logged In: YES
user_id=235620
Originator: NO
> fit f(x,y, column(3), column(4), column(5), ...) "data"
> can't work, because column() is only allowed in an extended
> using specification. The fitting function isn't one.
> 'fit' is different from plot. Very roughly said, it's like
> a datafile plot of the input file, followed by repeated
> function plots of the target function, for different
> parameter sets.
I understand the limitation of the column(n) function. But I was imagining that the fit algorithm would place the evaluation of the function inside the 'using' context. That is how it works for plot commands. But if I understand your rough description, 'fit' only reads the data file once and then repeated evaluates the function against the internal copy of the data. Is that right?
So does this mean that 'fit' would work on arbitrary columns also, at the cost of having to re-read the data file each time the function is evaluated, rather than working from an internal copy of the data?
Yeah, I know all answers lie in the source code. But the fit subsystem is a large chunk of code that I have so far had no reason to delve into.
Logged In: YES
user_id=27517
Originator: NO
> But if I
> understand your rough description, 'fit' only reads the data file once and
> then repeated evaluates the function against the internal copy of the data.
> Is that right?
Yes. 'using' is applied only once.
> So does this mean that 'fit' would work on arbitrary columns also, at the
> cost of having to re-read the data file each time the function is
> evaluated, rather than working from an internal copy of the data?
Yes. But doing that would not only be quite wasteful (not even to mention problematic if the file changes during the fit), and it would mean a major overhaul of the entire fit.c module. Basically the entire interface between gnuplot/user/data and the fitting algorithm itself would have to be redone.
Logged In: YES
user_id=1863852
Originator: YES
I beg to disagree, since the necessary columns can be filled and stored during the file read with minimal changes to the current code. I've attached an example usage which will work in the one- nad two-dimensional case, and with my patch also for higher dimensions.
I'd be happy to send in the patch for testing; however I'd like to clarify the syntax issues before submitting a "more finalized" version of the patch. Is it okay to upload it here rather than in the "Patches" section? Or would you like me to send it directly to you for inspection, Hans-Bernhard?
BTW, sorry for the delay, bad timing due to vacation...
Hanno
File Added: fit5d.zip
Example usage of multidimensional fit
gnuplot supports 5 independent variables in fits since version 4.4.