Menu
▾
▴
gnuplot-beta
|
From: Levente <ln...@dr...> - 2007-10-18 19:28:33
|
AFAIK calculating or approximating the 1st order derivative of a function or x-y dataset was not (easily?) possible a few years ago from within gnuplot, but maybe this has changed since then -- I don't follow closely gnuplot's development anymore. My problem is the following: from a dataset, I first plot the x-y values, then either fit a function to them or make a smoothing with acsplines. So far, no problem. But afterwards, I need the 1st order derivative of the fitted or smoothed curve. This would be easy if gnuplot had a way to access data not only from the _actual_ row but also from the following one in order to calculate Delta y / Delta x = y(row +1)-y(row)/(x(row+1)-x(row)). This was not possible earlier. Has the situation changed since then? If not, is it still possible to perform differentiation from within gnuplot (even if this means using some awk script or similar), as I would like to automate the process and gnuplot is easily driven from a shell script or batch file. Levente |
|
From: <HBB...@t-...> - 2007-10-18 20:13:47
|
Levente Novák wrote: > AFAIK calculating or approximating the 1st order derivative of a > function or x-y dataset was not (easily?) possible a few years ago from > within gnuplot, but maybe this has changed since then It hasn't. Numeric differentiation (and integration, and lots of other sides) is on the wrong side of the boundary between plotting and data processing. > But afterwards, I need the 1st order > derivative of the fitted or smoothed curve. For the fitted curve, that should be reasonably easy. Derive it algebraically (you know --- that thing you do with pencil, paper and scratching your skull), then have gnuplot plug in the fitted parameters and plot it. > If not, is it still possible to perform differentiation from within > gnuplot (even if this means using some awk script or similar), as I > would like to automate the process and gnuplot is easily driven from > a shell script or batch file. Nothing changed in that area, either. plot "< myscript datafile" works just as well as it always did. |
|
From: Juergen W. <wie...@fr...> - 2007-10-22 11:51:45
|
> AFAIK calculating or approximating the 1st order derivative of a > function or x-y dataset was not (easily?) possible a few years ago from > within gnuplot, but maybe this has changed since then -- I don't follow > closely gnuplot's development anymore. > > My problem is the following: from a dataset, I first plot the x-y > values, then either fit a function to them or make a smoothing with > acsplines. So far, no problem. But afterwards, I need the 1st order > derivative of the fitted or smoothed curve. This would be easy if > gnuplot had a way to access data not only from the _actual_ row but also > from the following one in order to calculate Delta y / Delta x = y(row > +1)-y(row)/(x(row+1)-x(row)). This was not possible earlier. Has the > situation changed since then? If not, is it still possible to perform > differentiation from within gnuplot (even if this means using some awk > script or similar), as I would like to automate the process and gnuplot > is easily driven from a shell script or batch file. Well, AFAICS it has become possible in 4.3, though not very easily accessible. There is a new assignment operator (and a comma operator), so you can plot something like: old_x = NaN plot 'test.dat' using 1:(dx=$1-old_x,old_x=$1,dx) This only works with data file plotting. You can use the '+' pseudo file for fitted data (though I would recommend the analytic derivative in this case). For smoothed curves, plot the results into a table file (help set table) and use this file as given above. -- Completely untested -- Juergen |
|
From: <pl...@pi...> - 2007-10-22 14:26:23
|
Delta y / Delta x =3D y(row+1)-y(row)/(x(row+1)-x(row)).
sorry I did reply to this one but it went to Levente only not the list.
this sort of trivial task lends itself nicely to the new assign feature =
=
(only in CVS AFAIK) and is exactly the sort of thing I had in mind when =
I =
requested it.
check out the syntax since it is still being developed and may change.
It does not enable you to look ahead or backwards in the data but settin=
g =
a variables like last_x last_y and diffing to the previous point rather =
=
than the next will do this nicely.
This can be done inline in the plot command or via a function call in th=
e =
plot command , that keeps the plot line less cluttered.
Here's two examples: sample max x and y values in plot range and =
trapezoidal integration area under graph. Again check syntax since I thi=
nk =
it has changed since I built this CVS snapshot.
xmax=3D0;ymax=3D0;
max(x,y)=3Dassign("ymax",(ymax<y)?y+0. * assign("xmax",x):ymax );
started=3D0;
add_aug(x,y)=3Dassign("area",(started>0)?\
area + (x-prev_x)*(y+prev_y)/2.0 =
=
+ 0.0*(assign("prev_x",x) \
+assign("prev_y",y) ) \
: 0.0*( assign("started",1) =
+ assign("prev_x",x) + assign("prev_y",y) ) \
) ;
this using clause is in my plot command:
using 1:(add_aug( ($1),($2) ))
HTH, Peter.
|
|
From: <ln...@dr...> - 2007-10-23 16:05:35
|
On Mon, 2007-10-22 at 13:51 +0200, Juergen Wieferink wrote: > Well, AFAICS it has become possible in 4.3, though not very easily > accessible. > > There is a new assignment operator (and a comma operator), so you > can plot something like: > > old_x =3D NaN > plot 'test.dat' using 1:(dx=3D$1-old_x,old_x=3D$1,dx) > > This only works with data file plotting. You can use the '+' pseudo > file for fitted data (though I would recommend the analytic > derivative in this case). For smoothed curves, plot the results > into a table file (help set table) and use this file as given > above. > > -- Completely untested -- > > Juergen On Mon, 2007-10-22 at 16:26 +0200, pl...@pi... wrote: > Delta y / Delta x =3D y(row+1)-y(row)/(x(row+1)-x(row)). > > sorry I did reply to this one but it went to Levente only not the list. > > this sort of trivial task lends itself nicely to the new assign feature > (only in CVS AFAIK) and is exactly the sort of thing I had in mind when= I > requested it. > > check out the syntax since it is still being developed and may change. > Thanks J=FCrgen and Peter! I will try the assignment operator, it sounds promising. This was exactly my impression, that something new was introduced to gnuplot, allowing for (sort of) simple differentiation, only I did not remember exactly which command. Concerning the spline differentiating question: I am aware that splines can cause unwanted oscillations, but in the past I was succesfully using acsplines with weights carefully chosen to approximate the curve I would draw by hand through the experimental data. Sometimes it is faster this way than trying to find a good function and fit it with the Marquardt-Levenberg method (I often don't know exactly which function describes the experimental behaviour -- e.g. in complex biological systems). Levente |
|
From: <pl...@pi...> - 2007-10-23 19:11:44
|
On Tue, 23 Oct 2007 18:05:29 +0200, <ln...@dr...> wrote: > Concerning the spline differentiating question: I am aware that splines > can cause unwanted oscillations, Yes one of the problems with splines is that the contraints placed on continuity and the fact that it must pass through all data points can lead to some fairly unexpected and sometimes extreme excursions from the zone where the real data lie. Sometimes it's just clearly the wrong way to respresent certain data. Sometimes it's less obvious but still wrong, this is more dangerous. Since the constraints on the continuity of the differencial are *completely artificially* imposed as being the very definition of the spline fit , I believe it is utterly wrong to place any scientific meaning on the derivative of a spline fit. I would invite you to reflect on what that means for your results and maybe have a quick look at the maths behind spline fitting to check what I say. The maths is pretty simple actually but it's important to know what you are doing in applying such a fit. Many people apply even least squares fits to completely inappropriate data because they do not realise the basic assumptions of the maths behind it, that is that y errors are >> x errors. If that's not true you get the wrong fit! > but in the past I was succesfully using > acsplines with weights carefully chosen to approximate the curve I would > draw by hand through the experimental data. Well I certainly hope you're not using that sort of bad science for anything more than personal curiosity. It does not sound like valid , reproducable scientific analysis. :) gnuplot claims not to do data processing (other than fitting user defined functions) although "smoothing" data with splines clearly is just that. I think you should be looking at applying some recognised data processing techniques to your data rather than hand tweeking splines. I hope my comments have highlighted some of the traps , in particular with regards to the derivative that you are interested in. regards, Peter. |
|
From: <ln...@dr...> - 2007-10-23 20:01:44
|
> On Tue, 23 Oct 2007 18:05:29 +0200, <ln...@dr...> wrote: > >> Concerning the spline differentiating question: I am aware that spline= s >> can cause unwanted oscillations, > > Yes one of the problems with splines is that the contraints placed on > continuity and the fact that it must pass through all data points can l= ead > to some fairly unexpected and sometimes extreme excursions from the zon= e > where the real data lie. > I beg to disagree: you certainly are not forced to go through all your da= ta points for a smoothed spline. Of course, csplines do this way, but acspli= nes not (and bezier curves also do not necessarily pass through all the points). > Sometimes it's just clearly the wrong way to respresent certain data. > Sometimes it's less obvious but still wrong, this is more dangerous. Si= nce > the constraints on the continuity of the differencial are *completely > artificially* imposed as being the very definition of the spline fit , = I > believe it is utterly wrong to place any scientific meaning on the > derivative of a spline fit. > Did you try acsplines (it seems me what you say applies mostly for csplin= es)? I wouldn't say that the differential of such an acspline (if it is carefull= y drawn) has no scientific meaning. After all, you certainly remember the t= ime there were no personal computers and everybody was forced to draw the cur= ve himself for a dataset on a piece of paper then graphically differentiate = this curve by placing tangents and calculating their slopes. This is exactly t= he same thing I would like to do, but now aided by the computer. (Maybe smoothing= is not the exact term for this, that's why we seem to disagree?) Every measureme= nt, fitting, analysing method has its own inaccuracies and problems in the re= al world you must be aware of. This applies either to curve fitting or smoot= hing. It is only in mathematics where things are clearly defined and rigorous. > I would invite you to reflect on what that means for your results and > maybe have a quick look at the maths behind spline fitting to check wha= t I > say. The maths is pretty simple actually but it's important to know wha= t > you are doing in applying such a fit. > > Many people apply even least squares fits to completely inappropriate d= ata > because they do not realise the basic assumptions of the maths behind i= t, > that is that y errors are >> x errors. If that's not true you get the > wrong fit! > >> but in the past I was succesfully using >> acsplines with weights carefully chosen to approximate the curve I wou= ld >> draw by hand through the experimental data. > > Well I certainly hope you're not using that sort of bad science for > anything more than personal curiosity. > It does not sound like valid , reproducable scientific analysis. :) > I used this method to calculate meaningful results. At least not less mea= ningful that I would have obtained by graphical differentiation on a piece of pap= er. See: you have a bunch of data points and you simply don't know which is t= he mathematical function you should fit them to. You sometimes even don't ex= actly know what the exact error on x or y values is, but have only a clue. Don'= t misunderstand me: I don't need to have something very precise, only at gi= ven extent (certainly more precise than doing a "differentiation" on the poin= ts themselves, which have a certain scatter due to imprecisions and perturba= tions). By the way, even if I fit a mathematically well-defined function to these= data with the Marquardt-Levenberg method, I am not sure that this is the funct= ion which describes most accurately the phenomenon solely because the curve h= as a similar shape than the points plotted. > gnuplot claims not to do data processing (other than fitting user defin= ed > functions) although "smoothing" data with splines clearly is just that. > > I think you should be looking at applying some recognised data processi= ng > techniques to your data rather than hand tweeking splines. > > I hope my comments have highlighted some of the traps , in particular w= ith > regards to the derivative that you are interested in. > > regards, Peter. > My biggest problem for the moment is the following: I did not find anythi= ng useful about 'assignment' in the help file (I have the most recent CVS ve= rsion of gnuplot, of course). How should I use it then? Is there a configure-ti= me switch needed to compile it into gnuplot? Levente |
|
From: <pl...@pi...> - 2007-10-25 00:28:18
|
On Tue, 23 Oct 2007 22:01:33 +0200, <ln...@dr...> wrote: >> >> Yes one of the problems with splines is that the contraints placed on >> continuity and the fact that it must pass through all data points can >> lead >> to some fairly unexpected and sometimes extreme excursions from the zone >> where the real data lie. >> > I beg to disagree: you certainly are not forced to go through all your > data > points for a smoothed spline. Of course, csplines do this way, but > acsplines not > (and bezier curves also do not necessarily pass through all the points). You are correct to an extent but acsplines are still basically applying a cspline and hence the artificial constraints that are the basis of that technique. Massaging the wieghts to get it to "look better" is adding error to error it does not correct the fundemental problem. For example csplines have problems following rapid changes, ie they dont follow and then they tend to overshoot badly. If you use acsplines and play with the wieghts around the change you may play down the overshoot by reducing the surrounding wieghts. But in doing this you mask the change that is real. You are further corrupting the data and masking real effects, not improving the fit. The ability to add a column of weights is valuable but these values should have some experimental justification not just be used as arbitary fiddle factors. I dont think this approach addresses my critisim of csplines. It may be that you are trying to clobber some fairly high frequency experimental noise on an effect that is fairly smooth in nature. In this case, as you point out, fiddling coeffs is a bit like a hand drawn line and taking a tangent with a ruler and pencil. If that fits your needs you're all set, it's good that you were able to find a solution for the derivatives. I think this case in point highlights a lack in gnuplot's offering in the "smoothing" department. I've touched on this before. csplines are pretty heavy handed when it comes to rounding things off and produce some severe artifacts unless the underlying nature of the data is itself long smooth curves. I know Hans-Bernhard is going to cry foul and say that gnuplot should not be doing data processing. The fact is it does. The trouble is cspines are a bit special and the current offering of csplines or bezier is a bit one-sided. It would be nice to have an alternative that suits other sorts of data. Some sort of mean or wieghted mean for example. regards, Peter. |
|
From: Ethan M. <merritt@u.washington.edu> - 2007-10-23 20:13:37
|
On Tuesday 23 October 2007 13:01, ln...@dr... wrote: > > My biggest problem for the moment is the following: I did not find anything > useful about 'assignment' in the help file (I have the most recent CVS version > of gnuplot, of course). How should I use it then? Is there a configure-time > switch needed to compile it into gnuplot? No, but it is a very recent feature. Peter <pl...@pi...> referred to it as an "assignment feature" because the original trial implementation was in the form of a separate function named assign(). In the end, the same functionality was incorporated by adding support for two standard operators: = and , Both act as they do in the C language syntax. '=' is an assignment operator, taking a variable name on the left and an expression on the right. ',' is a serial evaluation operator separating independently evaluatied expressions. Combining these operators allows functions or expressions to have side effects. In particular, a function or expression evaluation can have the side effect of storing its result in a named variable. An example of using these to track successive data points is provided in the new demo file data_feedback.dem http://gnuplot.sourceforge.net/demo_4.3/data_feedback.html -- Ethan A Merritt Courier Deliveries: 1959 NE Pacific Dept of Biochemistry Health Sciences Building University of Washington - Seattle WA 98195-7742 |
|
From: <ln...@dr...> - 2007-10-23 20:33:15
|
> On Tuesday 23 October 2007 13:01, ln...@dr... wrote: >> >> My biggest problem for the moment is the following: I did not find any= thing >> useful about 'assignment' in the help file (I have the most recent CVS= version >> of gnuplot, of course). How should I use it then? Is there a configure= -time >> switch needed to compile it into gnuplot? > > No, but it is a very recent feature. > > Peter <pl...@pi...> referred to it as an "assignment feature" b= ecause > the original trial implementation was in the form of a separate functio= n > named assign(). In the end, the same functionality was incorporated by > adding support for two standard operators: =3D and , > Both act as they do in the C language syntax. > > '=3D' is an assignment operator, taking a variable name on the left and > an expression on the right. > > ',' is a serial evaluation operator separating independently evaluatied > expressions. > > Combining these operators allows functions or expressions to have > side effects. In particular, a function or expression evaluation can > have the side effect of storing its result in a named variable. > > An example of using these to track successive data points is provided > in the new demo file data_feedback.dem > http://gnuplot.sourceforge.net/demo_4.3/data_feedback.html > Thanks, Ethan! Now I suppose I have to read a concise book on "C" to familiarise myself = with this syntax :) Levente |
|
From: Levente <ln...@dr...> - 2007-10-24 11:44:52
|
On Mon, 2007-10-22 at 13:51 +0200, Juergen Wieferink wrote: > There is a new assignment operator (and a comma operator), so you > can plot something like: >=20 > old_x =3D NaN > plot 'test.dat' using 1:(dx=3D$1-old_x,old_x=3D$1,dx) >=20 First of all, thanks to everybody who helped me! I succeeded in approximating the differential of data files the following way, using J=C3=BCrgen's example: # Approximation of the first derivative of x-y data # by the method: (y(n)-y(n-1))/(x(n)-x(n-1)) old_x =3D NaN old_y =3D NaN plot 'something.dat' using 1:(dx=3D$1-old_x,dy=3D$2-old_y,old_x=3D$1,old_y= =3D $2,dy/dx) Just to let know to whom it may be useful. Maybe a small demo file could be written to demonstrate its use (I am pleased to post such a demo if this is OK). Thanks again, folks! Levente |
|
From: <pl...@pi...> - 2007-10-24 12:44:14
|
On Wed, 24 Oct 2007 13:43:22 +0200, Levente Novák <ln...@dr...> wrote: > On Mon, 2007-10-22 at 13:51 +0200, Juergen Wieferink wrote: > >> There is a new assignment operator (and a comma operator), so you >> can plot something like: >> >> old_x = NaN >> plot 'test.dat' using 1:(dx=$1-old_x,old_x=$1,dx) >> > > First of all, thanks to everybody who helped me! I succeeded in > approximating the differential of data files the following way, using > Jürgen's example: > > # Approximation of the first derivative of x-y data > # by the method: (y(n)-y(n-1))/(x(n)-x(n-1)) > > old_x = NaN > old_y = NaN > plot 'something.dat' using 1:(dx=$1-old_x,dy=$2-old_y,old_x=$1,old_y= > $2,dy/dx) > > Just to let know to whom it may be useful. Maybe a small demo file could > be written to demonstrate its use (I am pleased to post such a demo if > this is OK). > > Thanks again, folks! > > Levente > Glad you found this feature useful. It's quite new so it needs some different uses to see if does all that's needed, thanks. if you want to post that as an example it should probably by plotting against ($1+old_x)/2 to prevent a shift in the position of the derivative. BTW the running mean example also suffers from this defect as is clearly seen in the plot that Ethan linked to. The running mean is offset 2.5 data points to the right. regards. |
|
From: Levente <ln...@dr...> - 2007-10-24 13:00:56
|
On Wed, 2007-10-24 at 14:45 +0200, pl...@pi... wrote: > On Wed, 24 Oct 2007 13:43:22 +0200, Levente Nov=C3=A1k =20 > <ln...@dr...> wrote: > > First of all, thanks to everybody who helped me! I succeeded in > > approximating the differential of data files the following way, using > > J=C3=BCrgen's example: > > > > # Approximation of the first derivative of x-y data > > # by the method: (y(n)-y(n-1))/(x(n)-x(n-1)) > > > > old_x =3D NaN > > old_y =3D NaN > > plot 'something.dat' using 1:(dx=3D$1-old_x,dy=3D$2-old_y,old_x=3D$1,ol= d_y=3D > > $2,dy/dx) > > > > Just to let know to whom it may be useful. Maybe a small demo file coul= d > > be written to demonstrate its use (I am pleased to post such a demo if > > this is OK). > > > Glad you found this feature useful. >=20 > It's quite new so it needs some different uses to see if does all that's = =20 > needed, thanks. >=20 > if you want to post that as an example it should probably by plotting =20 > against ($1+old_x)/2 to prevent a shift in the position of the derivative= . >=20 You are absolutely right, the x values need to be corrected (only I had so many datapoints that this shift was very small in my initial example). I will adapt the formula accordingly. Levente |
|
From: Ethan M. <merritt@u.washington.edu> - 2007-10-24 18:16:31
|
On Wednesday 24 October 2007 05:45, pl...@pi... wrote: > > if you want to post that as an example it should probably by plotting > against ($1+old_x)/2 to prevent a shift in the position of the derivative. > > BTW the running mean example also suffers from this defect as is clearly > seen in the plot that Ethan linked to. The running mean is offset 2.5 data > points to the right. The running mean example is correct for the typical use case. Have a look in your local paper for the stock price reports, or the weather summaries, etc. In calculating a running average you typically do not known the value of future points, only of the points measured to date. So you necessarily average over the previous 5 points, as stated in the plot title. This is not the same as for local approximation of the derivative. -- Ethan A Merritt |
|
From: <pl...@pi...> - 2007-10-25 00:44:57
|
On Wed, 24 Oct 2007 19:20:12 +0200, Ethan Merritt <merritt@u.washington.edu> wrote: > On Wednesday 24 October 2007 05:45, pl...@pi... wrote: >> >> if you want to post that as an example it should probably by plotting >> against ($1+old_x)/2 to prevent a shift in the position of the >> derivative. >> >> BTW the running mean example also suffers from this defect as is clearly >> seen in the plot that Ethan linked to. The running mean is offset 2.5 >> data >> points to the right. > > The running mean example is correct for the typical use case. > Have a look in your local paper for the stock price reports, > or the weather summaries, etc. > > In calculating a running average you typically do not known the value of > future points, only of the points measured to date. > So you necessarily average over the previous 5 points, as stated in the > plot title. > > This is not the same as for local approximation of the derivative. > Yes the title does clearly state what is shown. I did not say it was incorrect, I said it was a defect. Hand-wavy discussions about some "typical usage" are not much help in science and I certainly would not look to my local paper for an reference on the correct way to process data. I posted that comment since it is a common pitfall in calculating a running mean which also applied to the derviative code snip. Please dont take it the wrong way. ;) |
