gnuplot-beta — Developers and beta testers of new gnuplot code

Problem (bug?) with "smooth frequency" and logscale in 4.0 and 4.1

[Kostas O. <ko...@re...>]

(I posted this to gnuplot-bugs, but I see a lot of spam there, so I am 
re-posting here.)

Hello,

I have discovered a problem, perhaps a bug, with the following:

set xrange [0:10]
set logscale y
plot 'j' using (floor($1)):($2) smooth frequency with points

where the 'j' file is:

1    0.09
2    0.01
2.1    0.01
2.3    0.01
3    0.02
6    0.01
6.5    0.01
8    0.001
9    0.001
9.1    0.002

The result I get is very wrong: e.g., at x=2, I get y=1e-6.  And at x=9, 
I get 2e-6.
When logscale y is disabled, the results look correct.  Am I doing 
something wrong?  Or is this a bug?

*I do not subscribe to the list, so please CC me in your reply.*

             Kostas

wxWidgets terminal again

[<tim...@en...>]

Dear gnuplot enthusiasts,

I have updated the wxWidgets terminal hosted in gnuplot's developpement=20
pages :

http://sourceforge.net/tracker/index.php?func=3Ddetail&aid=3D1267434&grou=
p_id=3D2055&atid=3D302055

As discussed previously on this list, I now use pango and cairo for the=20
drawing code. I have begun in this version to separate wxWidgets and=20
cairo/pango codes, so that they can be treated specifically. Indeed, as=20
cairo is designed to fully support ps, pdf, png, etc. in future=20
releases, it can be interesting for gnuplot file terminals.

Using pango helped me to handle text layouts properly. I am pretty happy=20
with alignment, even in enhanced text mode. I have implemented text=20
encoding and now you can also use 'set term wxt...' to set your font=20
name and size, as with most other terminals.

There's a caveat unfortunately : pango is based on unicode, and thus=20
won't use fonts that have a mapping different from unicode. It is the=20
case of the Symbol font, where for example the character code 0x41 gives=20
'alpha' whereas unicode mapping asks for a 'a'. For the Symbol font=20
especially, I have implemented a translation step from the Symbol=20
character to its unicode counterpart with the help of the following page =
:
http://www.unicode.org/Public/MAPPINGS/VENDORS/APPLE/SYMBOL.TXT
I hope this solution will be good enough.

This version compiles under Windows with makefile.mgw
Since the first announce of a working code under Windows, I have managed=20
to get mouse interactivity. Compared to the default Windows terminal,=20
the wxWidgets terminal provides enhanced text mode, multiple plot=20
windows, and antialiased lines.
Yau can get a binary here :

http://tipote.free.fr/wgnuplot-wxt-20051210.zip

Here are screenshots :

http://tipote.free.fr/wxt-windows3.png       (under Windows)
http://tipote.free.fr/wxt15.png       (under Linux)

[Today I have problems to reach this files. It must be a temporary=20
problem of free.fr]

Please note that the patch no longer modifies any of gnuplot files,=20
apart from term.h to add the terminal, readline.c to add mouse support=20
under Windows, configure.in and makefile.mgw to make it compile, so it=20
consider it safe regarding other gnuplot functionnalities.

I now look forward for your advice. Do you think this patch has some=20
future or should I forget it to come back to my physics' courses ? What=20
can I do to get this included in gnuplot CVS one day ? What do you think=20
of cairo as an alternative to libgd, libpdf and no-postscript-lib-at-all =
?

Best regards,

Timoth=E9e Lecomte

Re: plot with image question

[Hans-Bernhard B. <br...@ph...>]

Mark Savenije wrote:

> My question: If I plot an image now I get the pixel numbers along the X 
> and Y axis. Is it possible to supply values for the X and Y axis (thus 
> besides the image). 

I see no particular reason why the normal 'extended using specs' should 
not work here:

	plot 'foo' u ($1*scale_x):($2*scale_y):3 with image

> P.S. There is no such thing as a mubar functionality in gnuplot, is 
> there? 

There's 'set arrow', which should get the job done.

plot with image question

[Mark S. <ma...@sc...>]

Hi all!

Recently I discovered the 4.1 version of gnuplot and the image 
plotting facilities. Without knowing, I was waiting for this. A big 
thank you guys!

My question: If I plot an image now I get the pixel numbers along the 
X and Y axis. Is it possible to supply values for the X and Y axis 
(thus besides the image). Or is it possible the give an expression 
for the X and Y axis. Instead pixel numbers I want to give 
micrometers as the images are microscopic of nature. Can be done by 
another datafile with X and Y values, or by an expression, let say 
X*(MicroMeterPerPixelValue) for the X axis.

Your ideas are appreciated!

P.S. There is no such thing as a mubar functionality in gnuplot, is 
there? I mean a little line onto the image indicating a specified 
amount of distance. Of course I can program this directly into the 
image by making a line of pixels black (or white), but if there is 
some gnuplot functionality for that I sure like to use it.

Thanks, Mark

--
Mark Savenije (software/system engineer Centre for Advanced Microscopy)
Swammerdam Institute for Life Sciences (SILS)
Faculty of Science, University of Amsterdam, The Netherlands
Kruislaan 316, 1098 SM, Amsterdam, The Netherlands
Tel: ..+31 20 5257860, fax: ..+31 20 5256271
E-mail: ma...@sc..., WWW: http://wwwmc.bio.uva.nl

plot with image question

[Mark S. <ma...@sc...>]

Hi all!

Recently I discovered the 4.1 version of gnuplot and the image 
plotting facilities. Without knowing, I was waiting for this. A big 
thank you guys!

My question: If I plot an image now I get the pixel numbers along the 
X and Y axis. Is it possible to supply values for the X and Y axis 
(thus besides the image). Or is it possible the give an expression 
for the X and Y axis. Instead pixel numbers I want to give 
micrometers as the images are microscopic of nature. Can be done by 
another datafile with X and Y values, or by an expression, let say 
X*(MicroMeterPerPixelValue) for the X axis.

Your ideas are appreciated!

P.S. There is no such thing as a mubar functionality in gnuplot, is 
there? I mean a little line onto the image indicating a specified 
amount of distance. Of course I can program this directly into the 
image by making a line of pixels black (or white), but if there is 
some gnuplot functionality for that I sure like to use it.

Thanks, Mark

--
Mark Savenije (software/system engineer Centre for Advanced Microscopy)
Swammerdam Institute for Life Sciences (SILS)
Faculty of Science, University of Amsterdam, The Netherlands
Kruislaan 316, 1098 SM, Amsterdam, The Netherlands
Tel: ..+31 20 5257860, fax: ..+31 20 5256271
E-mail: ma...@sc..., WWW: http://wwwmc.bio.uva.nl

Re: gd problems

[<bma...@we...>]

Actually this is a known problem and has bothered me too for a while.=20
When I made gnuplot use bgd.dll on windows, my modifications to config.nt=
=20
somehow did not make it into cvs - sorry. :( Anyway, with /MD it is working=
=20
fine here and I'll submit my new patches for gd.trm and config.nt today.

Btw. what updates to config.nt do you need?

Bastian


BBands <bb...@ya...> schrieb am 08.12.05 01:13:09:
> windows 2000, fully updated
>  gnuplot, current cvs
>  gd, 2.0.33
>  msvc, 6.0
> =20
>  I am rewriting config.nt and makefile.nt to bring them up to date. I hav=
e succeeded with everything except gd. The current version uses makemsvcimp=
ort.bat to produce a bgd.lib which in turn uses bgd.dll. Everything compile=
s and links fine, but gp crashes on the first plot command after set term p=
ng. Boutelle says this has to do with a requirement to link gd with the mul=
tithreaded dll option /MD. However the compilation alreay uses /MT, a confl=
ict. IIRC, this was introduced to allow the pdf term to succeed. Everything=
 else works perfectly. Any thoughts, pointers, etc. so I can put this one t=
o bed?
> =20
>      jab
> =20
>=20
> John Bollinger, CFA, CMT
> www.BollingerBands.com
>=20
> If you advance far enough, you arrive at the beginning.
>=20
> =20
> -----------------------------------------------------------------
>  Yahoo! Personals
>  Let fate take it's course directly to your email.
>  See who's waiting for you Yahoo! Personals

--=A0
Bastian=A0M=E4rkisch
Physikalisches=A0Institut,=A0Universit=E4t=A0Heidelberg

gd problems II

[BBands <bb...@ya...>]

 From the gd FAQ. http://www.boutell.com/gd/faq.html Why does bgd.dll crash with my C program? You are trying to use gd's stdio library functions with a different C runtime library. Read on for notes on individual compilers.  Microsoft Visual C++: If you want to use gdImageCreateFromPng, gdImagePng, and other functions that take a FILE *, you MUST use Visual C++ 6.0 or earlier and you MUST link with the "multithreaded DLL" runtime library option. If you wish to use the .NET compiler, read the "Borland C++" section for an alternative method. 
 If you do not take the time to understand this, your program simply will not work! Note: the "multithreaded debug DLL" option is NOT the same thing and will NOT work. If you don't want to use the "multithreaded DLL" runtime library option (see "code generation" under "settings"), then read the "Borland C++" section for an alternative method. 



John Bollinger, CFA, CMT
www.BollingerBands.com

If you advance far enough, you arrive at the beginning.
			
---------------------------------
Yahoo! Shopping
 Find Great Deals on Holiday Gifts at Yahoo! Shopping 

gd problems

[BBands <bb...@ya...>]

windows 2000, fully updated
 gnuplot, current cvs
 gd, 2.0.33
 msvc, 6.0
 
 I am rewriting config.nt and makefile.nt to bring them up to date. I have succeeded with everything except gd. The current version uses makemsvcimport.bat to produce a bgd.lib which in turn uses bgd.dll. Everything compiles and links fine, but gp crashes on the first plot command after set term png. Boutelle says this has to do with a requirement to link gd with the multithreaded dll option /MD. However the compilation alreay uses /MT, a conflict. IIRC, this was introduced to allow the pdf term to succeed. Everything else works perfectly. Any thoughts, pointers, etc. so I can put this one to bed?
 
     jab
 


John Bollinger, CFA, CMT
www.BollingerBands.com

If you advance far enough, you arrive at the beginning.
		
---------------------------------
 Yahoo! Personals
 Let fate take it's course directly to your email.
 See who's waiting for you Yahoo! Personals

Re: trouble building gnuplot from CVS

[Lars H. <lhe...@us...>]

Michele Simionato writes:
> Yes, indeed the problem was with automake. I downloaded automake 1.9.6, but now
> I am getting conflicts with the old version of automake :-( I will investigate
> on Debian newsgroups.
 
 There shouldn't be any problems even with several instances in your PATH
 if only the newer version is first in PATH. I always switch to a different
 environment when working on gnuplot because the auto* tools installed on
 this OS are quite old.

Re: trouble building gnuplot from CVS

[Michele S. <mic...@gm...>]

Yes, indeed the problem was with automake. I downloaded automake 1.9.6, but now
I am getting conflicts with the old version of automake :-( I will investigate
on Debian newsgroups.

  Michele Simionato

Re: trouble building gnuplot from CVS

[Hans-Bernhard B. <br...@ph...>]

Michele Simionato wrote:
> I am using the last versions of automake and autoconf:

No, you're not, and that's exactly the problem.

> $ automake --version
> automake (GNU automake) 1.4-p6

automake 1.4 is far outdated.  They're at 1.9.5 or something by now. 
gnuplot needs at least version 1.7.

Re: trouble building gnuplot from CVS

[Lars H. <lhe...@us...>]

 
> I am using the last versions of automake and autoconf:
> 
> $ automake --version
> automake (GNU automake) 1.4-p6
 
 The lastest version of automake is 1.9.6. The 1.4-pX versions are quite
 problematic and should not be used at all.

trouble building gnuplot from CVS

[Michele S. <mic...@gm...>]

I just made a CVS checkout on my Debian box and I get the following:

$ ./prepare
make: `Makefile.am' is up to date.
make: `Makefile.am' is up to date.
make: `Makefile.am' is up to date.
make: `Makefile.am' is up to date.
make: `Makefile.am' is up to date.
automake: configure.in: required file `../config.guess' not found
automake: configure.in: required file `../config.sub' not found

Some part of the preparation process failed.
Please refer to INSTALL for details.

I am using the last versions of automake and autoconf:

$ automake --version
automake (GNU automake) 1.4-p6

$ autoconf --version
autoconf (GNU Autoconf) 2.59

Any idea of hot to fix this?
Thank you very much,

       Michele Simionato

Re: volunteering for gnuplot work?

[Thomas M. <mat...@ph...>]

On 2-Dec-05, at 4:58 AM, Hans-Bernhard Broeker wrote:

> Thomas Mattison wrote:
>> On 1-Dec-05, at 4:38 AM, Hans-Bernhard Broeker wrote:
>>> Thomas Mattison wrote:
>
>>>> The major annoyance is that the errors from fits are not done in 
>>>> what I consider to be the correct way.
> [...]
>
>>> I know.  I made it that way, at least partly on purpose.
>
>> For cases where the model being is fit is good, the data is good, and 
>> the errors are appropriate, there will still be fluctuations in the 
>> value of chisquare from data set to data set.
>
> Yes.  But those should be small.  Small enough that the effect on 
> parameter errors doesn't really matter.  If you're seriously worried 
> about, say, a 10-percent change to the parameter errors, no simple 
> fitting program will do the job anyway.   Odds are that more 
> fundamental violations (non-gaussian data errors, mostly) will do much 
> more damage to the fit's behaviour than that.

Many people do worry about getting the errors right to less than the 
fluctuations in chisquare per degree of freedom.  If I were refereeing 
a paper, I would send it back for revision if it used the definition of 
error that gnuplot does.

And you don't need a more complicated fitting program than gnuplot to 
get those errors.  Gnuplot already does the right calculation 
internally, it just doesn't report it.  It reports only a non-standard, 
fluctuation-sensitive error (although I will grant that it is possible 
in most cases to recover the standard error from the fit log by some 
simple hand calculations).


> Maybe we could use a policy like what the PDG has for global parameter 
> fits of particle properties: report the unmodified errors and the 
> chisq/ndf individually as the usual result, but if the chisq/ndf is 
> bigger than 1, scale the errors instead, and add a note to this 
> effect.

This sounds like what I have been proposing: report both the 
non-fluctuating standard error, and the rescaled error that gnuplot 
produces now.  But I would report both, instead of choosing for the 
user based on the chisquare value.

Users should look at chisquare/ndf (or better, we should calculate the 
chi square probability given the degrees of freedom for them inside 
gnuplot).  If that is reasonable, then use the standard non-fluctuating 
error reported.  If the chisquare/ndf is bad, the rescaled error is a 
reasonable rough estimate of the errors, but they should be suspicious

If they did not even use errors when they did the fit, the chisquare 
does not have a well-defined meaning, and only the rescaled error has 
much meaning.  In that case, I would be willing to consider reporting 
only the rescaled error.

But the program and documentation would be simpler if it treated fits 
with errors and fits without errors in the same way, and left it up to 
the user to interpret the results appropriately.


>
>>> I think it would make a lot more sense to add a couple lines to 
>>> 'fit.log' instead of creating what would be an almost complete copy 
>>> of all of its content.
>
>> The reason I would like a separate file is that the fit.log file is 
>> intended to be read by humans, and is not particularly readable by 
>> gnuplot.
>
> It's relatively easy to have the best of both worlds.  Just comment 
> out all the lines meant for human eyes only, and change the format of 
> the others a bit to accomodate gnuplot's syntax a bit.

That's pretty similar to my compromise suggestion: put a comment 
character in the first column of every line presently sent to fit.log, 
and add new lines containing gnuplot-readable parameters and errors.

But for the applications I have in mind, it would be best to put all of 
the parameters and errors from a given fit into a single 
gnuplot-readable line.  That line might get rather wide, and would be 
not very compatible with human eyes.  So I would not change the 
existing human-readable format except to add a comment character, and 
just add one non-human-readable line (perhaps with a comment line 
containing parameter names and error labels in horizontal format 
instead of column format).


> But I think a continuously appended fit log is of rather limited 
> usability for automated re-use --- you can't feed pieces of it back 
> into gnuplot any easier than you could do with individual files 
> created by the 'update' command.  You could only 'load' all of it, but 
> that won't do you any good, because all variables would be those of 
> the last fit.

What I want is not the ability to 'load' the fit.log, but to 'plot' or 
'fit' the parameters embedded in it from multiple similar fits to 
different data sets.  After many similar fits, fit.log (after ignoring 
the commented human-readable lines) would effectively have columns of 
fit parameters and errors.  You could plot parameter 1 vs fit number 
with its error, for instance, or plot parameter 3 vs parameter 2.  The 
user could also use a text-editor to add columns to each 
gnuplot-readable line with information about the conditions of the data 
(the temperature in my example).


> I think that kind of work would be easier with external tools that
> collect data from individual 'update'd files than by working from a 
> single long log.

My goal is to make such tools unnecessary, and have gnuplot do most or 
all of the work, if it can be done by modest changes to the code.


> That's exactly what 'update' is for --- it just doesn't output the 
> errors and such yet, but that should be easy to add.

Adding the errors to "update" output would be easy, making the input 
parser ignore them for " fit via 'file' " would be more challenging.

But the format of "update" and "via" files is human-readable with one 
parameter per line, and that is not the best for feeding back into 
gnuplot for plotting.



<note to spectators: now the subject changes to the possibility of 
doing fits with errors on the x-variable as well as the y variable.  
One issue is how to tell the fit command which column means what>


>> I also suggested having the fit parser respect "with" options to make 
>> the distinction.
>
> I don't think 'with' is a good name to use for that --- it has a 
> rather different job in plot and splot.  Changing the number of 
> allowed/required using specifiers, and the interpretation of the data 
> they yield, is really only a side effect of selecting a plot style, 
> and a somewhat confusing one at that.  Exporting only the confusing 
> aspects of 'with' over to fit, while keeping none of the 
> straightforward meanings that excuse them, feels like a bad user 
> interface design to me.

I'm not sure I follow your logic.  From the user point of view, "plot 
with" for errorbars seems straightforward.  err or yerr means the third 
column is y errors, xerr means the third column has x errors, xyerr 
means 3=xerr, 4=yerr.  All these could be accepted by the y=f(x) 
fitter.

It's true that splot doesn't presently know how to deal with z-errors.  
But if it ever does, I suspect that it will be triggered by "with zerr" 
and will expect 4 data columns, for x:y:z:zerr.

Plot also accepts asymmetric errors (4-column form for err, xerr, yerr 
and 6-column for xyerr) that would have to be rejected by the fitter.  
If the errorbars are really asymmetric, and that really matters to the 
fit user, (s)he needs professional help that (s)he's not going to get 
from gnuplot!

Letting "fit" accept "with err" would allow it to be more similar to 
"plot" which I think is good interface design.  [I would keep the 
existing syntax for y=f(x) fits functioning if the user leaves off 
"with err" for the sake of compatibility]

The joker in this proposal is that it is not compatible with the 
present syntax of the fit command, where the existence of a 4th data 
column forces a z=f(x,y) fit [the existence of a 3d column forces a 
y=f(x) fit with y errors, only 2 columns means y=f(x) fit with uniform 
y errors].

But that syntax is already a bit awkward, because the user doesn't 
always have z-errors available.  At present, the user is told to invent 
one by saying "fit f(x,y) 'file' using x:y:z:(1).  But we don't force 
users to explicitly invent an error column for y=f(x) fits without y 
errors.

So my proposal is to invent an "sfit" command that does z=f(x,y) fits. 
It would expect 3 columns for x:y:z, and accept 4 columns for 
x:y:z:zerr.  I would make its parser also accept (but probably not 
require) "with zerr" right from the start.

gnuplot already makes a distinction between 'plot' for y(x) and 'splot' 
for z(x,y).  Adding 'sfit' is consistent with this.  After all, to 
check a 'fit' of y=f(x), the user does a 'plot' with both the data and 
the fit function.  After doing a z=f(x,y) fit, he needs to do an 
'splot' with both the data and the fit function. So why not call the 
fit an 'sfit' ?


> The impact of the 3D fitting feature on the parser is quite minimal as 
> it is.  It just checks whether there are 4 columns of data or not.

I'm trying to think of things from the user perspective, as well as the 
maintainer perspective.  I'm not sure that the present interface to 
z=f(x,y) fits gets the balance right.   And I am volunteering to do 
proposed the parser changes.  The point is to get opinions on what is 
the right thing to do, which involves both users and maintainers.


> This would become much more complicated with all the suggested 
> extensions, and then it might be necessary to add a new command.  But 
> we usually try to avoid adding top-level commands as far as possible. 
> 'fit' and 'update' were about the only additions to that set in the 
> last decade.  So I would prefer to do it by some other means than 
> adding a command 'sfit'.

A user probably doesn't care if a system has a small number of 
top-level commands with many options or a large number of top-level 
commands with few options.  The user cares about how easy the system is 
to use.  If having more top-level commands allows the options to be 
simpler to understand and use, it should be considered.  Some such 
changes can make the maintenance easier also, by allowing more code to 
be shared across functionalities.


So here's the summary of what I am proposing to do for gnuplot fitting

1.  Change fit error reporting to include both standard and rescaled 
errors
     (rescaled error == present gnuplot error)

2.  Write gnuplot-readable parameter and error summary lines to file
     (an additional file and/or fit.log, perhaps with commenting-out of 
fit.log lines)

3.  Add ability to fit data with both x errors as well as y errors
     (distinguished by "with xerr" or "with xyerr" in fit command,
      without breaking present interface, including x:y:z:zerr syntax 
for z=f(x,y) fits)

4.  Add ability of fit command parser to interpret "with err" and "with 
yerr"
     (without breaking present interface)

5.  Add 'sfit' command for z=f(x,y) fits, accepting x:y:z, x:y:z:err,
     and also "x:y:z:err with zerr"




                                         Cheers

Prof. Thomas Mattison, Dept. of Physics & Astronomy, Univ. of British 
Columbia

Present Address: Stanford Linear Accelerator Center
2575 Sand Hill Road, Menlo Park, CA, 94025
Building 48 (Research Office Building), Mail Station MS35
Office:  ROB-231   Phone: 650-926-5342   Fax: 650-926-8522

Re: volunteering for gnuplot work?

[Hans-Bernhard B. <br...@ph...>]

Thomas Mattison wrote:
> On 1-Dec-05, at 4:38 AM, Hans-Bernhard Broeker wrote:
>> Thomas Mattison wrote:

>>> The major annoyance is that the errors from fits are not done in what 
>>> I consider to be the correct way.
[...]

>> I know.  I made it that way, at least partly on purpose.

> For cases where the model being is fit is good, the data is good, and 
> the errors are appropriate, there will still be fluctuations in the 
> value of chisquare from data set to data set.  

Yes.  But those should be small.  Small enough that the effect on 
parameter errors doesn't really matter.  If you're seriously worried 
about, say, a 10-percent change to the parameter errors, no simple 
fitting program will do the job anyway.   Odds are that more fundamental 
violations (non-gaussian data errors, mostly) will do much more damage 
to the fit's behaviour than that.

> I agree that if the chisq/ndf is far greater than 1, one should 
> interpret the canonical error with a grain of salt.  That is why I 
> advocate reporting both the canonical error and the rescaled error.

I rest unconvinced that that's a good idea.  Maybe we could use a policy 
like what the PDG has for global parameter fits of particle properties: 
report the unmodified errors and the chisq/ndf individually as the usual 
result, but if the chisq/ndf is bigger than 1, scale the errors instead, 
and add a note to this effect.

The basic question is almost a philosophical one: what do we want to 
fit: the data, or their errors?  chisq/ndf > 1 means that the model 
doesn't fit the data errors --- either the errorbars were 
unrealistically small, or the model too simplistic.   The unscaled 
errors you would get in such a case try to fit the errorbars, gnuplot's 
output fits the data.  It treats the errorbars as weights, and nothing else.

>> I think it would make a lot more sense to add a couple lines to 
>> 'fit.log' instead of creating what would be an almost complete copy of 
>> all of its content.

> The reason I would like a separate file is that the fit.log file is 
> intended to be read by humans, and is not particularly readable by gnuplot.

It's relatively easy to have the best of both worlds.  Just comment out 
all the lines meant for human eyes only, and change the format of the 
others a bit to accomodate gnuplot's syntax a bit.

But I think a continuously appended fit log is of rather limited 
usability for automated re-use --- you can't feed pieces of it back into 
gnuplot any easier than you could do with individual files created by 
the 'update' command.  You could only 'load' all of it, but that won't 
do you any good, because all variables would be those of the last fit.

> In my original post I gave a real-world example from teaching with 
> gnuplot where students do multiple similar fits to different data sets, 
> then plot the fit parameters (not the fit data!) with gnuplot.  They 
> even fit the fit parameters (with errors) using gnuplot.

I think that kind of work would be easier with external tools that
collect data from individual 'update'd files than by working from a 
single long log.

> If gnuplot fits put their results in gnuplot-readable form into a file, 
> it would be simple and reliable.

That's exactly what 'update' is for --- it just doesn't output the 
errors and such yet, but that should be easy to add.

> I also suggested having the fit parser respect "with" options to make 
> the distinction.

I don't think 'with' is a good name to use for that --- it has a rather 
different job in plot and splot.  Changing the number of 
allowed/required using specifiers, and the interpretation of the data 
they yield, is really only a side effect of selecting a plot style, and 
a somewhat confusing one at that.  Exporting only the confusing aspects 
of 'with' over to fit, while keeping none of the straightforward 
meanings that excuse them, feels like a bad user interface design to me.

> Another possibility would be to make a new command name like "sfit" and 
> use that for fits with both x and y as independent variables.  That 
> would probably simplify the fit command parser (at the expense of 
> creating a new, but hopefully simple, command to parse), and allow more 
> parallelism between plot/fit and splot/sfit.

The impact of the 3D fitting feature on the parser is quite minimal as 
it is.  It just checks whether there are 4 columns of data or not.

This would become much more complicated with all the suggested 
extensions, and then it might be necessary to add a new command.  But we 
usually try to avoid adding top-level commands as far as possible. 
'fit' and 'update' were about the only additions to that set in the last 
decade.  So I would prefer to do it by some other means than adding a 
command 'sfit'.

Re: volunteering for gnuplot work?

[Thomas M. <mat...@ph...>]

Hi

To the spectators of this discussion, it's in the context of me 
volunteering to do actual work, not complaining that someone else 
should!


On 1-Dec-05, at 4:38 AM, Hans-Bernhard Broeker wrote:

> Thomas Mattison wrote:
>
>> The major annoyance is that the errors from fits are not done in what 
>> I consider to be the correct way.  There exists a definition for the 
>> error of a fit that is independent of the goodness of the fit, 
>> although of course it depends on the function, the distribution of 
>> the data, and the error bars on the data.  Gnuplot fits do not report 
>> this result.
>
> I know.  I made it that way, at least partly on purpose.
>
> The basic problem is that once the chisq/ndf is far away from one, 
> discussing correctness of parameter errors is a waste of energy.  In 
> such a case whole fit is plain and simply wrong, so no parameter error 
> value really has a right of calling itself correct.
>
> chisq/ndf far away from one means that either the data errors are 
> unrealistic, or the model is wrong (over-/underfitting), or both.
> In this situation, it's a quite completely arbitrary choice whether to 
> believe the input errors, or the residuals.  gnuplot chooses to 
> believe the residuals.
>

For cases where the model being is fit is good, the data is good, and 
the errors are appropriate, there will still be fluctuations in the 
value of chisquare from data set to data set.  It is wrong to say that 
the "lucky" data sets really determine the parameters better than the 
"unlucky" data sets.  The parameter errors should be independent of the 
fluctuations in the chisquare.  Every other fitting program I have run 
across reports the canonical error, which is independent of the 
chisquare.

I agree that if the chisq/ndf is far greater than 1, one should 
interpret the canonical error with a grain of salt.  That is why I 
advocate reporting both the canonical error and the rescaled error.



>> If the function fits the data perfectly at every point, the error 
>> returned by gnuplot is zero.  This is clearly nonsense.
>
> No --- the fit itself is clearly nonsense.  I have to insist that 
> outputting nonsense as the result is fully justified in such a case.

I disagree.  If we fit a straight line to two data points, the fit will 
be perfect, with a chisquare of zero.  If the errors on the two data 
points are known, then it is simple to calculate the errors on the 
slope and intercept by standard error propagation.  The canonical error 
from a standard fitting program agrees exactly with the 
error-propagation result.  The errors are well-defined, meaningful, and 
finite.

But the gnuplot convention says that the errors on the slope and 
intercept are either zero because the chisquare is zero, or undefined 
(because in the case of two data points and two parameters, there are 
zero degrees of freedom).



>> I would add a feature: create a parameter-log file that would contain 
>> gnuplot-readable fit summary information: chisquare, parm-A, 
>> normal-errorA, rescaled-error-A, parm-B, regular-errorB, 
>> rescaled-error-B, ...  Each fit would append to the end of the file, 
>> similar to the present fit log file.  I would precede each line with 
>> a copy of the fit command that produced the fit (with a # in front so 
>> gnuplot would consider it to be a comment).  Probably I would also 
>> have a commented line giving the time of the fit.  It would also be 
>> possible to create a comment line containing headers to show which 
>> column means what for the fit, using the user's names for the fit 
>> variables.
>
> I think it would make a lot more sense to add a couple lines to 
> 'fit.log' instead of creating what would be an almost complete copy of 
> all of its content.

The reason I would like a separate file is that the fit.log file is 
intended to be read by humans, and is not particularly readable by 
gnuplot.

In my original post I gave a real-world example from teaching with 
gnuplot where students do multiple similar fits to different data sets, 
then plot the fit parameters (not the fit data!) with gnuplot.  They 
even fit the fit parameters (with errors) using gnuplot.

It's tedious and error-prone to read the fit.log files, and type the 
parameters and errors into another file, before gnuplot can read them.  
If gnuplot fits put their results in gnuplot-readable form into a file, 
it would be simple and reliable.

Adding lines to fit.log that would be gnuplot-readable would be a 
partial solution, but a human would still have to delete the 
non-readable lines.  Perhaps a compromise of putting a comment 
character at the start of all the other lines of fit.log would be the 
best solution.


>> There is a more major feature addition that would be nice.  
>> Frequently, data has errors not only on the y-variable, but also on 
>> the x-variable.  Gnuplot (nicely) handles plotting data with both x 
>> and y errors, but it doesn't know how to fit such data.
>
> Neither do Mrs. Marquard and Levenberg, or anybody else I've heard 
> about, for generic non-linear fitting.

The theory behind it is simple enough, and some programs (including 
mine) can do it, they are just not as widely known as the simpler case 
of errors only in y.


> Rigorously, fitting doesn't even know what x values are.  It's just a 
> typical use case simplification to treat the model as a function
>
> 	y[i] = f(x[i], parameters)
>
> Internally, the algorithm only assumes
>
> 	y[i] = f[i](parameters)

Yes, as I stated you need to go beyond y[i] = f[i](parameters).  It's 
not a simple extension of standard algorithms, in the way that z[i] = 
f(x[i], y[i], parameters) is a simple extension of y[i] = f(x[i], 
parameters).

But it can be done, and errors on the x variables are common in 
practice, so it should be done more often.


>
>> But the next two lines don't do analogous things
>> plot       'file' using 1:2:3:4 with xyerr
>> fit f(x,y) 'file' using 1:2:3:4             via a,b,c
>> The plot command uses columns 3 and 4 for x and y errors, and the fit 
>> command uses columns 3 and 4 for z and z errors.

>> One solution would be to make the interpretation of the columns in a 
>> fit command depend on the number of variables in the function.
>
> It won't work as easy as that --- parameters can also be passed as 
> parameters, i.e. you can
>
> 	fit f(x,a,b,c) 'file' u 1:2:3 via a,b,c
>
> instead of
>
> 	fit f(x) 'file' u 1:2:3 via a,b,c
>
> so the argument count of the function is strictly a red herring.

I hadn't thought of that.  So that proposal doesn't work.

I also suggested having the fit parser respect "with" options to make 
the distinction.

Another possibility would be to make a new command name like "sfit" and 
use that for fits with both x and y as independent variables.  That 
would probably simplify the fit command parser (at the expense of 
creating a new, but hopefully simple, command to parse), and allow more 
parallelism between plot/fit and splot/sfit.


                                         Cheers

Prof. Thomas Mattison, Dept. of Physics & Astronomy, Univ. of British 
Columbia

Present Address: Stanford Linear Accelerator Center
2575 Sand Hill Road, Menlo Park, CA, 94025
Building 48 (Research Office Building), Mail Station MS35
Office:  ROB-231   Phone: 650-926-5342   Fax: 650-926-8522

Re: volunteering for gnuplot work?

[Hans-Bernhard B. <br...@ph...>]

Thomas Mattison wrote:

> The major annoyance is that the errors from fits are not done in what I 
> consider to be the correct way.  There exists a definition for the error 
> of a fit that is independent of the goodness of the fit, although of 
> course it depends on the function, the distribution of the data, and the 
> error bars on the data.  Gnuplot fits do not report this result.  

I know.  I made it that way, at least partly on purpose.

The basic problem is that once the chisq/ndf is far away from one, 
discussing correctness of parameter errors is a waste of energy.  In 
such a case whole fit is plain and simply wrong, so no parameter error 
value really has a right of calling itself correct.

chisq/ndf far away from one means that either the data errors are 
unrealistic, or the model is wrong (over-/underfitting), or both.
In this situation, it's a quite completely arbitrary choice whether to 
believe the input errors, or the residuals.  gnuplot chooses to believe 
the residuals.

> If the function fits the data perfectly at every point, the error 
> returned by gnuplot is zero.  This is clearly nonsense.

No --- the fit itself is clearly nonsense.  I have to insist that 
outputting nonsense as the result is fully justified in such a case.

> My proposed solution would be to report both versions of the error: the 
> conventional error that is independent of the goodness of fit, and the 
> rescaled error that gnuplot presently reports.  I would also explicitly 
> report the factor by which the data errors have been effectively 
> rescaled.  

That factor already is reported. It's sqrt(WSSR/ndf).

I don't really think that there's much value in outputting two numbers
as seemingly independent results where in reality there's just a 
multiplication by common factor needed to go from one to the other.

> A feature of this fix is that it would change the format of the 
> parameter and error report from the fits.  I don't have as good an idea 
> as you folks do about whether users would care about format changes.  

Probably not as much now as they would have before version 4.0, when we
introduced 'set fit errorvariables' which lets users get at the 
parameter errors directly from inside gnuplot, without parsing the 
fit.log or screen output to extract them.

> I would add a feature: create a parameter-log file that would contain 
> gnuplot-readable fit summary information: chisquare, parm-A, 
> normal-errorA, rescaled-error-A, parm-B, regular-errorB, 
> rescaled-error-B, ...  Each fit would append to the end of the file, 
> similar to the present fit log file.  I would precede each line with a 
> copy of the fit command that produced the fit (with a # in front so 
> gnuplot would consider it to be a comment).  Probably I would also have 
> a commented line giving the time of the fit.  It would also be possible 
> to create a comment line containing headers to show which column means 
> what for the fit, using the user's names for the fit variables.

I think it would make a lot more sense to add a couple lines to 
'fit.log' instead of creating what would be an almost complete copy of 
all of its content.

> I also have some minor annoyances that I think are worth fixing.  I 
> would change the default FIT_START_LAMBDA to be 0.01 as recommended by 
> Numerical Recipes rather than the method used now  (I tell my students 
> to do this and it frequently helps).  

Caution there --- the actual algorithm is not the same as that in NR 
(although it used to be), so not all recommendations issued by that book 
may apply to gnuplot unmodified.  I took the computation for the initial 
lambda from a textbook on numerical maths (Schwarz "Numerische 
Mathematik", Teubner Verlag, in German).  I.e. the default startup for 
lambda is not any particular fixed number.  It's computed from the problem.

> I would make the default value for uninitialized variables in fits to
> be 0.01 rather than 1e-30 (the numerical derivatives algorithm tends
> to break with such a tiny default).

It's not 1e-30 right now --- it's 1.0

 > I would fix the numerical derivatives algorithm so it would
> not break if the initial value of a parameter is zero.

> There is a more major feature addition that would be nice.  Frequently, 
> data has errors not only on the y-variable, but also on the x-variable. 
>  Gnuplot (nicely) handles plotting data with both x and y errors, but it 
> doesn't know how to fit such data.  

Neither do Mrs. Marquard and Levenberg, or anybody else I've heard 
about, for generic non-linear fitting.

 > The rigorously correct way involves
> fitting for adjustments to all the x-variable values, as well as the 
> parameters.  

Rigorously, fitting doesn't even know what x values are.  It's just a 
typical use case simplification to treat the model as a function

	y[i] = f(x[i], parameters)

Internally, the algorithm only assumes

	y[i] = f[i](parameters)

> But the next two lines don't do analogous things
> 
> plot       'file' using 1:2:3:4 with xyerr
> fit f(x,y) 'file' using 1:2:3:4             via a,b,c
> 
> The plot command uses columns 3 and 4 for x and y errors, and the fit 
> command uses columns 3 and 4 for z and z errors.

That's because that fit has no useful relation with the 'plot' command
you're comparing it to. The type of command to compare it with would be

	splot 'file' using 1:2:3:4 with errorbars

(which unfortunately still doesn't exist).

> One solution would be to make the interpretation of the columns in a fit 
> command depend on the number of variables in the function.  

It won't work as easy as that --- parameters can also be passed as 
parameters, i.e. you can

	fit f(x,a,b,c) 'file' u 1:2:3 via a,b,c

instead of

	fit f(x) 'file' u 1:2:3 via a,b,c

so the argument count of the function is strictly a red herring.

[Fwd: volunteering for gnuplot work?]

[Lars H. <lhe...@us...>]

----- Forwarded message from Thomas Mattison <mat...@ph...> -----

To: lhe...@us..., br...@us...,
	cga...@us...
From: Thomas Mattison <mat...@ph...>
Subject: volunteering for gnuplot work?
Date: Wed, 30 Nov 2005 13:21:24 -0800

Hi

I'm a physics professor, and I use gnuplot for teaching and my own 
work. There are some annoyances about the fitting aspects of gnuplot 
that I would like to volunteer to fix.

The major annoyance is that the errors from fits are not done in what I 
consider to be the correct way.  There exists a definition for the 
error of a fit that is independent of the goodness of the fit, although 
of course it depends on the function, the distribution of the data, and 
the error bars on the data.  Gnuplot fits do not report this result.  
Internally, gnuplot does calculate this error, but it then rescales it 
by the goodness of the fit.

If the function fits the data perfectly at every point, the error 
returned by gnuplot is zero.  This is clearly nonsense.

Consider the simplest possible case of a single data point with a y 
value and error, and fitting a single-parameter function whose value is 
just the parameter.  The value of the fit parameter should be the y 
value, and the error on the fit parameter should be the error on the 
data point.  A gnuplot fit returns the right parameter value, but since 
the goodness of fit is perfect, gnuplot says the error on the fit 
parameter is zero.

The rescaling that gnuplot does to the fit errors is equivalent to 
calculating a single scale factor on the data point errors from the 
residuals of the fit, and using the rescaled data errors in the fit.

There is some value to this rescaling in many common circumstances.  
The standard formula for fit parameter errors requires that valid data 
errors be input to the fit.  If the user does not have them, the 
gnuplot algorithm is equivalent to computing a uniform value for the 
data errors from the residuals of the fit.  I don't have a better 
suggestion for what to do in such cases.

My proposed solution would be to report both versions of the error: the 
conventional error that is independent of the goodness of fit, and the 
rescaled error that gnuplot presently reports.  I would also explicitly 
report the factor by which the data errors have been effectively 
rescaled.  If the user does not include errors in the fit, I would make 
both answers be the present gnuplot answer.  In this case, the above 
error-rescaling factor is in fact the effective error that gnuplot has 
calculated for the data points.  If the data were plotted with this 
constant error value, then they should agree with the fitted function 
within this error about 2/3 of the time.

A feature of this fix is that it would change the format of the 
parameter and error report from the fits.  I don't have as good an idea 
as you folks do about whether users would care about format changes.  
Perhaps a backward-compatibility switch could be put in, so the 
old-format report could still be available.

Another annoyance is the lack of a nice way to record fit error values 
(it would also be nice to record parameter errors from multiple similar 
fits to different data sets).

I would add a feature: create a parameter-log file that would contain 
gnuplot-readable fit summary information: chisquare, parm-A, 
normal-errorA, rescaled-error-A, parm-B, regular-errorB, 
rescaled-error-B, ...  Each fit would append to the end of the file, 
similar to the present fit log file.  I would precede each line with a 
copy of the fit command that produced the fit (with a # in front so 
gnuplot would consider it to be a comment).  Probably I would also have 
a commented line giving the time of the fit.  It would also be possible 
to create a comment line containing headers to show which column means 
what for the fit, using the user's names for the fit variables.

An example of using this feature is an experiment students do in my 
lab.  They fit the current-voltage relationship for a diode, which is 
I(V) = I0 * (exp(V/V0) - 1).  They do this with the diode dunked in 
liquid nitrogen, dry ice and alcohol, ice water, and boiling water.  
The I0 and V0 parameters depend on temperature, and I want the students 
to plot and fit the parameters vs temperature.  The students could take 
this new parameter-log file, add a temperature column to the file with 
a text editor, then plot and fit from this file.

I also have some minor annoyances that I think are worth fixing.  I 
would change the default FIT_START_LAMBDA to be 0.01 as recommended by 
Numerical Recipes rather than the method used now  (I tell my students 
to do this and it frequently helps).  I would make the default value 
for uninitialized variables in fits to be 0.01 rather than 1e-30 (the 
numerical derivatives algorithm tends to break with such a tiny 
default).  I would fix the numerical derivatives algorithm so it would 
not break if the initial value of a parameter is zero.

There is a more major feature addition that would be nice.  Frequently, 
data has errors not only on the y-variable, but also on the x-variable. 
 Gnuplot (nicely) handles plotting data with both x and y errors, but 
it doesn't know how to fit such data.  The rigorously correct way 
involves fitting for adjustments to all the x-variable values, as well 
as the parameters.  The naive way to do this produces impractically big 
matrices when there is a lot of data, even if the fit function is 
simple (the computation time goes as data-points cubed instead of 
proportional to points).  There is a more efficient but still rigorous 
way to do it (goes as parameters squared times points), and an even 
more efficient way (just a few times longer than the conventional fit, 
with not much extra coding required).

Very few commonly available programs that do fits deal correctly with 
errors in x variables as well as y variables.  I would be willing to 
try adding this to gnuplot.

Note that there is a user-interface consistency issue.

The following lines do analogous things.

plot     'file' using 1:2:3 with err
fit f(x) 'file' using 1:2:3          via a,b,c

But the next two lines don't do analogous things

plot       'file' using 1:2:3:4 with xyerr
fit f(x,y) 'file' using 1:2:3:4             via a,b,c

The plot command uses columns 3 and 4 for x and y errors, and the fit 
command uses columns 3 and 4 for z and z errors.

One solution would be to make the interpretation of the columns in a 
fit command depend on the number of variables in the function.  In this 
case,

fit f(x  ) 'file' using 1:2:3:4             via a,b,c
fit f(x,y) 'file' using 1:2:3:4             via a,b,c

would do different things.  The first one would be y vs x with x and y 
errors,
the second would be z vs (x,y) with z errors.

Another solution would be to allow fit commands to "with" options like 
plot commands.  If the "fit with" options were absent, gnuplot would do 
what it does now.  If the "fit with" options were present, they would 
override the current default.

I have my own c-language fitting package for nonlinear fits using 
numerical derivatives that has all the capability of the gnuplot 
fitting internals.  It would probably be easier for me to add xy error 
fitting by replacing the fitting guts of gnuplot with my own code than 
to reverse-engineer what's there now well enough to add what would be 
required (yes I have looked at it).  I would be willing to do bug-fixes 
and maintenance to the fitting stuff afterwards.

So what do you think of the possibility of me getting involved in this?


                                        Cheers

Prof. Thomas Mattison, Dept. of Physics & Astronomy, Univ. of British 
Columbia

Present Address: Stanford Linear Accelerator Center
2575 Sand Hill Road, Menlo Park, CA, 94025
Building 48 (Research Office Building), Mail Station MS35
Office:  ROB-231   Phone: 650-926-5342   Fax: 650-926-8522


----- End forwarded message -----

Re: GNUPlot character encoding/locales

[John C. S. <sp...@ly...>]

On Tue, 2005-11-29 at 17:44 +0000, John C. Spray wrote:
> When passing scripts to GNUPlot, I find that it does not understand
> things like using 3,1415 instead of 3.1415, so I convert all such
> numeric expressions to the 'C' locale style.

Hmmm, I just realised that I'm still calling it GNUPlot, and caught some
flack for that last time I posted to this list...

*ducks*

John

Re: GNUPlot character encoding/locales

[Hans-Bernhard B. <br...@ph...>]

John C. Spray wrote:
> This leads me to a more general question: is there a standard for
> character encoding of strings in GNUPlot?  Does it depend at all on the
> selected locale, or is it always iso-8859-1 or UTF-8 or so?  Does it
> depend on the terminal used for output?

'help encoding' will tell you most there is to know about that, as far 
as the gnuplot core goes.  The rest is terminal-specific.

GNUPlot character encoding/locales

[John C. S. <sp...@ly...>]

Hello,

When passing scripts to GNUPlot, I find that it does not understand
things like using 3,1415 instead of 3.1415, so I convert all such
numeric expressions to the 'C' locale style.

This leads me to a more general question: is there a standard for
character encoding of strings in GNUPlot?  Does it depend at all on the
selected locale, or is it always iso-8859-1 or UTF-8 or so?  Does it
depend on the terminal used for output?

Cheers,
John

Re: Re: Question: intent of "set datafile binary"?

[Petr M. <mi...@ph...>]

>>> So far as I can tell, the command
>>>     set datafile binary
>>> does not accomplish anything at all.
>>
>> I don't think the documentation says it should accomplish anything.
>> The wording in 'help set datafile binary' could do with some
>> clarification, but I read it as saying: specifies defaults for what the
>> 'binary' keyword in {s}plot does.  It doesn't say anything about it
>> turning on 'binary'.
>
> That seems like how it was intended.  But Ethan's interpretation is 
> understandable.  One of those things where I didn't think of all the 
> possible nuances of the syntax beforehand.  Either it should have meaning 
> or produce an error message if it does nothing.

I've put to cvs a patch that gives an error message if users types just "set 
datafile binary".

---
PM

Re: gnuplot web extension

[V. <gae...@no...>]

On Tue, Nov 29, 2005 at 12:27:29PM +0100, Hans-Bernhard Broeker wrote:
> Yes, nobody seems to like that state of affairs very much --- but nobod=
y=20
> is doing anything about it, either.

    Sure, I know what you mean, and I can't blame anybody for that. I
was just expressing a mere dream.

--=20
    Ga=EBl

Re: gnuplot web extension

[Hans-Bernhard B. <br...@ph...>]

Ga=EBl Varoquaux wrote:
>     However it would be quite interesting to have gnuplot's core
> function available as a library. I am afraid this is not the current
> policy but it would indeed be interesting to be able to use gnuplot's
> powerfull plotting engine in other programme (think SciPy for instance)=
=2E

Many other programs already manage that just fine, Python scripts=20
included.  There are limitations to the piping concept, sure, but OTOH,
we're really in no shape to publish any library API other than=20
'do_line()' to the gnuplot core engine.  The command-line interface is=20
the only one that's anywhere near consistent and stable enough to be=20
useful right now, and that's distributed all over the program.

Yes, nobody seems to like that state of affairs very much --- but nobody =

is doing anything about it, either.

Re: gnuplot web extension

[V. <gae...@no...>]

On Tue, Nov 29, 2005 at 11:36:25AM +0100, Hans-Bernhard Broeker wrote:
> gnuplot is not a "web extension" (whatever exactly that may be supposed=
=20
> to mean).  It's an interactive program, which can also be used as a=20
> back-end by other programs if you redirect its input and/or output.  If=
=20
> that's not information for you, I'm afraid gnuplot isn't what you need.

    However it would be quite interesting to have gnuplot's core
function available as a library. I am afraid this is not the current
policy but it would indeed be interesting to be able to use gnuplot's
powerfull plotting engine in other programme (think SciPy for instance).

--=20
    Ga=EBl