gnuplot-beta — Developers and beta testers of new gnuplot code

Re: fileexist(f)

[Daniel J S. <dan...@ie...>]

Ethan A Merritt wrote:
> On Saturday 04 March 2006 12:45 pm, Petr Mikulik wrote:
> 
>>Ah, these in addition:
> 
> 
> Unfortunately, I don't understand the codepath for reading
> binary files well enough to back out of these failures cleanly.
> At the time the message is printed, we are several routines
> deep in the binary file code.  I think you would have to
> add error return codes to each layer of routines, and
> error-handling code at all the call sites.
> 
> Maybe Daniel would be willing to take a look at it
> after I sort out the normal [ascii] input file case.
> It was his code originally, I think. 
> 
> 
>>gnuplot> plot 'xxx' binary filetype=edf with image
>>          Can't open data file "xxx"
>>          util.c: No such file or directory
>>gnuplot> plot 'xxx' binary filetype=avs with image
>>          Can't open data file "xxx"
>>          util.c: No such file or directory
>>
>>gnuplot> plot 'demo.edfxxx' binary filetype=auto with image
>>                                                  ^
>>          Unsupported file type

I waited for a delayed gnuplot-beta-list email to appear on this, but nothing has arrived.  I don't fully follow the issue here.  Is there a problem recognizing some file in the present working directory?  The above first two complaints are indicating that.  The third complaint is that the file type is to be recognized automatically.  Gnuplot makes this decision based only upon the file extension.  So, 'edf' is in the table of "extension to function pointer" mappings, but 'edfxxx' isn't.  If we want to have something where gnuplot will also search into the file itself and look for keys about type, then we'd probably have to add another function pointer in the table containing a function that does the searching in addition to the one that handles parameter settings for the file.

Dan

Re: fileexist(f)

[Ethan A M. <merritt@u.washington.edu>]

On Saturday 04 March 2006 12:45 pm, Petr Mikulik wrote:
> Ah, these in addition:

Unfortunately, I don't understand the codepath for reading
binary files well enough to back out of these failures cleanly.
At the time the message is printed, we are several routines
deep in the binary file code.  I think you would have to
add error return codes to each layer of routines, and
error-handling code at all the call sites.

Maybe Daniel would be willing to take a look at it
after I sort out the normal [ascii] input file case.
It was his code originally, I think. 

> gnuplot> plot 'xxx' binary filetype=edf with image
>           Can't open data file "xxx"
>           util.c: No such file or directory
> gnuplot> plot 'xxx' binary filetype=avs with image
>           Can't open data file "xxx"
>           util.c: No such file or directory
> 
> gnuplot> plot 'demo.edfxxx' binary filetype=auto with image
>                                                   ^
>           Unsupported file type
> 
> 

-- 
Ethan A Merritt
Biomolecular Structure Center
University of Washington, Seattle 98195-7742

Re: fileexist(f)

[Ethan A M. <merritt@u.washington.edu>]

On Saturday 04 March 2006 02:13 pm, Petr Mikulik wrote:
> > Here's another.  The new function does not recognize pipes
> 
> The routine fileexist() is for files only; it makes no sense for pipes.
> For pipes, do
>  	if (fileexist('a.dat')) plot '<tr . , a.dat'

Sure. But that is only possible if you already know whether it is a
pipe or not.  What about the case where you want to put the
fileexist() test in a loadable routine that is passed the filename
string as a parameter?  I tried to show a plausible use for such
a thing in the previous mail.  At the time you are writing the
loadable routine, you don't know whether the caller will pass you
a file or a pipe at some future time.
 

-- 
Ethan A Merritt
Biomolecular Structure Center
University of Washington, Seattle 98195-7742

Re: fileexist(f)

[Petr M. <mi...@ph...>]

> Here's another.  The new function does not recognize pipes

The routine fileexist() is for files only; it makes no sense for pipes.
For pipes, do
 	if (fileexist('a.dat')) plot '<tr . , a.dat'

> gnuplot> !printenv GNUPLOT_LIB

It looks for the file in the current directory by default.

> was a clever idea to use GNUPLOT_LIB to point to any one of
> several data directories:

Then, there is a need for function (taking the name from Octave)
 	file_in_loadpath(name)

We can add it.

=> return the absolute path/name for the given file anywhere along the 
loadpath.

Then, you could use either of
 	if (fileexist(f))
 	if (fileexist(file_in_loadpath(f)))

> Anyhow, is there anysystem that provides "ls" that does not also
> provide "find"?

On OS/2 and Windows
   - there is no "ls" by default
   - there is "find", but with completely different syntax from GNU find

> if (defined(VAR))

Thanks, that I was looking for (in Octave & Matlab, it's called exist()).

---
PM

Re: fileexist(f)

[Petr M. <mi...@ph...>]

>>> gnuplot> plot 'a.dat', 'bla'
>>>                         ^
>>>           can't read data file "bla"
>>>           util.c: No such file or directory
>
> Here's a simple patch to fix this.
> Please try it and comment.

I like it, please commit it to cvs then ... with just some fix for:

> I think the 2D case is probably solid.  I have not tested all
> possible 3D plot modes, so it is possible that some additional
> checks need to be added there.

I have found only this non-working ('bla' does not exist):
   plot x, 'bla'
   splot x*y, 'bla'

this gives no plot and:

gnuplot> plot x, 'bla'
                  ^
          warning: Skipping unreadable file "bla"
                        ^
          x range is invalid


I would prefer just the function x is plotted.

---
PM

Re: fileexist(f)

[Ethan M. <merritt@u.washington.edu>]

On Friday 03 March 2006 10:29 am, Ethan Merritt wrote:
> On Friday 03 March 2006 10:05 am, Petr Mikulik wrote:
> >
> > gnuplot> plot 'a.dat', 'bla'
> >                         ^
> >           can't read data file "bla"
> >           util.c: No such file or directory

Here's a simple patch to fix this.
It is directly parallel to the recently added test that skips
files containing no data points.  

Please try it and comment.  

I already know that it emits an extraneous error message
for a couple of special cases (e.g. after 'set xdata time'),
and that it doesn't handle missing files passed to "fit".
I think the 2D case is probably solid.  I have not tested all
possible 3D plot modes, so it is possible that some additional
checks need to be added there.

	Ethan

--- gnuplot/src/datafile.c	2006-01-27 08:36:48.000000000 -0800
+++ gnuplot-cvs/src/datafile.c	2006-03-03 15:19:13.000000000 -0800
@@ -1275,7 +1275,9 @@
         if ((data_fp = loadpath_fopen(df_filename, df_binary ? "rb" : "r")) ==
 #endif
             (FILE *) NULL) {
-            os_error(name_token, "can't read data file \"%s\"", df_filename);
+            /* os_error(name_token, "can't read data file \"%s\"", df_filename); */
+	    int_warn(name_token, "Skipping unreadable file \"%s\"", df_filename);
+	    return -1;
         }
     }
 /*}}} */
--- gnuplot/src/plot2d.c	2005-11-28 11:02:56.000000000 -0800
+++ gnuplot-cvs/src/plot2d.c	2006-03-03 15:33:24.000000000 -0800
@@ -1787,6 +1787,11 @@
 		++line_num;
 	    }
 	    if (this_plot->plot_type == DATA) {
+		if (specs < 0) {
+		    /* Error check to handle missing or unreadable file */
+		    this_plot->plot_type = NODATA;
+		    goto SKIPPED_EMPTY_FILE;
+		}
 		/* actually get the data now */
 		if (get_data(this_plot) == 0) {
 		    /* EAM 2005 - warn, but keep going */
--- gnuplot/src/plot3d.c	2006-01-12 11:55:14.000000000 -0800
+++ gnuplot-cvs/src/plot3d.c	2006-03-03 15:32:54.000000000 -0800
@@ -1560,6 +1560,9 @@
 		struct surface_points *first_dataset = this_plot;
 		    /* pointer to the plot of the first dataset (surface) in the file */
 		int this_token = this_plot->token;
+		/* Error check to handle unreadable or missing data file */
+		if (specs < 0)
+		    df_eof = 1;
 		while (!df_eof) {
 		    this_plot = *tp_3d_ptr;
 		    assert(this_plot != NULL);


-- 
Ethan A Merritt
Biomolecular Structure Center
University of Washington, Seattle WA

Re: fileexist(f)

[Ethan M. <merritt@u.washington.edu>]

On Friday 03 March 2006 10:44 am, Petr Mikulik wrote:
> > Can we discuss this some more first?
> > Or at least let me try out the patch first?
> > I'm not 100% convinced it will always work, even on linux.
>
> It is enclosed, please try it. I think it must work everywhere where
> fopen() is implemented. (And gnuplot without fopen() cannot live.)

I think this function could use some more polishing to remove
rough edges. 

Here's one scenario which could cause much confusion.
The new function does not use the same path to look for
files that is used by the actual plot command:

gnuplot> !printenv GNUPLOT_LIB
../demo
gnuplot> print (fileexist("../demo/silver.dat")) ? "OK" : "File not 
found"
OK
gnuplot> print (fileexist("silver.dat")) ? "OK" : "File not found"
File not found
gnuplot> plot "silver.dat"
[plots normally even though fileexist didn't find it]


Here's another.  The new function does not recognize pipes

gnuplot> DATAFILE = "< head -40 ../demo/silver.dat"
gnuplot> plot DATAFILE
gnuplot> print (fileexist(DATAFILE)) ? "OK" : "File not found"
File not found


I agree that no one is likely to type such a sequence of commands
from the command line.  But I take it that the purpose of the new
command would be to place in an automated script, so that the
script can handle non-existence of a data file cleanly.  But then
it is much harder for the caller to understand why these example
do not work as expected.   For example, the user might think it
was a clever idea to use GNUPLOT_LIB to point to any one of
several data directories:

setenv GNUPLOT_LIB   /jan2006/data
gnuplot script.gp
setenv GNUPLOT_LIB   /feb2006/data
gnuplot script.gp

Where script.gp also tries to be clever, but in an incompatible way:
# Loop over data files and plot each one we find
if (!defined(plotno)) plotno = 1
file = sprintf("datafile.%d",plotno)
if (!fileexist(file)) exit
plot file
plotno = plotno+1
reread

The  pipe failure could also arise from a plausible attempt to
automate a sequence of plots that involve pre-processing the
data.

-- 
Ethan A Merritt
Biomolecular Structure Center
University of Washington, Seattle WA

Re: fileexist(f)

[Ethan M. <merritt@u.washington.edu>]

On Friday 03 March 2006 10:44 am, Petr Mikulik wrote:
> > It seems to me easier to use  system("find . -name <foo> -newer
> > <bar>")
>
> This is not portable either.

Not universally.  But it works without any changes to gnuplot.
Why add a new routine to gnuplot if it doesn't give any advantage
over what you can already do?

> Actually, the most trivial solution seems to be:
I think you meant:

   if (!defined(prev)) prev = ""      # first time only

   new=system("ls -l bla.dat")
   if (new ne prev) plot 'bla.dat'
   prev = new
   pause 5; reread


But this doesn't work very well.
`ls -l` will not tell whether new data has been written since your
previous plot.  For one thing, the time is only given to the nearest
minute. I thought you wanted a real-time test, that would update on
the order of seconds?  There is a gnu-specific extension:
 `ls --full-time`, but this won't help you on general unix boxes and
it certainly won't help on VMS or other less unix-like systems.

Anyhow, is there anysystem that provides "ls" that does not also
provide "find"?

> ??? how to detect that a variable has not been initialized?

if (defined(VAR))


-- 
Ethan A Merritt
Biomolecular Structure Center
University of Washington, Seattle WA

Re: fileexist(f)

[Daniel J S. <dan...@ie...>]

Petr Mikulik wrote:
>>> Example 1:
>>> plot 'measure.dat'
>>> if fileexist('fit.dat') replot 'fit.dat'

Not so sure about the syntax "fileexist".


>>> Actually, what some users ask for real-time plotting, is something
>>> like
>>>    pause untilfileexist "next.dat"
>>>    pause untilfilechanged "next.dat"

When a file is open by another program and is being written to, it appears in the directory.  So wouldn't one have to wait until the file is closed to be certain that reading it will give you all the data meant for plotting?

Dan

Re: fileexist(f)

[Petr M. <mi...@ph...>]

> Can we discuss this some more first?
> Or at least let me try out the patch first?
> I'm not 100% convinced it will always work, even on linux.

It is enclosed, please try it. I think it must work everywhere where fopen() 
is implemented. (And gnuplot without fopen() cannot live.)

>>>> Actually, what some users ask for real-time plotting, is something
>>>> like
>>>>    pause untilfileexist "next.dat"
>>>>    pause untilfilechanged "next.dat"
>
> It seems to me easier to use  system("find . -name <foo> -newer <bar>")

This is not portable either.

Actually, the most trivial solution seems to be:

prev=`ls -l bla.dat`   	or  prev=`dir bla.dat` 
if (!variable_exist(new)) new=prev  # 1st time here ??????????????
if (pi > 3.0) pi=0; new=prev        # 1st time here ??????????????
if (new == prev) pause 5; reread

plot 'bla.dat'
new = prev
reread


??? how to detect that a variable has not been initialized?

---
PM

Re: fileexist(f)

[Ethan M. <merritt@u.washington.edu>]

On Friday 03 March 2006 10:05 am, Petr Mikulik wrote:
> > A missing data file should no longer prevents the plot from being
> > constructed.
>
> No:
>
> gnuplot> plot 'a.dat', 'bla'
>                         ^
>           can't read data file "bla"
>           util.c: No such file or directory

OK. I'll try to fix that.
I guess it's only the "file exists but contains no data" 
condition that is now acceptable.


> So, I'll commit the "fileexist" patch which uses fopen().

Can we discuss this some more first?
Or at least let me try out the patch first?
I'm not 100% convinced it will always work, even on linux.


> >> Actually, what some users ask for real-time plotting, is something
> >> like
> >>    pause untilfileexist "next.dat"
> >>    pause untilfilechanged "next.dat"
> >
> > I doubt there is a universally portable solution to this one.
> > It is difficult even if you restrict the solution to linux/unix.
>
> Why? What about "select" and "stat" functions? 

I don't think these do what you want.  "select" will tell you if
there is more data available to read on an open file.  But you are
asking for a test that the file exists in the first place, which is an
entirely different question.

"stat" is essentially the same as "test", if you mean the command
line version.  If you mean the library call "stat",  I really have no
idea whether it is available outside of the usual linux/unix/POSIX
world.  And even if the routine itself is available, I don't know if
the interpretation of the subfields describing modification time
is universal.  You would have to do a lot of bookkeeping to 
implement a gnuplot routine that essential did
     if (file_has_changed_since_the_last_time_I_asked("filename"))
Could you track multiple file?  What if the file no longer exists?

It seems to me easier to use  system("find . -name <foo> -newer <bar>")
and let the system itself do all the bookkeeping.   The "find" command
is in POSIX, so you get exactly the same level of guarantee that you
would for building in a call to "stat", with the advantage that if your
system doesn't follow POSIX you could fix it in your script file rather
than having to edit and rebuild from source.

-- 
Ethan A Merritt
Biomolecular Structure Center
University of Washington, Seattle WA

Re: fileexist(f)

[Petr M. <mi...@ph...>]

>> Example 1:
>> plot 'measure.dat'
>> if fileexist('fit.dat') replot 'fit.dat'
>> pause 5
>> reread
>
> So far as I know, with the current cvs version you might
> just as well say
>  plot 'measure.dat','fit.dat'
>  pause 5
>  reread
>
> A missing data file should no longer prevents the plot from being
> constructed.

No:

gnuplot> plot 'a.dat', 'bla'
                        ^
          can't read data file "bla"
          util.c: No such file or directory


>> Example 2:
>
> I do not follow you there at all.

It was not a good example, sorry.


So, I'll commit the "fileexist" patch which uses fopen().


>> Actually, what some users ask for real-time plotting, is something
>> like
>>    pause untilfileexist "next.dat"
>>    pause untilfilechanged "next.dat"
>
> I doubt there is a universally portable solution to this one.
> It is difficult even if you restrict the solution to linux/unix.

Why? What about "select" and "stat" functions? If select() is not available, 
a poor-man solution is to check the status with one-second resolution.
(Well, I never used select() but I think it provides the waiting.)

---
PM

Re: fileexist(f)

[Ethan M. <merritt@u.washington.edu>]

On Friday 03 March 2006 02:25 am, Petr Mikulik wrote:
> > Can you give an example of why such a thing is needed?
>
> Example 1:
> plot 'measure.dat'
> if fileexist('fit.dat') replot 'fit.dat'
> pause 5
> reread

So far as I know, with the current cvs version you might
just as well say
  plot 'measure.dat','fit.dat'
  pause 5
  reread

A missing data file should no longer prevents the plot from being
constructed.



> Example 2:
> Execution of commands passed via pipes between gnuplot and octave
> (for example) takes some time and it's hard to stop/wait without
> breaking the script (error message). The above solution is fine for
> some cases.

I do not follow you there at all.
Neither your `test` nor your access() or fopen() will work on a pipe.

> Actually, what some users ask for real-time plotting, is something
> like 
>    pause untilfilechanged "next.dat" 

I doubt there is a universally portable solution to this one.
It is difficult even if you restrict the solution to linux/unix.
The best I can think of without writing an external helper that
monitors the file is something like:

   ready = words(system("find . -name 'next.dat' -newer 'lock'"))
   if (ready) plot "next.dat"
   if (ready) system("touch lock")
   pause 1
   reread



-- 
Ethan A Merritt
Biomolecular Structure Center
University of Washington, Seattle WA

Re: fileexist(f)

[Petr M. <mi...@ph...>]

>> What do you think about adding function
>>  	fileexist("a.dat")
>> that would return 1 or 0 if file exists or does not exist?
>>
>> I have already needed that several times ... solution via `test` is
>> not portable ...
>
> Is access() really that much more portable than test?
>
>> The code is as easy as
>>    return (access(filename, F_OK) != 0))
>> see the attachment.

> That's not terribly portable, then.  Access() is POSIX, but that doesn't 
> help us much on Windows, OS/2 and others.

Then OS/2 is OK, also GNU compilers under windows (mingw, cygwin). Cannot 
say about others ... #ifdef HAVE_UNISTD_H should suffice.

> If we want to be portable, the only real option is to ry to fopen() the 
> file and see if it works.

> man 2 access
> [...]
> RESTRICTIONS
>  access may not work correctly on NFS  file  systems  with  UID
>  mapping enabled,  because UID mapping is done on the server
>  and hidden from the client, which checks permissions.
>
> If your test doesn't work across NFS-mounted volumes, that's
> not very portable!

So you recommend to use fopen() even for unices, and thus get rid of 
access()? No problem, I will change the patch.

> Can you give an example of why such a thing is needed?

Example 1:
plot 'measure.dat'
if fileexist('fit.dat') replot 'fit.dat'
pause 5
reread

Example 2:
Execution of commands passed via pipes between gnuplot and octave (for 
example) takes some time and it's hard to stop/wait without breaking the 
script (error message). The above solution is fine for some cases.

Actually, what some users ask for real-time plotting, is something like
   pause untilfileexist "next.dat"
   pause untilfilechanged "next.dat"

---
PM

Re: fileexist(f)

[Ethan M. <merritt@u.washington.edu>]

On Thursday 02 March 2006 02:10 pm, Petr Mikulik wrote:
> What do you think about adding function
>  	fileexist("a.dat")
> that would return 1 or 0 if file exists or does not exist?
>
> I have already needed that several times ... solution via `test` is
> not portable ...

Is access() really that much more portable than test?

> The code is as easy as
>    return (access(filename, F_OK) != 0))
> see the attachment.

man 2 access
[...]
RESTRICTIONS
  access may not work correctly on NFS  file  systems  with  UID
  mapping enabled,  because UID mapping is done on the server
  and hidden from the client, which checks permissions.

If your test doesn't work across NFS-mounted volumes, that's
not very portable!

Can you give an example of why such a thing is needed?


-- 
Ethan A Merritt
Biomolecular Structure Center
University of Washington, Seattle WA

fileexist(f)

[Petr M. <mi...@ph...>]

What do you think about adding function
 	fileexist("a.dat")
that would return 1 or 0 if file exists or does not exist?
I think there is a good chance this name does not clash with most user 
variables.

I have already needed that several times ... solution via `test` is not 
portable ...

The code is as easy as
   return (access(filename, F_OK) != 0))
see the attachment.

---
PM

Re: How to check in changes?

[<br...@ph...>]

Lars Hecking wrote:

>  Or, if you made the changes in your checked out copy directly, use
>  "cvs diff -u". But then you need to exclude the CVS directories.

Not special work on that needed --- 'cvs diff' already excludes those.

>  The cleanest way is probably:
> 
>  - In the checked-out directory, perform a full build and then "make dist".
>  - Unpack the dist file somewhere else and rename it gnuplot-x.y.z.orig.
>  - Unpack the dist file again in the same directory (gnuplot-x.y.z). This
>    is where you make your edits.

If that's what you want, there's an easier way to get it: use "cvs 
export" instead of "cvs checkout".

>  The -u option creates a unified diff which is more readable than other diff
>  output formats, and -r recurses through the directory tree (not necessary
>  with cvs diff). This will also include every new file as a patch against a
>  zero-size file.

Not quite.  It will report all removed and added files as "Only in 
{old_dir}" or "Only in {new_dir}".  If you want newly created files 
spelled out in the diff, you need the ---unidirectional-new-file option 
of GNU diff (used to be -P), if you want deleted files listed, too, you 
need -N.

A tarball with a diff -ur, plus added files, and a README explaining 
what was done and why may be preferrable to a file-creating diff.

And BTW: it is possible to submit more than one file to an entry in the 
patch tracker --- you can add more files later, once the entry is made. 
  The limit is one file per [Submit] of the page, not one file per 
tracker entry.

Re: How to check in changes?

[Daniel J S. <dan...@ie...>]

Lars Hecking wrote:
>  
> 
>>OK.  Exactly what format of diff-ing should I use, and against what,
>>to create the patch?  A cvs diff against the head?  A diff against 4.0?
>>What options on the diff command?

I'd say work against the most recent version in CVS for your own benefit of having the latest of everything and won't have a major update when 4.2 comes out.  You'll invariably have hunks rejected at some point that require fixing, so might as well work against the most recent.

>> 
>>Also, there are some new test scripts and test data files to demonstrate
>>cases where the existing fits do the wrong thing, and that the changes make
>>it do the right thing.  How should that be handled?  It appears that the patch
>>submission page on sourceforge expects only one file per patch.  Should I
>>tar together the diff output and the new test scripts and data files?  Or make
>>multiple "patches"?
> 
>  
>  You have two gnuplot directory trees: the original, be it off cvs or
>  4.0, and your edited tree. Then you just run
> 
> $ ls
> gnuplot-x.y.z.orig gnuplot-x.y.z
> $ diff -ur gnuplot-x.y.z.orig gnuplot-x.y.z >foobar.diff
> $ gzip/compress/whatever foobar.diff
> 
>  and upload the gzipped diff file.
> 
>  Or, if you made the changes in your checked out copy directly, use
>  "cvs diff -u". But then you need to exclude the CVS directories.
> 
>  The cleanest way is probably:
> 
>  - In the checked-out directory, perform a full build and then "make dist".
>  - Unpack the dist file somewhere else and rename it gnuplot-x.y.z.orig.
>  - Unpack the dist file again in the same directory (gnuplot-x.y.z). This
>    is where you make your edits.

I do pretty much the above, but I get a copy from CVS and put it in, say, gnuplot-newfit.  Then in the gnuplot-newfit directory where CVS has placed "gnuplot" I

> cp -a gnuplot gnuplot-cvs
> cp -a gnuplot gnuplot-mod

The names are a preference thing, the basic idea is three copies: an untouched CVS version, a source tree modifications only version and a compiled version.  In gnuplot-mod is where changes go.  In "gnuplot" I will create symbolic links to the files in gnuplot-mod that have been changed.  Compile, edit and test everything in "gnuplot".

What this does is prevent extraneous and hidden files (from editors) from building up in the "gnuplot-mod" directory.  Otherwise bloated files appear in the diff sometimes without paying attention.

> diff -ur gnuplot-cvs gnuplot-mod > gnuplot-newfit-ddmmmyyyy.patch

e.g., gnuplot-newfit-2mar2006.patch.

Also, I'd suggest not leaving in old, commented out text or placing your initials next to changes you've made.  That's cruft that will only build up.  I don't think people generally do that.  (The patch file is often the place to go first for developers so they can see precisely what the changes are; no need for programmer tags.)  The time to leave your initials is when there may be some tricky part with a comment, or something that needs to be fixed in the future because of something else in gnuplot that needs to be addressed.  Leaving your initials will then help others find the person to ask when it comes time to fix the problem.

Oh, yeah, then when it comes to updating your patch against the latest CVS, repeat the "gnuplot/gnuplot-cvs/gnuplot-mod" thing and then from within "gnuplot-mod" do

> patch -p1 --dry-run < ../gnuplot-newfit-ddmmmyyyy.patch
> patch -p1 < ../gnuplot-newfit-ddmmmyyyy.patch

and go about fixing and discarding all the hunk reject files "<file>.rej".

Dan

Re: How to check in changes?

[Daniel J S. <dan...@ie...>]

Lars Hecking wrote:
>>How do I commit the changes?  I started by CVS downloading
> 
> 
>  Use the patch tracker on SF.

Yes, patch tracker...  I think the feeling right now is that we are working toward a 4.2 release and do not want to change much or add new features until after that.  Mostly maintenance for now.

There was a discussion about plot layout in 3D, for which I think there has been no conclusion about what to do.  I will take the silence on that to mean that perhaps people would really like to have it done better.  That may be too much for 4.2.

As an aside, I'd hope that after 4.2 there is a slightly more regular release schedule.  I think things have fundamentally changed, where pre 3.8/4.0 there were a lot of bugs that went unnoticed and unattended, to today where there are few bugs and any that are introduced are found and fixed by developers in a day or two.

Dan

Re: How to check in changes?

[Lars H. <lhe...@us...>]

 
> OK.  Exactly what format of diff-ing should I use, and against what,
> to create the patch?  A cvs diff against the head?  A diff against 4.0?
> What options on the diff command?
>  
> Also, there are some new test scripts and test data files to demonstrate
> cases where the existing fits do the wrong thing, and that the changes make
> it do the right thing.  How should that be handled?  It appears that the patch
> submission page on sourceforge expects only one file per patch.  Should I
> tar together the diff output and the new test scripts and data files?  Or make
> multiple "patches"?
 
 You have two gnuplot directory trees: the original, be it off cvs or
 4.0, and your edited tree. Then you just run

$ ls
gnuplot-x.y.z.orig gnuplot-x.y.z
$ diff -ur gnuplot-x.y.z.orig gnuplot-x.y.z >foobar.diff
$ gzip/compress/whatever foobar.diff

 and upload the gzipped diff file.

 Or, if you made the changes in your checked out copy directly, use
 "cvs diff -u". But then you need to exclude the CVS directories.

 The cleanest way is probably:

 - In the checked-out directory, perform a full build and then "make dist".
 - Unpack the dist file somewhere else and rename it gnuplot-x.y.z.orig.
 - Unpack the dist file again in the same directory (gnuplot-x.y.z). This
   is where you make your edits.

 The first step is not needed if you start from a distribution archive.

 The -u option creates a unified diff which is more readable than other diff
 output formats, and -r recurses through the directory tree (not necessary
 with cvs diff). This will also include every new file as a patch against a
 zero-size file.

Re: How to check in changes?

[Thomas M. <mat...@ph...>]

Lars Hecking <lhecking <at> users.sourceforge.net> writes:

> > How do I commit the changes? 
> 
>  Use the patch tracker on SF.

OK.  Exactly what format of diff-ing should I use, and against what,
to create the patch?  A cvs diff against the head?  A diff against 4.0?
What options on the diff command?

Also, there are some new test scripts and test data files to demonstrate
cases where the existing fits do the wrong thing, and that the changes make
it do the right thing.  How should that be handled?  It appears that the patch
submission page on sourceforge expects only one file per patch.  Should I
tar together the diff output and the new test scripts and data files?  Or make
multiple "patches"?

> 
>  You are not registered as a gnuplot developer on SF, so you have no
>  write access to CVS. 

Fair enough.

>  This status is awarded on a merit basis - the
>  quality of your contributions to gnuplot development e.g. in the form
>  of code and mailing list discussions. As you have only appeared on
>  our radar last December and posted 4 msgs here in total (including
>  today's), this might not happen just yet :)

I read the discussions, but since I'm generally happy with the way gnuplot
plots data, and don't claim that my opinions have more weight than others,
I haven't weighed in.  I'm more interested in improving the fitting part, and
there hasn't been discussion of that.

Re: How to check in changes?

[Lars H. <lhe...@us...>]

> How do I commit the changes?  I started by CVS downloading

 Use the patch tracker on SF.

> gnuplot 4.1, made my changes, and have done a CVS update since
> making my changes, so I should be in synch with everyone else.
>  
> But when I tried to CVS commit, I got a broken pipe abort.
> I'm working on a Linux system.

 You are not registered as a gnuplot developer on SF, so you have no
 write access to CVS. This status is awarded on a merit basis - the
 quality of your contributions to gnuplot development e.g. in the form
 of code and mailing list discussions. As you have only appeared on
 our radar last December and posted 4 msgs here in total (including
 today's), this might not happen just yet :)

Proposed changes to gnuplot fitting

[Thomas M. <mat...@ph...>]

Hi

I have made a bunch of changes to gnuplot fitting in fit.c.
Generally, they are flagged by comments containing "tsm feb06".
In many cases, old code is still there for reference but commented out.
This should probably be cleaned up before a release.
Some of the changes have user-variables to control them.

I have not yet figured out how to commit the changes,
so they aren't out there yet.



1.  Error message for undefined function evaluation
The old message in call_gnuplot() just said "Undefined value during 
function
evaluation."  I added the data point number, x, y, z, and parameter 
values.


2.  Check for error value equal to zero
If the user supplies errors but any of them are zero, there will be a 
divide by
zero, trashing the fit.  I added a check in fit_command(), which prints 
the data
point number, x, y, z, and error, then aborts back to command line.


3.  Made new message when fit stops because lambda is too large
Previously, the message in regress() said that the fit had converged in 
this case
which wasn't always true.  Other output unchanged.


4.  Zero-change in chisquare is now "BETTER" rather than "WORSE"
Previously, if the exact minimum of chisquare was found, so the 
chisquare change
was zero, this was called "WORSE" in marquardt().  The loop in 
regress() didn't exit,
and further iterations were done, which were also "WORSE" and increased 
lambda
until the upper limit on lambda or iterations was reached.  This was a 
waste of
time, and potentially confusing, for no benefit.


5.  Final parameter cleanup fixes
The gnuplot internal variables for the fit parameters may contain values
from an iteration that made the chisquare worse rather than better, if
the loop in regress() terminates due to maximum iterations or maximum 
lambda
rather than convergence.  There was code at the end of regress() that 
restores
the internal variable for only the last parameter, for a different 
reason
(the last variable is altered in call_gnuplot() to calculate 
derivatives,
but not restored there).  I changed the code at the end of regress() to 
restore
_all_ internal variables, from the array that contains the parameters 
that gave
the best chisquare so far.  I also put a copy of the last-parameter-fix 
code
into call_gnuplot(), where it logically belonged anyway.  Without this, 
if the
user interrupted the fit and tried to plot the current function on top 
of the data,
the last internal parameter was not correct.


6.  Changed convergence criterion, with new user-variable FIT_LIMIT_ABS.
The new simpler criterion in regress() is absolute reduction in 
chisquare
for an iteration of less than epsilon*chisquare plus epsilonAbs.  
Default
for new internal variable epsilonAbs is zero, user-settable by 
FIT_LIMIT_ABS.
Default for the existing internal variable epsilon is 1.0e-4, 
user-settable
by FIT_LIMIT (unchanged).  In most cases, the old convergence criterion 
was
_relative_ change in chisquare of less than epsilon.  But if the 
chisquare
was less than NEARLY_ZERO = 1.0e-30, (which could happen if fitting data
with magnitude less than 1.0e-15 with no errors), the old criterion was
_absolute_ change in chisquare less than epsilon.  Any change in 
chisquare
would probably be less than epsilon in these cases, so the fitter would
announce convergence immediately rather than finding the minimum.  Users
would probably prefer the default relative convergence criterion in 
these cases,
but there was no way for them to impose it.  All they could do was to 
adjust
FIT_LIMIT, which would only let them adjust the _absolute_ convergence 
criterion,
which would require them to guess what their minimum really was.  With 
the
new criterion, the default convergence criterion is always relative no 
matter
what the chisquare is, but users now have the flexibility of adding an 
absolute
convergence criterion through FIT_LIMIT_ABS.  The new scheme defaults to
doing the same thing as the old scheme in almost all cases, is more 
useful
in some (rare) cases, is easier to understand, and is more flexible.


The load-file newfit0.dem tries to fit a line to the file newfit0.dat.
With old gnuplot, the fit gives up when the parameters are still pretty 
bad,
with the new convergence criteria the fit continues until it is right.


7.  New one-line progress-report, revert by FIT_CLASSIC_PROGRESS = 1
The old report was many lines per iteration, so you could not hold many
iterations in the scroll buffer, and it was hard to see how chisquare,
parameters, and lambda were changing.  It also said nothing about 
iterations
where the chisquare increased (except to print an asterisk).  It printed
"WSSR" with no explanation that this means weighted sum of squared 
residuals,
rather than the more common term "chisquare".  The new report is one 
line
per iteration, with everything in neat columns so it's easy to track 
the progress.
The first column is the chisquare, the second is the change in 
chisquare divided
by the convergence limit, so convergence means a value between -1 and 
0, the
lambda value, and one column per parameter.  It also shows iterations 
where
the chisquare increases.  Implementation is through new function 
show_fit1(),
called by regress() [and now also by marquardt()]. Both console output 
and
fit.log file are affected by the change.  The user can revert to the old
progress report show_fit() by setting FIT_CLASSIC_PROGRESS = 1.


8.  New final fit parameter report format, revert by FIT_CLASSIC_RESULT 
= 1
The old default final report label "final sum of squares of residuals"
was misleading because if the user supplied errors, the number printed
was the chisquare (_weighted_ sum of squared residuals or WSSR). The new
report labels are "chi-squared, "degrees of freedom," and "chisq/ndf"
when the user supplies errors.  If the user did not supply errors, it 
prints
"Sum of squared residuals" and "degrees of freedom", and not SSR/ndf 
because
it is not particularly meaningful in that case.  In both cases, it 
prints
an error-rescaling factor (square root of internal chisquare per degree 
of freedom,
calculated with error=1 if errors are not supplied).  A new internal 
variable
errColumn is set to 1 in fit_command() if the user supplied errors, 
otherwise
it is zero.  If the user supplied errors, both the raw and rescaled 
parameter
errors are printed.  If the user did not supply errors, only the 
rescaled
parameter errors are printed.  In both cases, the same thing goes to 
the fit.log
file.  The old printing code was moved into new function 
PrintResults(), and
the new printing code is in  PrintResults2(). A new internal variable 
classicResults
controls which function is called by regress().  It defaults to 0 (new 
results)
but setting FIT_CLASSIC_RESULT = 1 restores the old results report.


The load-file newfit1.dem fits lines to the files newfit1.dat and 
newfit1a.dat,
switching back and forth between the old and new progress and results 
reports.


9.  Error-rescaling control
The old default was to always rescale parameter errors by the square 
root
of the "chisquare" per degree of freedom (unless there were zero degrees
of freedom, in which case errors were considered undefined).  While 
this is
the only sensible thing to do if the user does not supply errors to the 
fit,
it is not the only sensible thing to do if errors are supplied.  The 
parameter
errors are calculated in regress() as before, but now they are only 
rescaled
by the square root of chisquare per degree of freedom if new internal 
variable
rescaleErrs is true (non-zero).  The rescaleErrs variable is set in 
fit_command()
along with errColumn, but at the moment it is always set to 1 whether 
the user
supplied errors or not.  This means that the internal variables for 
parameter errors
(not default, but available through recompiling with a preprocessor 
option) and
the (non-default) old-style result report always give only rescaled 
errors, as before,
unless someone changes the source and recompiles.  The new default 
results report
from FitResults2() checks rescaleErrs and errColumn so it can print the 
right values
(both raw and rescaled if the user supplied errors, only rescaled if 
the user did
not supply errors) no matter how rescaleErrs is set.


10.  Derivative-step algorithm improvement, revert by 
FIT_CLASSIC_DRV_STEP = 1
The fit needs derivatives with respect to parameters of the prediction 
at each
data point. These are calculated numerically by changing the parameters 
by a small
amount.  The old step size was DELTA=0.001 times the parameter value 
(or 1e-33
if the parameter is less than 1e-30).  In some cases, this step is so 
large that
non-linearities in the function make the derivative inaccurate.  This 
slows or
prevents convergence, and may make the location of the minimum wrong.  
In other
cases, the step is so small that the predicted function value does not 
change
to machine precision, so the calculated derivative is zero.  This 
usually causes
a singular matrix error.  There is an optimal step size that balances 
roundoff error
and non-linearities (see the discussion of numerical derivatives in 
Numerical
Recipes).  It requires an estimate of the roundoff error of the 
function, and an
estimate of the second derivative.  The function that we ultimately 
care about
is the chisquare.  We can estimate the roundoff error from the data 
values, and
the matrices that the fit accumulates anyway can be used to estimate 
the second
derivative.  The calculation adapts to the data, function, and 
parameter values.
In most cases, the step size is much less than 0.001 times the 
parameter value,
although it is larger when it needs to be to avoid roundoff errors.  
The new
algorithm is implemented by allocating (and freeing) in marquardt() a 
new array
drvStepDA, initialized by calling new InitDrvSteps(), updated during 
convergence
by calling new CalcDrvSteps()and changing calculate() to use the array.
The new algorithm is the default, but the old algorithm (re-implemented 
in the
new functions) can be restored by setting FIT_CLASSIC_DRV_STEP = 1.


The load file newfit2.dem shows a case where the new derivative step 
algorithm
works but the old algorithm step is too small, resulting in a singular 
matrix error.

The load file newfit3.dem shows a case where the old derivative step
is too large so the fit does not really converge, while the new 
algorithm is OK.


11.  Improved treatment of parameters near zero
Previously, an initial parameter value less than NEARLY_ZERO = 1.0e-30
was silently changed to 1.0E-30. But in some cases it is natural for
parameters to be this small or smaller.  Rather than silently overriding
the user's input when near zero, the new version uses any initial value
with no change.  The exception is that an initial value of EXACTLY zero
is disallowed.  The reason is that the numerical derivative algorithm
needs at least the order of magnitude of the parameter for 
initialization.
There is a message telling the user to provide a non-zero
value of the right magnitude, or the magnitude of the expected error
if the expected value is zero.  The old code was removed from 
fit_command(),
the new code is CheckParms(), called at the start of regress().
The old behavior of creating a new variable with initial value 1.0
when a fit parameter doesn't yet exist is retained, but a print 
statement
was added to createdvar() advising the user that it is better to provide
a non-zero explicit value.


12.  Parameter step size limit, controlled by FIT_MAX_PAR_STEP
For non-linear functions, particularly when the initial parameter 
values are far
from optimum, the fit may try to change the parameters to values where 
the function
is undefined, which terminates the fit with an error (at least in the 
new code,
the user gets more information about the parameter values and data 
values that
caused the undefined result!).  The only thing the user could do was to
change the starting point and pray.  A new function LimitParSteps() 
called by
marquardt() to scale down the steps in all parameters so the ratio of 
any parameter
change to its value is no larger than internal variable maxParStep.  
The new user
variable FIT_MAX_PAR_STEP controls maxParStep.  The default value of 
maxParStep
is 1.5, which allows a factor of 2.5 increase per iteration, and also 
for the sign
to change.  Note that maxParStep values of less than 1.0 would not 
allow the
parameter to change sign, which could either be useful, or dangerous.


13.  Scale-independence through multiplicative lambda, revert by 
FIT_CLASSIC_LAMBDA
The lambda parameter in the Levenberg-Marquardt algorithm reduces the 
step size
if the calculated step actually increases the chisquare.  The amount of 
reduction
for the different parameters depends on details of the implementation.  
One common
implementation is "multiplicative", which is dimensionless and is 
insensitive to
parameter scale differences.  The implementation in gnuplot is 
"additive" which
makes the performance sensitive to the relative scale of parameters and 
errors.
Additive lambda requires a somewhat arbitrary initial value 
calculation, while
multiplicative lambda can be simply initialized to 0.01.  The 
interpretation of
multiplicative lambda is easy (much larger than 1 means the step sizes 
are reduced,
much less than 1 means the step sizes are "ideal"), while it is hard to 
interpret
the value of additive lambda.  My preference is for therefore for 
multiplicative
lambda, although there are certainly combinations of data, function, 
and starting
point where additive lambda will work and multiplicative lambda will 
fail.  The
implementation is through changes to marquardt() in both initialization 
and usage
of lambda.  The default is now multiplicative lambda initialized to 
0.01, but
additive lambda is still available by setting FIT_CLASSIC_LAMBDA = 1.  
Users can
set FIT_START_LAMBDA to change the initial lambda value in either case. 
  But for
additive lambda, supplying a FIT_START_LAMBDA value overrides the 
initial value
calculation, and there is no way to turn it back on without restarting 
gnuplot
(this has always been true).

The load file newfit4.dem shows a case where a simple line fit gives 
the wrong
answer with additive lambda (because it thinks it is converged when it 
has not),
and multiplicative lambda gives the right answer.


14. "Skeptical" chisquare for nonlinear fits, controlled by 
FIT_SKEPTICAL
In a nonlinear fit, the influence of a given data point on the fit 
parameters
(which depends on the parameter derivatives at the data point) can be 
highly
sensitive to the values of the parameters.  This can result in slow 
convergence,
unless the initial parameters are chosen very close to the (unknown) 
desired minimum.
An example is a function like a*exp(b*x), where the derivatives at 
large x can be
orders of magnitude larger than at small x.  This interferes with 
convergence,
because Levenberg-Marquardt refuses to take a step if the chisquare 
goes up,
so if it gets close to the data at high x, even tiny parameter changes 
make the
chisquare worse (unless they are balanced in a nonlinear way).  These 
problems
can be reduced in some cases by adjusting the weights of data points to 
cancel
out the dependence of the derivatives on the parameters.  I call this
"skeptical chisquare" because it doesn't take the initial parameter 
values
as seriously.  In some cases, the convergence time goes from thousands 
of
iterations to only a few.  However, the location of the minimum of the 
skeptical
chisquare will be close to the minimum of the normal chisquare if the 
functional
form can reproduce the data well, but it will not necessarily be very 
close if
the function does not reproduce the data, at least over the region 
being fit.
Also, the errors calculated with the skeptical chisquare are not very 
meaningful.
For these reasons, a skeptical chisquare fit should only be used to 
determine better
initial values for a normal chisquare fit.  There are if-statements in 
marquardt()
that test new variable skepticalChisq, and re-scale the internal errors.
The default is 0 for normal chisquare.  The user can turn it on by 
FIT_SKEPTICAL = 1,
and off by FIT_SKEPTICAL = 0.


The load file newfit5.dem gives an example with a*exp(b*x) where 
"convergence"
takes thousands of iterations with normal chisquare, even starting with 
both
parameters close to right, but only a few with skeptical chisquare, and 
the
final refit with normal chisquare takes only a few iterations.

15. Monte Carlo search for initial fit parameters
There are many problems where fits only converge when the initial 
parameters are
rather close to the solution, and it can be tedious finding such a 
starting point.
To make this easier, I have implemented a Monte Carlo search for the 
minimum chisquare
over a user-defined range, which is used as the initial parameters for 
the normal
Levenberg-Marquardt fit.  The range information is input using the "via 
file"
mechanism.  The "via file" parser in fit_command() was modified to 
allow (but not
require) two values per line.  If any lines have two values, a Monte 
Carlo search
for starting parameter values is done.  Parameters with only a single 
value are held
constant for the Monte Carlo search, but still vary in the final fit. 
The existing
"# FIXED" syntax for non-parameter constants was not changed.  The 
default Monte Carlo
search is 1000 iterations.  The user can change this by setting 
FIT_MONTE_CARLO_TRIALS.
The default behavior is to continue with a regular fit from the best 
chisquare point.
But if FIT_MONTE_CARLO_TRIALS is negative, after (minus) that many 
trials, control goes
back to the user without a fit, but with the internal parameter values 
set to the
best trial.  This lets the user plot the function with these values as 
a diagnostic.
The random number generator is not re-seeded, so repeating the search 
with the same number
of iterations on the same function and data always gives the same 
result.

The load file newfit6.dem shows sine-wave data where fits fail unless 
the frequency
parameter is quite close to the right value.  The Monte Carlo search 
does quite a good
job in 1000 iterations, which takes only seconds, and the fit converges 
well from this
starting point.


I compared the new fitting code results with the old version on fit.dem.
The first fit starts with both parameters initialized to zero, which is 
now
illegal, so I changed fit.dem to initialize them to 0.001. When the 
switches
are set to revert to the old algorithm, the results are nearly 
identical.
The first fit takes a few extra iterations because of maxParStep.
With the default changes to the derivative and lambda algorithms, the
results are noticably different on the hemisphere fit. I traced this to
the fact that the function has a square root, and for some of the data
points, the argument is negative.  The fit only uses the real part of
the function, so it is zero for those data points.  So the chisquare
actually has multiple minima, depending on how many data points give
negative square roots.  The modified fit.dem now makes a plot of the
argument of the square root, and also defines a new fit function with
a Taylor expansion of the square root that is defined for all arguments.
With the modified fit function, the new and old versions give the same
result (and both converge in far fewer iterations, implying that most
of the iterations were being confused by the multiple minima).
The new version generally tends to converge in somewhat fewer 
iterations,
although for some of the cases in fit.dem, the new version takes more 
iterations.



                                         Cheers

Prof. Thomas Mattison, Dept. of Physics & Astronomy, Univ. of British 
Columbia

Present Address: Stanford Linear Accelerator Center
2575 Sand Hill Road, Menlo Park, CA, 94025
Building 48 (Research Office Building), Mail Station MS35
Office:  ROB-231   Phone: 650-926-5342   Fax: 650-926-8522

How to check in changes?

[Thomas M. <mat...@ph...>]

Hi

I have made a bunch of improvements to gnuplot fitting.
They are in src/fit.c, plus some changes to demo/fit.dem.
They are described in a separate message.

How do I commit the changes?  I started by CVS downloading
gnuplot 4.1, made my changes, and have done a CVS update since
making my changes, so I should be in synch with everyone else.

But when I tried to CVS commit, I got a broken pipe abort.
I'm working on a Linux system.

I have also made some new short demo files to demonstrate
the reason for and effect of the improvements.  These would
be new files, and they are really for developers rather than
users.  How should they be handled?

Finally, the documentation should be changed to reflect the
changes in fitting.  What is the proper way to edit the source
for documentation, and to test the changes?


                                         Cheers

Prof. Thomas Mattison, Dept. of Physics & Astronomy, Univ. of British 
Columbia

Present Address: Stanford Linear Accelerator Center
2575 Sand Hill Road, Menlo Park, CA, 94025
Building 48 (Research Office Building), Mail Station MS35
Office:  ROB-231   Phone: 650-926-5342   Fax: 650-926-8522

Re: Expecting all terminals to be the same [was: Re: x11 point symbols]

[Ethan A M. <merritt@u.washington.edu>]

On Wednesday 01 March 2006 03:32 am, Petr Mikulik wrote:
> 
> Thus yet another missing compatibility are dashed styls -- set terminal 
> {no}dashed or set termoption {no}dashed .. and then dashed styles ...

Right. It's the same issue. Only a few terminal types can draw dashed lines.
Adding the command itself is easy; making it do something useful is
another question.

-- 
Ethan A Merritt
Biomolecular Structure Center
University of Washington, Seattle 98195-7742