gnuplot-beta — Developers and beta testers of new gnuplot code

Re: memory leak?

[Daniel J S. <dan...@ie...>]

Dave Denholm wrote:
 
> load makes a recursive call into the command parser. So you have set
> up an infinite recursion.

[snip]

> 
> Problem goes away if you spell it
> 
> $ cat /tmp/recurse 
> load '/tmp/pause'
> reread

I ran with that construct for a few hours.  I'm still seeing memory very slowly increasing.

Was your thought that the command parser or command memory is slowly increasing and that should account for the memory increase?

Dan

Re: border inside key

[Ethan M. <merritt@u.washington.edu>]

On Monday 13 March 2006 12:03 am, Daniel J Sebald wrote:
> I just noticed that for the "all.dem" many of the plots with a
> bordered key don't have any space between the left edge and the text.
>  Anyone notice if that started recently?  Or has that been like that
> for a while?

I am not noticing anything that matches your description.
Which plots in particular?  Is it driver-dependent?
Maybe you had better send me a screen-shot.

	Ethan

-- 
Ethan A Merritt
Biomolecular Structure Center
University of Washington, Seattle WA

Re: memory leak?

[Daniel J S. <dan...@ie...>]

Daniel J Sebald wrote:

> All give that a try.

I'LL!... typping two flast.

Re: memory leak?

[Daniel J S. <dan...@ie...>]

Dave Denholm wrote:

> Problem goes away if you spell it
> 
> $ cat /tmp/recurse 
> load '/tmp/pause'
> reread

All give that a try.

Dan

Re: memory leak?

[Dave D. <dde...@es...>]

Daniel J Sebald <dan...@ie...> writes:

> Last night I started a script "allinf.dem" consisting of
>
> load "all.dem"
> load "allinf.dem"
>
> run with the system command line:
>
> gnuplot allinf.dem < /dev/null
>


load makes a recursive call into the command parser. So you have set
up an infinite recursion.

If you attached a debugger, you'd probably see a very deep call stack.

Yeah - if I have

$ cat /tmp/pause 
pause 1 "hello"

$ cat /tmp/recurse
load "/tmp/pause"
load "/tmp/recurse"

Then I run it for a while, and then

(gdb) where
#0  0xdfb7a923 in _libc_sigsuspend ()
#1  0xdfb939f0 in _libc_usleep ()
#2  0xdfbca8ad in usleep ()
#3  0x80515fb in pause_command () at command.c:997
#4  0x8050831 in command () at command.c:511
#5  0x8050409 in do_line () at command.c:368
#6  0x8079032 in load_file (fp=0x811d010, name=0x8122f30 "/tmp/pause", can_do_args=0 '\000') at misc.c:267
#7  0x805143b in load_command () at command.c:871
#8  0x8050831 in command () at command.c:511
#9  0x8050409 in do_line () at command.c:368
#10 0x8079032 in load_file (fp=0x811d000, name=0x8122f18 "/tmp/recurse", can_do_args=0 '\000') at misc.c:267
#11 0x805143b in load_command () at command.c:871

[snip]

#164 0x8050831 in command () at command.c:511
#165 0x8050409 in do_line () at command.c:368
#166 0x8079032 in load_file (fp=0x811cd90, name=0x8122b70 "/tmp/recurse", can_do_args=0 '\000') at misc.c:267
#167 0x805143b in load_command () at command.c:871
#168 0x8050831 in command () at command.c:511
#169 0x8050409 in do_line () at command.c:368
#170 0x8079032 in load_file (fp=0x811cd80, name=0x804714b "/tmp/recurse", can_do_args=0 '\000') at misc.c:267
#171 0x807fe59 in main (argc=1, argv=0x8046f98) at plot.c:622



Problem goes away if you spell it

$ cat /tmp/recurse 
load '/tmp/pause'
reread


dd
-- 
Dave Denholm              <dde...@es...>       http://www.esmertec.com

Re: [Gnuplot-developers] [ gnuplot-Patches-1448674 ] Stop X11 palette refresh unnecessarily for mouse redraw

[Daniel J S. <dan...@ie...>]

Daniel J Sebald wrote:
> ...  but hold on a bit, I have to fix the bug that 
> Juergen found.

Updated...

memory leak?

[Daniel J S. <dan...@ie...>]

Last night I started a script "allinf.dem" consisting of

load "all.dem"
load "allinf.dem"

run with the system command line:

gnuplot allinf.dem < /dev/null

What I found was that last night the minimum memory usage in the cycle for gnuplot/gplt_x11 was 6.4/5.9 MB, but this morning it was 12.2/5.9 MB which suggests.  I can't think of why it should go up.

Is there some better way to test for memory leaks?

Dan

Re: [Gnuplot-developers] [ gnuplot-Patches-1448674 ] Stop X11 palette refresh unnecessarily for mouse redraw

[Daniel J S. <dan...@ie...>]

Ethan A Merritt wrote:
> On Sunday 12 March 2006 11:36 pm, Daniel J Sebald wrote:
> 
>>Here is a fix for preventing the palette from reloading
>>when not necessary for mouse redrawing, i.e. to stop
>>"allocating colors..." from appearing so often and
>>slowing things down.
> 
> 
> Is there a test script that demonstrates this?
> I don't recall ever seeing such a message.
> 
> 
>>That keeps gplt_x11.c from having to reconstruct the
>>color tables unless necessary.  The refresh speedup is
>>clearly back to what it once was.
> 
> 
> You mean this broke recently?  I must not have been
> paying attention.  What broke it?

It has been like this for quite some time after a fix about a year ago where the gplt_x11 palette was changing when more than one x11 window appeared on the screen.

The fix resulted in "allocating colors..." to appear on the x11 title bar (not stdout).

Try

> set pm3d
> splot x

and then use the mouse to rotate the image.  The sluggish behavior is fixed by the patch...  but hold on a bit, I have to fix the bug that Juergen found.

Dan

Re: palette allocation during mouse rotation

[Daniel J S. <dan...@ie...>]

Juergen Wieferink wrote:
> Am Montag, 13. M=E4rz 2006 08:36 schrieb Daniel J Sebald:
>=20
>>OK, I've made a fix for the palette allocation problem and placed it on=
 the
>>SourceForge patch page (1448674).  This should make us all happier.
>>
>>For reference, the original email was of Jan 7, 2006.  Basically, Petr
>>could apply this at any time, but I left it on the patch page for Hans =
and
>>Ethan to take a run through.  I made a complete "copy_at()" in eval.c
>>patterned after "free_at()".  Using free_at() as a model should mean no
>>memory is assigned by copy_at() that won't be deleted by free_at().  So=
,
>>take a look at that.
>=20
>=20
> I've curiously taken a look into your copy_at() because IIRC
> free_at() was written by me. :-)
>=20
> I do see a minor problem with your code:
>=20
> eval.c (copy_at):
>=20
> +	    if ( a->index =3D=3D PUSHC || a->index =3D=3D DOLLARS ) {
> +                if ((a->arg.v_arg.type =3D=3D STRING)
> +			&& a->arg.v_arg.v.string_val
> +			&& (b->arg.v_arg.v.string_val
> +			    =3D (char *) gp_alloc(strlen(a->arg.v_arg.v.string_val)+1, "cop=
ied=20
> v_arg")))
> +		    strcpy(b->arg.v_arg.v.string_val, a->arg.v_arg.v.string_val);
> +		else
> +		    b->arg.v_arg.v.string_val =3D NULL;
> +            }
>=20
> If the current action "a" pushes an integer or real value (quite
> common), this code seems to init ...string_val to NULL.  But
> "string_val" is the char* member of a *union*, resetting the numeric
> value to be pushed.  AFAICS, you should omit the "else" part.
>=20
> I haven't compiled or tested your implementation, and probably it
> won't harm for your specific usage of copy_at().

You are right.  It is a union, which I knew, but I just wasn't thinking s=
traight.  I will fix that and update the patch.

Thanks Juergen,

Dan

Re: [Gnuplot-developers] [ gnuplot-Patches-1448674 ] Stop X11 palette refresh unnecessarily for mouse redraw

[Petr M. <mi...@ph...>]

> Is there a test script that demonstrates this?
> I don't recall ever seeing such a message.

Do
     splot x with line palette
or
     set pm3d
     splot x

and use mouse to rotate plots.

---
PM

Re: [Gnuplot-developers] [ gnuplot-Patches-1448674 ] Stop X11 palette refresh unnecessarily for mouse redraw

[Ethan A M. <merritt@u.washington.edu>]

On Sunday 12 March 2006 11:36 pm, Daniel J Sebald wrote:
> 
> Here is a fix for preventing the palette from reloading
> when not necessary for mouse redrawing, i.e. to stop
> "allocating colors..." from appearing so often and
> slowing things down.

Is there a test script that demonstrates this?
I don't recall ever seeing such a message.

> That keeps gplt_x11.c from having to reconstruct the
> color tables unless necessary.  The refresh speedup is
> clearly back to what it once was.

You mean this broke recently?  I must not have been
paying attention.  What broke it?
 
> Basically, Petr could apply this at any time, but I left it
> on the patch page for Hans and Ethan to take a run through.


-- 
Ethan A Merritt
Biomolecular Structure Center
University of Washington, Seattle 98195-7742

border inside key

[Daniel J S. <dan...@ie...>]

I just noticed that for the "all.dem" many of the plots with a bordered key don't have any space between the left edge and the text.  Anyone notice if that started recently?  Or has that been like that for a while?

Dan

Re: Instructions in FAQ for anonymous CVS don't seem to work

[Daniel J S. <dan...@ie...>]

Was that problem with the CVS archive ever fixed.  It works intermitently, but today wasn't working at all.  Keep getting:

Logging in to :pserver:ano...@cv...:2401/cvsroot/gnuplot
CVS password:
cvs [login aborted]: end of file from server (consult above messages if any)

and

cvs [checkout aborted]: end of file from server (consult above messages if any)

Dan

Re: palette allocation during mouse rotation

[Daniel J S. <dan...@ie...>]

OK, I've made a fix for the palette allocation problem and placed it on the SourceForge patch page (1448674).  This should make us all happier.

For reference, the original email was of Jan 7, 2006.  Basically, Petr could apply this at any time, but I left it on the patch page for Hans and Ethan to take a run through.  I made a complete "copy_at()" in eval.c patterned after "free_at()".  Using free_at() as a model should mean no memory is assigned by copy_at() that won't be deleted by free_at().  So, take a look at that.

Also, in getcolor.c are a couple routines

copy_udft()
clear_udft()

Although they are conceptually similar to copy_at() and free_at(), I figured they really aren't as portable so avoided bloating things too much.  (Note copy_udft() and clear_udft() only get included in the core gnuplot routine, not gplt_x11.)

Dan


Petr Mikulik wrote:
>> Here's a patch for that gnuplot "allocating colors..." redraw problem 
>> with rotation.  The princple is as follows:
> 
> 
> It looks OK. I passed "pm3dcolors.dem" on it (with "set pm3d map" => 
> "set pm3d"), rotated by mouse, it's no more reallocating colors and no 
> strange thing happened.
> 
> BTW, I prefer to surround unused code by #if 0 #endif instead of /**  
> */; it's more readable.
> 
>> OK, so that's one problem down and two to go.  I'll see if I can get 
>> to another one next weekend.
> 
> 
> Please resend the patch when it's final for cvs.
> 

Re: Gridding and color from input data

[<br...@ph...>]

Julien Duponchelle wrote:
> "color.plot", line 3: Gridding of the color column is not implemented

That message is quite clear, I think.  You're using dgrid3d, but it 
doesn't know how to interpolate colors.  So don't use dgrid3d. 
Transform your data file into a gnuplot grid dataset (--> help splot 
data, help glossary) by inserting structuring blank lines.

> I want to kown when this  features will be add to the dev version 

It's somewhat unlikely it'll ever be added.  I'm not even sure it'll 
read RGB triplet from a file in the first place, but it certainly won't 
do interpolation among them.

Re: Changes to gnuplot fitting submitted to sourceforge as patch

[Thomas M. <mat...@ph...>]

On 10-Mar-06, at 10:45 AM, Bastian Maerkisch wrote:

>
> Personally I would prefer new options to `set fit` instead of dozens 
> of new
> FIT_xxx variables. This would be more consistent with setting other
> options in gnuplot.

Controlling things through a "set fit" mechanism rather than FIT_xxx
is an appealing idea, I wish I had thought of it.  It makes controlling
fits more like controlling other things in gnuplot.  Certainly for
_new_ controls where there is no back-compatibility issue it sounds
better than more FIT_xxx clutter in the namespace.  I just copied the
code in fit.c which didn't use set, because that was easier.  I'll
look at your patch to learn how to use set and make an alternative 
patch.

Perhaps we could also have "set fit xxx" control things presently
controlled via FIT_xxx, then slowly deprecate FIT_xxx.  We'd have
to have a rule for how to resolve conflicts if both mechanisms were
trying to control things at the same time.  The simplest is just to
let the most recent change apply, and change both the set/show state
and the FIT_xxx state.  It would then be nice to issue a warning if
the value is set with one mechanism then changed using the other
mechanism, since that is more likely to be a mistake than changing
the value many times through the same mechanism.  I don't know if
dual-control would be easy or hard, I haven't looked at the 
implementations.

I'd also like to make the fit_command parser accept the same syntax
for errors that the plot command does, for further consistency 
improvement.
But that's not my highest priority.

> Attached you find an extension to gnuplot's fit which I have been using
> for a while. It adds the possibility to turn off error scaling via
> `set fit errorscaling` and saving of fit information (chisq, dof, WSSR)
> to user variables as requested by Hans Boie (SF #1117724 [fit] access
> to resulting chisquare) via `set fit fitvariables`. It also fixes
> (SF #1324672 ] doc: false reference to "set fit").
>

Having a switch like this may be the only way to get me and Hans-
Bernhard to stop going back and forth about it ;-)

And having variables for chisq, etc turned on by GP_FIT_ERRVARS
switch is probably a good idea (although then we'd have to agree on 
names.. ;-)

> These are just tiny modifications and could probably be integrated
> into this large and very nice patch.
>
> Bastian
>

It's nice to be appreciated!

Re: gnuplot, wxterminal and antialiasing

[<tim...@en...>]

> On Friday 10 March 2006 03:10 pm, Timoth=E9e Lecomte wrote:
>
>> This problem comes from the terminal API, which works with integers
>> everywhere. That's why 'plot x', on all platforms, looks wobbly to
>> some extent instead of being a straight diagonal.
>
> I was able to mitigate this problem dramatically for the svg termimal
> simply by increasing the driver's internal scale by a factor of 10.
> Screen coordinates are still integers, but the resolution is 10x better=
,
> and the wobble is essentially eliminated.
>
>> Please take a look at this screeshot
>
> Here are a pair of screenshots, before and after increasing the
> svg internal scale by a factor of 10.
>
> http://www.bmsc.washington.edu/people/merritt/gnuplot/scale_1.png
> http://www.bmsc.washington.edu/people/merritt/gnuplot/scale_10.png

Ok, so this is the same method as what I called "oversampling". The only
difference is that I used a higher scale factor, 10000.

>> As for me, I prefer the rendering with the modified oversampling
>> wxterminal because it gets rid of the annoying aliased and wobbling
>> lines.
>
> Does it affect the speed?
> I'm already concerned that the wxterminal is slow compared to x11.

I see exaclty the same speed as without "oversampling", that is to say it
is still much slower than x11, but not worse than without oversampling.

Regarding the speed of the wxWidgets/Cairo rendering, there may be severa=
l
bottlenecks :
- first, my code is surely not optimised (a lot of mutex lock, etc.),
- then Cairo itself is still young so the upcoming 1.2 version may contai=
n
speed improvements,
- and finally I can use a backend drawing with Glitz, so that Cairo will
use  OpenGL and profit from the gpu acceleration

I sincerely think that the first point is the most pertinent, because eve=
n
if I disable antialiasing, the speed does not improve much...
However, in theory, the antialiasing makes the rendering slower, that's s=
ure.

Best regards,

Timoth=E9e

Re: gnuplot, wxterminal and antialiasing

[Ethan M. <merritt@u.washington.edu>]

On Friday 10 March 2006 03:10 pm, Timoth=E9e Lecomte wrote:

> This problem comes from the terminal API, which works with integers
> everywhere. That's why 'plot x', on all platforms, looks wobbly to
> some extent instead of being a straight diagonal.

I was able to mitigate this problem dramatically for the svg termimal
simply by increasing the driver's internal scale by a factor of 10.
Screen coordinates are still integers, but the resolution is 10x better,
and the wobble is essentially eliminated.

> Please take a look at this screeshot

Here are a pair of screenshots, before and after increasing the
svg internal scale by a factor of 10.

http://www.bmsc.washington.edu/people/merritt/gnuplot/scale_1.png
http://www.bmsc.washington.edu/people/merritt/gnuplot/scale_10.png

> As for me, I prefer the rendering with the modified oversampling
> wxterminal because it gets rid of the annoying aliased and wobbling
> lines.

Does it affect the speed?=20
I'm already concerned that the wxterminal is slow compared to x11.


=2D-=20
Ethan A Merritt
Biomolecular Structure Center
University of Washington, Seattle WA

gnuplot, wxterminal and antialiasing

[<tim...@en...>]

Hi !

I've had the pleasure to see the first comment of H.B. Broeker on the
behaviour of the wxWidgets/Cairo terminal. Definitely, this comment raise=
s
an interesting question (at least it interests me).

Here is the comment :

   One thing jumped at me almost immediately: the anti-aliased
   line drawing seems not work out all that well.  Two main
   issues there are:

   1) the usual "plot x" line at default settings is
   terribly
   wobbly.  Yes, the windows terminal does that, too.  But with
   anti-aliasing it looks even worse, somehow.

   2) the diagonal line of 'plot x' looks considerably
   fatter than the key sample.

   Both effects appear even more striking in

       set polar
       set size ratio -1
       plot 2.23

   So: may I be so bold as to ask for a terminal switch that
   turns anti-aliasing off?  Maybe it's only my LCD screen
   that's acting up here, but frankly, I don't think
   anti-aliasing is doing gnuplot all that much good.  At least
   here on my screen it seems to make typical aliasing
   artefacts (like the wobble on both the diagonal line and the
   circle) stand out even more, instead of working against them.


I am well aware of the problem, and I have several ideas about it. This
problem comes from the terminal API, which works with integers everywhere.
That's why 'plot x', on all platforms, looks wobbly to some extent instea=
d
of being a straight diagonal.

Please take a look at this screeshot http://tipote.free.fr/compare.png
I have done this screenshot to compare the different available interactiv=
e
terminals regarding the wobbling effect. You will see :

- Top Left : the wxterminal I am working on, in its current state.
It features antialiasing and its side-effects as H.B.B noted them :
wobbling more visible than in other terminals because our eyes are no mor=
e
disturbed by the aliasing, and key sample thinner because it is drawn
exactly on the centers of the pixels so it doesn't use/need any
antialiasing.

- Bottom Left : the Windows terminal, through Wine.
Here, aliased lines, definitely wobbling too.

- Bottom Right : the X11 terminal.
Basically the same rendering as the Windows terminal (apart from the fat
box), with aliased lines (and fonts) and wobbling.

- Top right : a modified wxterminal, which cheats by saying to gnuplot
that it has 10000 times more pixels than it really has. This oversampling
allows to use the Cairo API at its full extent, as it is based on doubles
instead on integers (motivated by the so-called subpixel accuracy of the
antialiasing method). This way, the diagonal of 'plot x' is purely
straight and clean ! Drawbacks : some ticks, the key samples and more
generally all horizontal or vertical lines are no longer placed on intege=
r
coordonates, so they are blurred by the antialiasing algorithm.

To see the same terminals render the polar 'plot 2.23' example quoted by
H.B.B., see here http://tipote.free.fr/compare2.png


As for me, I prefer the rendering with the modified oversampling
wxterminal because it gets rid of the annoying aliased and wobbling lines.
However, it definitely blurs all the lines of the box, of the ticks and o=
f
the key.

What do you think of it ?

Should there be an option in the wxterminal to choose between :
   noantialiasing,
   antialiasing,
   antialiasing+oversampling
?

On the long run, a solution could be to rewrite or complete the term API
in terms of double instead of integers, but still use integers for the
box/ticks/key. Do you think it's feasible ?

Thank you for your help and you rinterest in the wxWidgets terminal !

Timoth=E9e

Re: palette allocation during mouse rotation

[Daniel J S. <dan...@ie...>]

Hans-Bernhard Br=F6ker wrote:
> Daniel J Sebald wrote:
>=20
>> As part of a palette comparison a copy of the previous palette is=20
>> stored.  Part of that is copying a udft_entry.  Right now it is only a=
=20
>> partial copy of udft_entry as the color routines don't use all parts=20
>> of the udft_entry in the test.
>=20
>=20
> No.  Don't even think of it.
>=20
> If you need a copy of a data structure, you make a copy, period.  Copie=
s=20
> that aren't copies are a nightmare we absolutely don't need.

OK.  Full copy it is.

Dan

Re: Changes to gnuplot fitting submitted to sourceforge as patch

[Bastian M. <bma...@we...>]

>=20
> Line breaks with some indentation might work, though.  Maybe yet anothe=
r
> new parameter: FIT_LOG_WRAP_COLUMN (zero means don't wrap)?

Personally I would prefer new options to `set fit` instead of dozens of n=
ew
FIT_xxx variables. This would be more consistent with setting other
options in gnuplot.

Attached you find an extension to gnuplot's fit which I have been using
for a while. It adds the possibility to turn of error scaling via
`set fit errorscaling` and saving of fit information (chisq, dof, WSSR)
to user variables as requested by Hans Boie (SF #1117724 [fit] access
to resulting chisquare) via `set fit fitvariables`. It also fixes
(SF #1324672 ] doc: false reference to "set fit").

These are just tiny modifications and could probably be integrated
into this large and very nice patch.

Bastian

--=20
Bastian M=E4rkisch
Physikalisches Institut, Universit=E4t Heidelberg

Gridding and color from input data

[Julien D. <dup...@ep...>]

Hello,

I had test the last devellopement version from CVS and i need a feature.

I want to draw a grid  from a file like this:
0 0 0 0x00ff00
0 1 0 0x0000ff
0 2 0 0x00ffff
0 3 0 0xff0000
0 4 0 0xff0000
1 0 0 0xff0000
1 1 0 0xff0000
1 2 0 0xff0000
1 3 0 0xff0000
1 4 0 0xff0000
2 0 0 0xff0000
2 1 0 0xff0000
2 2 0 0xff0000
2 3 0 0xff0000
2 4 0 0xff0000d


And a plot like this:
set dgrid3d 5,5,1
set pm3d corners2color median
splot "test.dat" using 1:2:3:4 with lines lc rgb variable


But gnuplot display this error message:
"color.plot", line 3: Gridding of the color column is not implemented

I want to kown when this  features will be add to the dev version to see if
i need to search another solution.


Thanks for your great job

--
Julien Duponchelle
Etudiant =E0 l'Epitech (http://www.epitech.net)

Re: Changes to gnuplot fitting submitted to sourceforge as patch

[<br...@ph...>]

Thomas Mattison wrote:
> On 8-Mar-06, at 10:34 AM, Hans-Bernhard Bröker wrote:

[A side note: you should subscribe to gnuplot-beta if you're going to
send mail there --- if you don't, each of your submissions will sit in
limbo until I get round to approving it...]

>>> Yeah, I wondered whether that would be sufficiently portable.
>>> Is there a random generator internal to gnuplot that I should use?
>>
>> Hmm, we do have specfun.c:ranf().  That should fit the bill.
> 
> I looked at specfun.c.  There's something declared
> static double ranf(struct value *init)
> but it's not visible outside specfun.c, and I didn't know what to do
> with the struct value* .

ranf() is the real function you need.  struct value *init() is
equivalent to state argument of the thread-safe variant of drand48.
f_rand is an interface to ranf() to be called by users, i.e. the thing
they get if they "print rand()" at the command prompt.

Well.  For the moment, let's just stick to rand()/RAND_MAX for the
moment, and fix this later.  I can export a drand48() like interface
from specfun.c any time.

>> ["TM" is Thomas Mattison, HBB are my, Hans-Bernhard Broeker's, comments]

[Points not replied to should be considered as agreed upon.]

>> 4.  Zero-change in chisquare is now "BETTER" rather than "WORSE"

[...]

> I think it would be hard to construct a test case where an iteration
> produces EXACTLY zero change in chisquare, where the following iterations
> succeeded in improving the chisquare.  The only cases I can think of
> where the chisquare change would be EXACTLY zero are that the parameters
> are already at the minimum to machine precision, or the value of lambda
> is so large that the parameter steps are so small that the chisquare
> doesn't change.  In the first case, it's OK if we stop iterating.  In
> the second case, a result of WORSE will make lambda even bigger, so
> the next step will be even smaller, so it will also produce zero change
> in chisquare.

One possibility I can imagine is that a local extremum isn't a point,
but a large exactly horizontal plateau.  This would be an ill-posed
problem, of course, but since AFAIK nobody has ever found a way of
teaching users not to pose any of those, there we go.  The correct
reaction to that kind of situation, should be to find the boundaries of
that plateau and see if the slope is up or down, out there.  If the
change does that, I'm all for it.

> I have added a print statement that tells the user the
> truth when the fit has terminated because lambda hit the maximum. But
> in order for this to be useful, we need to make sure that this
> _isn't_ printed for a fit that has actually found the minimum. The
> way to do that is to make EXACTLY zero change in chisquare be BETTER
> instead of WORSE, so the loop exits then, rather than waiting for
> lambda to hit the limit.

Ah, now there's an argument even I can understand ;-).  OK, then.

>> TM: Without this, if the user interrupted the fit and tried to
>> plot the current function on top of the data, the last
>> internal parameter was not correct.
>>
>> HBB: in that case, the fit interruption handler is what needs fixing,
>> not call_gnuplot
> 
> That would be another way of fixing things, but is there any significant
> reason why calculate() should not undo the perturbation of the last
> parameter that it had just done?

The problem is that a hard interruption could, in principle, happen
anywhere inside calculate().  At least that's how I remember this being
handled in the Linux versions, where this is done directly by SIGINT.

>> 6.  Changed convergence criterion, with new user-variable FIT_LIMIT_ABS.

>> TM:
[...]

>> HBB: I have to disagree here.  Of course users could impose a
>> change: they could supply the missing error values.  Fake them by a

> But this looks like a case where it's easier to fix the code so it works
> even when somewhat abused, rather than explaining what constitutes abuse,
> why it causes the problem, and how to get the results they want.

Let's just say I'm not fully convinced that the exact usage rules of
FIT_LIMIT_ABS will be less prone to incorrect usage than error-less fits
already are in general.  Users tend to just see a knob they can turn
which will let their fits print "converged" at the end, and never look
back to find out how it works, or whether it should be used in a
particular case.  We may be giving the poor guys a gun to shoot
themselves with instead of teaching them how to fish.

>> 7.  New one-line progress-report, revert by FIT_CLASSIC_PROGRESS = 1

>> HBB: the meaning of all abbreviations is explained in the
>> documentation.  There's no excuse for user not reading the documention
>> of a tool as complex as this.  Considerable work has gone into that
>> documentation in the past.  One result of much deliberation during that
>> activity was was to call that quantity WSSR, instead of chisquare.
>> I'd rather not go through all that again.

> Ah, if they would only read documentation....

Well, making it easier to avoid reading it isn't going to help with that
problem.  So let's not make it any easier than we have to.

> Is the discussion you mention recorded anyplace?  

I don't think so.  That was a rather lengthy email exchange back in
1997, between me and Lucas Hart of "orst.edu" (no where that is).
 From a quick peek at it, Lucas' point against using "chisquare" in the
printouts was that it must be the same as the one in "chisquare
distribution" and "chisquare test".  But that's true only in some cases.

> But it's easy to change my column label to WSSR if that's really
> more universally understood, or even to SSR for no-errors fits
> and WSSR for fits with errors.

The truth is gnuplot always does weighted SSR (sometimes the weights
are just 1.0), i.e. printing WSSR is never really wrong.  It's just
confusing until people read the docs.  Which, IMHO, is actually a Good
Thing(TM).

[...]

> I considered adding line breaks, but decided against it.  They don't
> solve the readability problem for narrow consoles, compared to letting
> the text wrap.  

Line breaks with some indentation might work, though.  Maybe yet another
new parameter: FIT_LOG_WRAP_COLUMN (zero means don't wrap)?

>> 9.  Error-rescaling control

> I do advocate changing the default behavior.  It's statistically wrong to
> rescale the errors according to the chisquare, if the user provided valid
> errors.  

Well, it's statistically wrong to take seriously _anything_ the fit
prints when the chisquare/ndf is far enough away from 1 to make a
difference.  Such fits are plain any simply inacceptable.  So to some
extent, it doesn't matter at all what we do with them: any result will
be just as wrong as any other.

 From a different point-of-view, a chisq/ndf far from 1 means the data
errors don't explain the differences between data and model.  Either the
data errors are correct --- then the model is wrong.  Or the data errors
are (as they so often are) bollocks.  'fit' has no way of knowing which
is the case.  It has to favour one of them blindly, or it has to give up
right away, and just refuse to print errors at all.

> If we repeat the same experiment and fit many times, the data will
> have statistical fluctuations, the fit parameters will have statistical
> fluctuations, and the chisquare will have statistical fluctuations.  

... and the parameter errors will also have statistical fluctuations.
Which will generally be no smaller than those of chisquare itself.
So the dividing them doesn't actually increase the variation of the
reported parameter errors considerably.

>> 10.  Gnuplot-readable parameters and errors in one line in fit.log file

> The point is that I want to _append_ many fit results to a
> gnuplot-readable file, so the results from many similar fits
> could be conveniently plotted along with their errors.  The
> file from the update command doesn't seem to be appropriate for
> appending results from many fits.

Not yet.  But 'update's job is more similar to what you're doing than
that of the fit.log file.

Sticking that machine-readable data into the middle of a human-readable
data stream, from which it'll have to be extracted before it can be
used, doesn't really look like a good idea.  A new command "update
append 'myfits.dat'" or whatever would make much more sense, from a user
interface point-of-view.  For one thing, it gives the user an
opportunity to choose which fits to put into the summary data file, and
which not to.

> Can you give a reason _why_ fit.log is the wrong place?

Because its primary purpose is to be human-readable, not
machine-readable.  Because for all you know, it already contains a lot
of data the moment you start gnuplot.  fit.log is, basically, an
electronic lab notebook, not a worksheet to collect data from various
steps of a single experiment in.

>> 13.  Parameter step size limit, controlled by FIT_MAX_PAR_STEP

>> what Marquardt's lambda parameter is designed to do.  Even if it's not
>> an exact duplicate, this seems to almost beg for a fight between
>> those two mechanisms over who gets to decide how big the step sizes
>> should be...
>> Couldn't the same effect be had by a simple penalty on the chisquare
>> to steer the existing algorithm away from those regions?

> It's not a duplication of what lambda does, it addresses a different
> problem.  

But it does so in a similar manner: limiting the step size.

> On the first iteration of a nonlinear fit, particularly if
> the initial parameters are poor, the first parameter step may be
> to a region where the function is not even defined, which results in
> a failure.  

This is not particular to the first iteration.  The algorithm can come
close to the boundary of the models definition space any time.  For all
we know, the minimum itself could be exactly on the boundary.

> Lambda doesn't reliably prevent this, because it can take
> several iterations for lambda to increase far enough to limit step
> sizes by itself.

So let it take several iterations.  If necessary, help it by signalling
undefined values with a penalty on chisquare.

>> 14.  Scale-independence through multiplicative lambda,
>> revert by FIT_CLASSIC_LAMBDA = 1

>> TM: implementation is "multiplicative", which is dimensionless and is
>> insensitive to parameter scale differences.  The implementation
>> in gnuplot is "additive" which makes the performance sensitive to
>> the relative scale of parameters and errors.
>>
>> HBB: I must admit you've lost me there.  Could you pass me some
>> references about these two different lambda's?

[...]

> It's true that a sophisticated user can rescale the problem to circumvent
> the problems with additive lambda.  

It's also true that a sophisticated fitting program can do that for him,
automatically...

 > But that's never necessary with
> multiplicative lambda.  I think multiplicative is a better default,
> because it's scale-independent and it's easier to interpret the lambda 
> value.

I'll have to ponder this for a while longer.

>> 16. Monte Carlo search for initial fit parameters

>> HBB: Nice.  So all we now miss for a complete typical fitting
>> mess-of-tools is the Nelder-Meade Simplex search algorithm.

> I thought about adding simplex, but from what I've read and heard,
> its main strength is dealing with discontinuous derivatives, which
> chisquare fits don't normally have (though it might do better on the
> original hemisphere-fit demo problem....)  

"Normally" is not something we can easily rely on.  The Simplex method 
is, of course, necessary where derivatives can't be used at all, and 
it's better than MC at finding a starting point in completely unknown 
terrain because it doesn't restrict itself to a limited parameter range. 
  It worked well for me when I was still using CERN's MINUT a lot.

Re: [ gnuplot-Patches-908040 ] Break PostScript prologue file out of post.trm

[<br...@ph...>]

[Sorry, forgot to Cc: the group.  This is important for others, too.]

>>> If you look, you'll see that the help directory is
>>> similarly set to
>>>     GIHDIR = ${prefix}/share/gnuplot/4.1

>> I see curly braces they --- those are wrong.

Well, on second inspection, the curly braces are correct.  The
actual problem is the reverse: the above definition is not wrong.  If
anything it's more correct than we're used to.  What does that mean, you
ask?

Well, if you compare Makefiles generated with and without a --prefix=
option to configure, you find a strange difference. Those generated
without a prefix fall back to a *hard-coded* setting of
datadir=/usr/local/share (and all others depending on ${datadir}).
That's wrong, and has been wrong ever since --with-gihdir was added to
configure.in.  The culprit is this snippet:

1.127        (lhecking 03-Feb-03): dnl .gih help file location
1.127        (lhecking 03-Feb-03): eval gp_datadir=$datadir
1.127        (lhecking 03-Feb-03): if test "$gp_datadir" = NONE/share; then
1.127        (lhecking 03-Feb-03):   datadir="/usr/local/share"
1.127        (lhecking 03-Feb-03): fi

Without this, @datadir@ in a Makefile.in would have been expanded to
${prefix}/share all along.  This makes datadir behave different from all
other predefined autoconf/automake directory variables.  The immediate
consequence is that GIHDIR can't be passed into the build via config.h,
because the preprocessor doesn't know how to expand ${prefix}.  The
reason this actually works is that we don't try that:  only HELPFILE
uses GIHDIR, and HELPFILE is passed to $(CC) as a -D... flag, which
means make expands ${prefix} in the process, and all's well.

I.e. the fact that this seemed to work at all for GNUPLOT_PS_DIR was
caused by a combination of bugs.  The solution is: handle GNUPLOT_PS_DIR
like HELPFILE.  We can fix configure.in later.

> By the way, this is probably the same problem that I'm
> seeing with Timothée's wxWidgets patchset. 
> The path name for his icon set gets expanded as
> "NONE/share/gnuplot/" on a default configuration.

Yes.  NONE is what the prefix looks like while configure runs, if no 
prefix was specified.

As a workaround, you can make it a habit to always use
--prefix=/usr/local or --prefix=$HOME.

Re: palette allocation during mouse rotation

[<br...@ph...>]

Daniel J Sebald wrote:
> As part of a palette comparison a copy of the previous palette is 
> stored.  Part of that is copying a udft_entry.  Right now it is only a 
> partial copy of udft_entry as the color routines don't use all parts of 
> the udft_entry in the test.

No.  Don't even think of it.

If you need a copy of a data structure, you make a copy, period.  Copies 
that aren't copies are a nightmare we absolutely don't need.