gnuplot-beta

What's left for a 4.2 release?

[Ethan M. <merritt@u.washington.edu>]

This attempts to be a complete list of unresolved issues that I see
as release-critical.   It's a very short list.

Bugs
========

#1512210  buffer overflow using pgnuplot
		
	Sounds serious, but no one has confirmed the bug.
	We never heard back from the original reporter.


Patches
========

#1510816  Postscript prologues files

	Works fine under linux/DU4/irix/etc, and OK though not
	perfectly on VMS.  The sticking point is Windows.
	Is this workable under Windows, or do we have to provide a
	compile-time option to include all of the PostScript
	boilerplate in the driver itself?  I'd hate to take what
	I consider to be a step backwards, but if we must - we must.

	On the other hand, maybe this isn't really release critical.
	If it can be fixed by providing a Windows installer bundle,
	then we could freeze and release now, and add a bundled 
	installer on the "Files" page afterwards.

#1505261  wgnuplot: open file-open-dialog in current dir

	I don't care whether this goes in, or gets vetoed,
	but we need a decision one way or the other.


Documentation
=============

#######	 Nothing in particular. Just a thorough check to make sure
	we are properly describing the new version.


Feature Requests
================

#1117724  [fit] access to resulting chisquare

	I proposed exporting user-visible variables named

	FIT_CONVERGED - 1 if the fit has converged, 0 otherwise
	FIT_NDF  - number of degrees of freedom (#obs - #param)
	FIT_RMS  - RMS fit of model to data;
	           a.k.a. "std fit"
                   a.k.a. sqrt(sum_of_squares/ndf)
	FIT_CHI2 - variance (reduced chi-squared residual) after fit;
	           a.k.a. (sum_of_squares/ndf)

	Hans-Bernhard has objected to these names, but no one has
	suggested anything better. Please speak up.

#1513118  Standardize all config and Makefiles for 4.2

	We have set the ./configure script to select most of the
	new features by default.  But the makefiles for platforms
	not using the configure script do not match this (see below).
	Should they?


			Ethan


The relevant #defines in config.h are

/* Define to enable parsing of deprecated syntax */
#define BACKWARDS_COMPATIBLE 1

/* Define if you want to support files in binary format. */
#define BINARY_DATA_FILE 1

/* Define to allow reading strings from datafiles */
#define EAM_DATASTRINGS 1

/* Define to enable histograms plot style. */
#define EAM_HISTOGRAMS 1

/* Define to allow placement of rectangles and other objects */
#define EAM_OBJECTS 1

/* Define to allow 'fit' to create parameter error variables. */
#define GP_FIT_ERRVARS 1

/* Define to allow command line macros. EXPERIMENTAL */
#define GP_MACROS 1

/* Define to allow string variables. */
#define GP_STRING_VARS 2

/* Define to enable quadtree optimization in hidden3d code. */
#define HIDDEN3D_QUADTREE 1

/* Define to enable handling point size in hidden3d code. */
#define HIDDEN3D_VAR_PTSIZE 1

/* Define if you want to have image plotting support. */
#define WITH_IMAGE 1



-- 
Ethan A Merritt
Biomolecular Structure Center
University of Washington, Seattle WA

Re: What's left for a 4.2 release?

[<tim...@en...>]

Ethan Merritt wrote:
> Patches
> =3D=3D=3D=3D=3D=3D=3D=3D
>
> #1510816  Postscript prologues files
>
> 	Works fine under linux/DU4/irix/etc, and OK though not
> 	perfectly on VMS.  The sticking point is Windows.
> 	Is this workable under Windows, or do we have to provide a
> 	compile-time option to include all of the PostScript
> 	boilerplate in the driver itself?  I'd hate to take what
> 	I consider to be a step backwards, but if we must - we must.
>
> 	On the other hand, maybe this isn't really release critical.
> 	If it can be fixed by providing a Windows installer bundle,
> 	then we could freeze and release now, and add a bundled=20
> 	installer on the "Files" page afterwards.
>  =20
The Windows-specific code is in my patch.

If you are sure that other platforms are ok with GNUPLOT_PS_DIR (I doubt=20
OS/2 is), then just take my patch without the "#ifndef GNUPLOT_PS_DIR",=20
without the *.h and without the script.

Timoth=E9e

Re: What's left for a 4.2 release?

[Petr M. <mi...@ph...>]

> Bugs
> ========
>
> #1512210  buffer overflow using pgnuplot
>
> 	Sounds serious, but no one has confirmed the bug.
> 	We never heard back from the original reporter.

Thus it should be closed.

> #1117724  [fit] access to resulting chisquare
>
> 	I proposed exporting user-visible variables named
>
> 	FIT_CONVERGED - 1 if the fit has converged, 0 otherwise
> 	FIT_NDF  - number of degrees of freedom (#obs - #param)
> 	FIT_RMS  - RMS fit of model to data;
> 	           a.k.a. "std fit"
>                   a.k.a. sqrt(sum_of_squares/ndf)
> 	FIT_CHI2 - variance (reduced chi-squared residual) after fit;
> 	           a.k.a. (sum_of_squares/ndf)
>
> 	Hans-Bernhard has objected to these names, but no one has
> 	suggested anything better. Please speak up.

I prefer the proposed names FIT_*

> #1513118  Standardize all config and Makefiles for 4.2
>
> 	We have set the ./configure script to select most of the
> 	new features by default.  But the makefiles for platforms
> 	not using the configure script do not match this (see below).
> 	Should they?

Yes, at least those maintained.

I've confirmed it for config.{os2, mgw, cyg, nt}

> /* Define to enable quadtree optimization in hidden3d code. */
> #define HIDDEN3D_QUADTREE 1

But this is not default in ./configure.
I never tried this option. It's nowhere written whether it is useful. If it 
is, then let us have it on.

---
PM

Re: What's left for a 4.2 release?

[Ethan M. <merritt@u.washington.edu>]

On Tuesday 11 July 2006 01:27 pm, Petr Mikulik wrote:
>
> I've confirmed it for config.{os2, mgw, cyg, nt}

Great.

> > /* Define to enable quadtree optimization in hidden3d code. */
> > #define HIDDEN3D_QUADTREE 1
>
> But this is not default in ./configure.
> I never tried this option. It's nowhere written whether it is useful.
> If it is, then let us have it on.

Sorry for the confusion.
I've had this option selected in my test-build script for
so long that I forgot is was not the default.

So far as I know, the only reason to turn it off is so that you
can try HIDDEN3D_GRIDBOX instead.  The two options are mutually
incompatible.

-- 
Ethan A Merritt
Biomolecular Structure Center
University of Washington, Seattle WA

Re: What's left for a 4.2 release?

[Daniel J S. <dan...@ie...>]

Ethan Merritt wrote:
> This attempts to be a complete list of unresolved issues that I see
> as release-critical.   It's a very short list.
> 
> Bugs
> ========
> 
> #1512210  buffer overflow using pgnuplot
> 		
> 	Sounds serious, but no one has confirmed the bug.
> 	We never heard back from the original reporter.

I still think #1488168 should be addressed before 4.2.

set xyplane at 0
splot x+y

The line going down beyond the used range is a clear bug and restricts any users from manually adjusting things.

Dan

Re: What's left for a 4.2 release?

[Petr M. <mi...@ph...>]

> I still think #1488168 should be addressed before 4.2.
>
> set xyplane at 0
> splot x+y
>
> The line going down beyond the used range is a clear bug and restricts any 
> users from manually adjusting things.

I like the patch and propose it for committing now.

---
PM

Re: What's left for a 4.2 release?

[<br...@ph...>]

Petr Mikulik wrote:
>> Bugs
>> ========
>>
>> #1512210  buffer overflow using pgnuplot
>>
>> 	Sounds serious, but no one has confirmed the bug.
>> 	We never heard back from the original reporter.
> 
> Thus it should be closed.

No.  It should be left alone.  Just because we're in a hurry doesn't 
mean we can ignore all bug reports that can't handle that speed.

>> #1117724  [fit] access to resulting chisquare

>> 	Hans-Bernhard has objected to these names, but no one has
>> 	suggested anything better. Please speak up.
> 
> I prefer the proposed names FIT_*

My objection was not to the FIT_* part, but to what comes after it.

>> /* Define to enable quadtree optimization in hidden3d code. */
>> #define HIDDEN3D_QUADTREE 1

> But this is not default in ./configure.
> I never tried this option. It's nowhere written whether it is useful. If it 
> is, then let us have it on.

It's useful.  The only reason I didn't enable it by default for 4.0 was 
that it had seen very little testing by anybody but myself, since I 
implemented it.  I think it should be made the ./configure default for 4.2

Re: What's left for a 4.2 release?

[Ethan M. <merritt@u.washington.edu>]

On Wednesday 12 July 2006 01:46 pm, Hans-Bernhard Br=F6ker wrote:
> >> /* Define to enable quadtree optimization in hidden3d code. */
> >> #define HIDDEN3D_QUADTREE 1
> >
> > It's nowhere written whether it is useful.
> > If it is, then let us have it on.
>
> It's useful.  The only reason I didn't enable it by default for 4.0
> was that it had seen very little testing by anybody but myself, since
> I implemented it.  I think it should be made the ./configure default
> for 4.2

I agree. All testing that I have done for the last 6 months or more
has had this option enabled.

=2D-=20
Ethan A Merritt
Biomolecular Structure Center
University of Washington, Seattle WA

Re: What's left for a 4.2 release?

[<br...@ph...>]

Ethan Merritt wrote:
> This attempts to be a complete list of unresolved issues that I see
> as release-critical.   It's a very short list.
> 
> Bugs
> ========
> 
> #1512210  buffer overflow using pgnuplot
> 		
> 	Sounds serious, but no one has confirmed the bug.
> 	We never heard back from the original reporter.

Let's put this on the back-burner.  Whatever it is, I'm reasonably sure 
it's been exactly like that since pgnuplot was first written.  There's 
no hurry fixing it now.

> #1510816  Postscript prologues files
> 
> 	Works fine under linux/DU4/irix/etc, and OK though not
> 	perfectly on VMS.  The sticking point is Windows.
> 	Is this workable under Windows, or do we have to provide a
> 	compile-time option to include all of the PostScript
> 	boilerplate in the driver itself?  I'd hate to take what
> 	I consider to be a step backwards, but if we must - we must.

I'll try to have a look at this, but: my time is severely limited.  New 
job, new town, you get the drift.

> #1505261  wgnuplot: open file-open-dialog in current dir
> 
> 	I don't care whether this goes in, or gets vetoed,
> 	but we need a decision one way or the other.

I vote for keeping it as-is.  The potential benefit is too small to 
warrant the risk of a last-minute fix.

IMHO, if there's any Windows-specific issue that *really* needs to be 
addressed, it's the #1 FAQ issue about wgnuplot: font in the text window 
is unreadably small for first-time users.  Has to be fixed once and the 
fix solidified by writing a wgnuplot.ini.  Unfortunately, this broke by 
outside interference, i.e. we changed nothing, but Redmond did, and 
poof, the original code that worked since wgnuplot 3.5 on Windows 3.1, 
suddenly fails.  Windows seems to have forgotten what "just give me the 
default font" means.

> 	FIT_CONVERGED - 1 if the fit has converged, 0 otherwise
> 	FIT_NDF  - number of degrees of freedom (#obs - #param)
> 	FIT_RMS  - RMS fit of model to data;
> 	           a.k.a. "std fit"
>                    a.k.a. sqrt(sum_of_squares/ndf)
> 	FIT_CHI2 - variance (reduced chi-squared residual) after fit;
> 	           a.k.a. (sum_of_squares/ndf)
> 
> 	Hans-Bernhard has objected to these names, but no one has
> 	suggested anything better. Please speak up.

Oh well, since I don't have the time to put my money where my mouth is 
--- go ahead and check it in as-is.

Re: What's left for a 4.2 release?

[<ha...@on...>]

On Wed, Jul 12, 2006 at 11:12:07PM +0200, Hans-Bernhard Broeker wrote:
> Ethan Merritt wrote:
> 
> > 	FIT_CONVERGED - 1 if the fit has converged, 0 otherwise
> > 	FIT_NDF  - number of degrees of freedom (#obs - #param)
> > 	FIT_RMS  - RMS fit of model to data;
> > 	           a.k.a. "std fit"
> >                    a.k.a. sqrt(sum_of_squares/ndf)
> > 	FIT_CHI2 - variance (reduced chi-squared residual) after fit;
> > 	           a.k.a. (sum_of_squares/ndf)
> > 
> > 	Hans-Bernhard has objected to these names, but no one has
> > 	suggested anything better. Please speak up.
> 
> Oh well, since I don't have the time to put my money where my mouth is 
> --- go ahead and check it in as-is.
> 

I agree with Hans-Bernhard's point in Feature Request 1117724
that it is preferable to be consistent and use the same variable 
names as occur in the printout and documentation rather than
introduce additional notation for the same quantities.

Thus FIT_STDFIT instead of the above FIT_RMS, to indicate that the
denominator is ndf not #obs-1.  While STDFIT may not be familiar, 
it is documented.

However, gnuplot.doc could be clearer wrt rms of the residuals, i.e, 
when estimating the mean value of the residuals, one would use #obs-1, 
and refer to standard deviation of the residuals.  For standard 
deviation of the fit, one uses #obs-#param.  Thus 'variance of residuals' 
and 'rms of residuals' in gnuplot.doc are generic descriptions for the 
lack of an alternative characterization of WSSR/ndf and sqrt(WSSR/ndf).)

By analogy to STDFIT, I would suggest FIT_VARFIT for variance of 
the fit rather than some other notation for WSSR/ndf.

It is not clear in the request why one wants both FIT_STDFIT and 
FIT_VARFIT, each readily computed from the other.

BTW It is difficult to find posts relevant to a particular item when 
all have the same subject so I may have missed some mention of the fit
discussions related to a 4.2 release.  Any suggestions for a subject 
for any fit follow-up? 

I don't recall having seen any mention of incorporating patch 
 1445064  Gnuplot fitting improvements  2006-03-07
submitted by Thomas Mattison after a number of exchanges on the
mail list between he and HBB.

The discussions re that patch brought out the point that currently 
the documentation and the fit output label are consistent, but those 
are inconsistent with the values output by the fit routine, i.e., 
the documentation and fit output explicitly refer to parameter 
errors as "asymptotic standard error", i.e., as calculated from the 
variance-covariance matrix, with the intent that notation might cause
the less experienced fitter to consult the documentation and become 
aware of the limitations in using those values to determine confidence 
levels.

However, fit.c has the comment
 * HBB (br...@ph...) : fit didn't calculate the errors
 * in the 'physically correct' (:-) way, if a third data column
 * containing the errors (or 'uncertainties') of the input data was given.

and the code
	/* scale parameter errors based on chisq */
		chisq = sqrt(chisq / (num_data - num_params));
			for (i = 0; i < num_params; i++)
				    dpar[i] *= chisq;

Somehow, that discrepency was overlooked in earlier releases.

I would present the asymptotic standard error and leave it to the user
to determine any relationship between parameter errors as determined by 
the curvature of the chisquare hypersurface in the region of the minimum 
and confidence limits for those parameters rather than apply an
arbitrary scaling to the asymptotic standard errors.

That may be a point of contention - IIRC, the patch has a control
variable which allows the user to select unscaled errors, as documented,
or scaled errors, if such are desired, or for consistency with earlier
gnuplot versions.

- Lucas Hart

Re: What's left for a 4.2 release?

[Bastian M. <bma...@we...>]

Lucas Hart wrote:
=2E..
> I don't recall having seen any mention of incorporating patch=20
>  1445064  Gnuplot fitting improvements  2006-03-07
> submitted by Thomas Mattison after a number of exchanges on the
> mail list between he and HBB.
>=20
> The discussions re that patch brought out the point that currently=20
> the documentation and the fit output label are consistent, but those=20
> are inconsistent with the values output by the fit routine, i.e.,=20
> the documentation and fit output explicitly refer to parameter=20
> errors as "asymptotic standard error", i.e., as calculated from the=20
> variance-covariance matrix, with the intent that notation might cause
> the less experienced fitter to consult the documentation and become=20
> aware of the limitations in using those values to determine confidence =

> levels.
>=20
> However, fit.c has the comment
>  * HBB (br...@ph...) : fit didn't calculate the error=
s
>  * in the 'physically correct' (:-) way, if a third data column
>  * containing the errors (or 'uncertainties') of the input data was giv=
en.
>=20
> and the code
> 	/* scale parameter errors based on chisq */
> 		chisq =3D sqrt(chisq / (num_data - num_params));
> 			for (i =3D 0; i < num_params; i++)
> 				    dpar[i] *=3D chisq;
>=20
> Somehow, that discrepency was overlooked in earlier releases.
>=20
> I would present the asymptotic standard error and leave it to the user
> to determine any relationship between parameter errors as determined by=
=20
> the curvature of the chisquare hypersurface in the region of the minimu=
m=20
> and confidence limits for those parameters rather than apply an
> arbitrary scaling to the asymptotic standard errors.
>=20
Seconded. This scaling has been a major source of irritation to
to students in our lab courses.
But the scaling is not arbitrary. Effectively the errors of the
datapoints are scaled such that  reduced chi^2 =3D=3D 1.
That's very sensible if no data errors are supplied.

Btw. other widely used data analysis packages (including ROOT and
Origin) do not scale errors by default.

> That may be a point of contention - IIRC, the patch has a control
> variable which allows the user to select unscaled errors, as documented=
,
> or scaled errors, if such are desired, or for consistency with earlier
> gnuplot versions.

Personally, I would prefer new options to `set fit` instead of numerous n=
ew
FIT_xxx control variables. This would be much more consistent with the re=
st
of gnuplot.

--=20
Bastian M=E4rkisch
Physikalisches Institut, Universit=E4t Heidelberg