gnuplot-beta — Developers and beta testers of new gnuplot code

New body of code in development version: complex special functions

[Ethan A M. <me...@uw...>]

Since splitting off the stable branch for version 5.4 a year ago,
I have been working on new directions for development.
One of these is the expansion of support for complex functions
and in particular for complex special functions.
These are generally incomplete or altogether missing in other packages,
so it is an area where gnuplot can fill a gap.

This work is now merged into the main git repository.
Demos for some of the additions are in the on-line collection:

    http:/gnuplot.info/demo_5.5/

You can exercise them interactively in a new demo

    load 'special_functions.dem'

Here is a summary from the documentation.
The full manual contains additional information about the implementations
with references.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
This version of gnuplot provides a larger set of complex-valued functions
and updated versions of functions that were present in earlier versions. 

*  Updated incomplete gamma function with improved domain and precision. 
   Complex arguments accepted. 
*  Updated incomplete beta function with improved domain and precision. 
*  New function for the inverse incomplete gamma function. 
*  New function for the inverse incomplete beta function. 
*  New complex function LambertW(z,k) returns the kth branch of multivalued 
   function W_k(z). 
   Note that the older function lambertw(x) = real(LambertW( real(z), 0 )). 
*  New complex function lnGamma(z). 
   Note that existing function lgamma(x) = real(lnGamma(real(z)). 
*  Synchrotron function F(x). 
*  acosh(z) domain extended to cover negative real axis. 
*  asin(z) asinh(z) improved precision for complex arguments. 
*  Complex function conj(z) returns the complex conjugate of z. 
*  Predefined variable I = sqrt(-1) = {0,1} for convenience.

Additional special functions are supported if a suitable external
library is found at build time.
 *  Complex Bessel functions Iν(z), Jν(z), Kν(z), Yν(z) of order ν (real)
    with complex argument z.
 *  Complex Hankel functions H1ν(z), H2ν(z) of order ν with complex z.
 *  Complex Airy functions Ai(z), Bi(z).
 *  Complex exponential integral of order n.
 *  Fresnel integrals C(x) and S(x).

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%


There is room for further development.
Please add comments and suggestions!

- Existing built-in functions, old and new, are inconsistent about how
  they return very large values (floating overflow)
    Some return VERYLARGE (system-dependent but generally 9.e+307)
    Some return NaN
    Some return +Inf
  It would be nice to agree on a uniform convention and on how
  to handle it while plotting.

- Support for complex igamma(z) in the domain real(z) < 0 

- Support for complex elliptic integrals

- Modify "fit" to automatically handle complex-valued functions

- Support is dependent on external libraries found during configuration
  (libcerf libopenspecfun libamos).  Many linux distros have packages
  for libcerf and libopenspecfun.  I don't know how available they are
  for Windows or OSX. Much of the code in these libraries originally
  came from netlib and could be included in the gnuplot repository if
  we wanted to do so.

	Ethan

Re: 5.4.0: Mac OS X version available

[Ethan A M. <me...@uw...>]

On Wednesday, 23 September 2020 18:39:55 PDT Erik Luijten wrote:
> Dear Ethan et al.,
> 
> A Mac OS X package for version 5.4.0 is now available at
> https://csml-wiki.northwestern.edu/index.php/Binary_versions_of_Gnuplot_for_OS_X
> 
> If you encounter any problem, please let me know.
> 
> There is one small compilation issue that *seems* to be due to the way that
> src/Makefile is constructed. On the more recent versions of OS X, the X11
> libraries and header files are in /opt/X11, which ./configure does not
> detect. This is fine, since I can use the options '--x-include=/opt/X11/include
> --x-libraries=/opt/X11/lib' to specify this. However, the X11 library
> location does not get used in the compilation of gnuplot (only for
> gnuplot_x11, via the XLIBS variable), even though the main executable needs
> the Xpm library. Manually adding -L/opt/X11/lib to LDFLAGS in src/Makefile
> fixes this, but should not be necessary.
> 
> Regards,
> 
> Erik

Does the patch below fix it?

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
diff --git a/src/Makefile.am b/src/Makefile.am
index 0b261ea47..3a8b9b499 100644
--- a/src/Makefile.am
+++ b/src/Makefile.am
@@ -69,6 +69,7 @@ pkglibexec_PROGRAMS += gnuplot_x11
 gnuplot_x11_SOURCES = gplt_x11.c gplt_x11.h gpexecute.c gpexecute.h mousecmn.h version.c ver
sion.h
 XLIBS = @LIBRARIES_FOR_X@
 gnuplot_x11_LDADD = getcolor_x11.o $(XLIBS)
+gnuplot_LDADD += $(XLIBS)
 endif
 
 getcolor_x11.o: getcolor.c
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

	Ethan

Re: [PATCH] cairolatex: respect transparency setting

[Ethan M. <eam...@gm...>]

On Thursday, 24 September 2020 11:30:11 PDT Henri Menke wrote:
> Dear Ehtan,
> 
> I just wanted to ask what is the status of this patch?  It doesn't
> change the default behaviour but makes the notransparent option actually
> do something.  If you want I can write sentence in the documentation
> about the layer ordering.

As I understand it, you want this not because of a problem in the gnuplot
latex output but rather because of a bug/misfeature/quirk of some downstream
program that fails to validate the pdf file produced by running pdflatex on
the gnuplot output.  Do I have that right?

The thing is, if a user selects the "notransparent" option with your patch
applied then the output from gnuplot is incorrect.   
It loses the "back" layer of text, including the default axis tic labels.
This can be partially worked around by 
     set tics front
     set grid front
but then the axis borders and tics in a 3D plot incorrectly occlude the 
plotted surface.  This has the paradoxical effect of making the
"notransparent" option cause a pm3d surface to look transparent!

Therefore I do not like the patch as it stands.  In order to produce
correct 3D output further changes to the program would be required
to separate the handling of border text and lines.

Would it not be better to document that for some purposes
it may be helpful to post-process the png image?

       mogrify -flatten myfigure-inc.png

I think that achieves exactly the same result (including the problematic 3D border)
at the cost of one extra step.  But I could be wrong; maybe your validation program
still doesn't like it eve after flattening?

]    cheers,

	Ethan


> 
> Cheers, Henri
> 
> On 21/08/20, 13:34, Henri Menke wrote:
> > Currently the cairolatex terminal does not respect transparency options
> > given in the terminal setting.  In this minimal example the
> > notransparent option is ignored:
> > 
> >     set terminal cairolatex png notransparent
> >     set output "test.tex"
> >     plot sin(x)
> > 
> > Running this through gnuplot and then using `file' reveals that the
> > alpha channel is still present:
> > 
> >     $ gnuplot test.gnuplot && file test.png
> >     test.png: PNG image data, 1500 x 900, 8-bit/color RGBA, non-interlace
> > 
> > With this patch the transparency setting will be honoured.  The current
> > default setting of implicitly assuming transparency is unaltered.
> > ---
> >  term/cairo.trm | 2 +-
> >  1 file changed, 1 insertion(+), 1 deletion(-)
> > 
> > diff --git a/term/cairo.trm b/term/cairo.trm
> > index ee748dfe2..3ef76ecc6 100644
> > --- a/term/cairo.trm
> > +++ b/term/cairo.trm
> > @@ -849,7 +849,7 @@ void cairotrm_graphics()
> >  	plot.background.g = cairo_params->background.g;
> >  	plot.background.b = cairo_params->background.b;
> >  	gp_cairo_set_background(cairo_params->background);
> > -	if (ISCAIROLATEX || cairo_params->transparent)
> > +	if (cairo_params->transparent)
> >  		gp_cairo_clear_background(&plot);
> >  	else
> >  		gp_cairo_solid_background(&plot);
> 
> 
> _______________________________________________
> gnuplot-beta mailing list
> gnu...@li...
> Membership management via: https://lists.sourceforge.net/lists/listinfo/gnuplot-beta
> 

Re: [PATCH] cairolatex: respect transparency setting

[Henri M. <hen...@gm...>]

Dear Ehtan,

I just wanted to ask what is the status of this patch?  It doesn't
change the default behaviour but makes the notransparent option actually
do something.  If you want I can write sentence in the documentation
about the layer ordering.

Cheers, Henri

On 21/08/20, 13:34, Henri Menke wrote:
> Currently the cairolatex terminal does not respect transparency options
> given in the terminal setting.  In this minimal example the
> notransparent option is ignored:
> 
>     set terminal cairolatex png notransparent
>     set output "test.tex"
>     plot sin(x)
> 
> Running this through gnuplot and then using `file' reveals that the
> alpha channel is still present:
> 
>     $ gnuplot test.gnuplot && file test.png
>     test.png: PNG image data, 1500 x 900, 8-bit/color RGBA, non-interlace
> 
> With this patch the transparency setting will be honoured.  The current
> default setting of implicitly assuming transparency is unaltered.
> ---
>  term/cairo.trm | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/term/cairo.trm b/term/cairo.trm
> index ee748dfe2..3ef76ecc6 100644
> --- a/term/cairo.trm
> +++ b/term/cairo.trm
> @@ -849,7 +849,7 @@ void cairotrm_graphics()
>  	plot.background.g = cairo_params->background.g;
>  	plot.background.b = cairo_params->background.b;
>  	gp_cairo_set_background(cairo_params->background);
> -	if (ISCAIROLATEX || cairo_params->transparent)
> +	if (cairo_params->transparent)
>  		gp_cairo_clear_background(&plot);
>  	else
>  		gp_cairo_solid_background(&plot);
> -- 
> 2.28.0
> 

5.4.0: Mac OS X version available

[Erik L. <eri...@gm...>]

Dear Ethan et al.,

A Mac OS X package for version 5.4.0 is now available at
https://csml-wiki.northwestern.edu/index.php/Binary_versions_of_Gnuplot_for_OS_X

If you encounter any problem, please let me know.

There is one small compilation issue that *seems* to be due to the way that
src/Makefile is constructed. On the more recent versions of OS X, the X11
libraries and header files are in /opt/X11, which ./configure does not
detect. This is fine, since I can use the options '--x-include=/opt/X11/include
--x-libraries=/opt/X11/lib' to specify this. However, the X11 library
location does not get used in the compilation of gnuplot (only for
gnuplot_x11, via the XLIBS variable), even though the main executable needs
the Xpm library. Manually adding -L/opt/X11/lib to LDFLAGS in src/Makefile
fixes this, but should not be necessary.

Regards,

Erik


On Thu, Jul 16, 2020 at 2:28 PM Ethan A Merritt <me...@uw...> wrote:

> Gnuplot version 5.4 - first release
>
> I have placed a source tarball for 5.4 on SourceForge
>
>
> https://sf.net/projects/gnuplot/files/gnuplot/5.4.0/gnuplot-5.4.0.tar.gz
>
> Release Notes with new feature list
>
>    http://gnuplot.info/ReleaseNotes_5_4.html
>
> Version 5.4.0 is the start of a new "major release" series.
> It supersedes version 5.2, first released in September 2017.
>
> I realize that there may be unresolved issues with building Windows
> binaries
> for Windows 10, but I decided that it is not reasonable to hold up release
> any
> longer for issues that appear to be more related to system configuration
> than to
> the gnuplot source.  If necessary we can put out a Windows-specific
> patchlevel 1.
> As always, I am totally reliant on and extremely grateful to the volunteers
> who put together Windows binaries from a source release package.
>
>         cheers, and happy plotting
>
>                 Ethan
>
>
>
>
>
>
>
>
> _______________________________________________
> gnuplot-beta mailing list
> gnu...@li...
> Membership management via:
> https://lists.sourceforge.net/lists/listinfo/gnuplot-beta
>

Re: New Release package - gnuplot version 5.4

[Karl-Friedrich R. <mai...@gm...>]

Many thanks!

I noticed the new feature "Help/Edit (w)gnuplot.ini" in the menu.
Next to the editor (works fine), it additionally opens a black text
window titled "gnuplot 5.4 patchlevel rc1" (?!?) with a blinking
cursor in it, but anything typed there lands in the regular gnuplot
shell window. Closing that window kills wgnuplot.exe.

   Karl


Am 19.09.2020 um 18:00 schrieb Bastian Märkisch:
> Dear all,
>
> You can now find Windows binaries of the 5.4.0 release for testing on SourceForge:
>
>   https://sf.net/projects/gnuplot/files/gnuplot/testing/
>
> Please give them a try.
>
>   Bastian

Aw: New Release package - gnuplot version 5.4

[Bastian M. <bma...@we...>]

Dear all,

You can now find Windows binaries of the 5.4.0 release for testing on SourceForge:

  https://sf.net/projects/gnuplot/files/gnuplot/testing/

Please give them a try.

  Bastian


> Gesendet: Donnerstag, 16. Juli 2020 um 21:25 Uhr
> Von: "Ethan A Merritt" <me...@uw...>
> An: gnu...@li...
> Betreff: New Release package - gnuplot version 5.4
>
> Gnuplot version 5.4 - first release
>
> I have placed a source tarball for 5.4 on SourceForge
>
>    https://sf.net/projects/gnuplot/files/gnuplot/5.4.0/gnuplot-5.4.0.tar.gz
>
> Release Notes with new feature list
>
>    http://gnuplot.info/ReleaseNotes_5_4.html
>
> Version 5.4.0 is the start of a new "major release" series.
> It supersedes version 5.2, first released in September 2017.
>
> I realize that there may be unresolved issues with building Windows binaries
> for Windows 10, but I decided that it is not reasonable to hold up release any
> longer for issues that appear to be more related to system configuration than to
> the gnuplot source.  If necessary we can put out a Windows-specific patchlevel 1.
> As always, I am totally reliant on and extremely grateful to the volunteers
> who put together Windows binaries from a source release package.
>
>         cheers, and happy plotting
>
>                 Ethan
>
>
>
>
>
>
>
>
> _______________________________________________
> gnuplot-beta mailing list
> gnu...@li...
> Membership management via: https://lists.sourceforge.net/lists/listinfo/gnuplot-beta
>

Re: Strange autoscaling with rgbimage and inverted yranges

[Ethan A M. <me...@uw...>]

On Sunday, 13 September 2020 13:17:59 PDT Dima Kogan wrote:
> Hi. I do observe your sequence work at you state:
> 
> > 1) start gnuplot
> > 2) load script
> > 3) zoom
> > 4) 'u'
> >    plot is now incorrectly scaled
> > 5) 'n'
> >    shows zoom state from (3)
> > 6) 'p'
> >    correctly unzooms to original plot
> 
> Step 6 may be 'u' or 'p'. The reason things don't look the same after
> steps 4 and 6 is the value of
> 
>   { axis_array[FIRST_Y_AXIS].min, axis_array[FIRST_Y_AXIS].max }
> 
> at the start of the refresh_bounds() call. This function assumes
> min<max. And if we're reversing some axis, this is done at the very end
> of refresh_bounds() with the axis_check_range() calls.
> 
> In step 3, refresh_bounds() sees {min,max} = {4000,-500}
> In step 6, refresh_bounds() sees {min,max} = {1850,1400}
> 
> This is dependent on the actual zoom bounds. Both of these have the
> unexpected min>max, obviously, but in step 6 we get lucky, and it ends
> up doing the right thing anyway, in this case.

OK.
But no autoscaling at all should be necessary during zoom or unzoom.
That is true even for nonvolatile data, but is doubly true for volatile data.
The whole point of zoom is that the bounds are being chosen
interactively by the mouse, not by the content of the plot.

My thought is to add a flag somewhere that zoom/unzoom is in progress,
and skip all autoscaling operations if it is set.  In fact there already is a flag
"inside_zoom" but it isn't used for much.  I have not yet inventoried all the
places in the code that would need to be changed.  

	Ethan

Re: Strange autoscaling with rgbimage and inverted yranges

[Dima K. <gn...@di...>]

Hi. I do observe your sequence work at you state:

> 1) start gnuplot
> 2) load script
> 3) zoom
> 4) 'u'
>    plot is now incorrectly scaled
> 5) 'n'
>    shows zoom state from (3)
> 6) 'p'
>    correctly unzooms to original plot

Step 6 may be 'u' or 'p'. The reason things don't look the same after
steps 4 and 6 is the value of

  { axis_array[FIRST_Y_AXIS].min, axis_array[FIRST_Y_AXIS].max }

at the start of the refresh_bounds() call. This function assumes
min<max. And if we're reversing some axis, this is done at the very end
of refresh_bounds() with the axis_check_range() calls.

In step 3, refresh_bounds() sees {min,max} = {4000,-500}
In step 6, refresh_bounds() sees {min,max} = {1850,1400}

This is dependent on the actual zoom bounds. Both of these have the
unexpected min>max, obviously, but in step 6 we get lucky, and it ends
up doing the right thing anyway, in this case.

Specifically the diverging behavior comes from

  refresh_bounds()          calls
  process_image()           calls
  STORE_AND_UPDATE_RANGE()  calls
  store_and_update_range()

The logic in store_and_update_range() is effectively:

    if ( curval > axis->max && curval >= axis->min )
      axis->max = curval;

Which is correct only if axis->min < axis->max

I'm not entirely sure what the values of axis->min, axis->max should be
at the start of refresh_bounds(), but reordering them makes this thing
work correctly. Patch (usable for this specific test case only):

  diff --git a/src/plot2d.c b/src/plot2d.c
  index b2877eb89..7e4e57259 100644
  --- a/src/plot2d.c
  +++ b/src/plot2d.c
  @@ -299,6 +299,17 @@ refresh_bounds(struct curve_points *first_plot, int nplots)
       struct curve_points *this_plot = first_plot;
       int iplot;		/* plot index */

  +
  +    if(axis_array[FIRST_Y_AXIS].min > axis_array[FIRST_Y_AXIS].max)
  +    {
  +        double t = axis_array[FIRST_Y_AXIS].min;
  +        axis_array[FIRST_Y_AXIS].min = axis_array[FIRST_Y_AXIS].max;
  +        axis_array[FIRST_Y_AXIS].max = t;
  +    }
  +
  +
  +
  +
       for (iplot = 0;  iplot < nplots; iplot++, this_plot = this_plot->next) {
          int i;		/* point index */
          struct axis *x_axis = &axis_array[this_plot->x_axis];

Applying this patch makes the test case work, even before any of your
patches from yesterday. Clearly, this should be more general, but I'm
hazy on the details, so it'd be better if you finish this, probably.

Thanks!

Re: Strange autoscaling with rgbimage and inverted yranges

[Ethan A M. <me...@uw...>]

On Saturday, 12 September 2020 21:07:58 PDT Dima Kogan wrote:
> Dima Kogan <gn...@di...> writes:
> 
> > Ethan A Merritt <me...@uw...> writes:
> >
> >> And even after tracking it that far I still don't understand why
> >> 'p' works and 'u' doesn't.     If you can figure that out I'd be 
> >> happier with declaring it fixed.
> >
> > In my tests just now 'p' and 'u' both show this issue. Are you 100% sure
> > that 'p' does NOT show the issue for you?
> 
> I just tried this with the x11, qt and wxt terminals, and in all 3 cases
> both 'u' and 'p' show this issue. I'm at the git HEAD as off a few days
> ago:
> 
>   commit 160d7511bcbdfd0d1943a624af5fc42504a9f84c
>   Author: Ethan A Merritt <merritt@u.washington.edu>
>   Date:   Thu Sep 3 20:20:23 2020 -0700
> 
>       new function (long)argb = rgbcolor("name")
> 
> The test is:
> 
> 1. start gnuplot
> 2. "load" the script in the first email in this thread
> 3. interactively zoom
> 4. 'u' or 'p'
> 
> I'm on GNU/Linux. If I can reproduce the behavior you're seeing ('p' and
> 'u' work differently), then I'll be happy to debug it.

Try this:

1) start gnuplot
2) load script
3) zoom
4) 'u'
   plot is now incorrectly scaled
5) 'n'
   shows zoom state from (3)
6) 'p'
   correctly unzooms to original plot

Now that I try more sequences I notice that replacing 'p' with 'u'
in step (6) also works correctly.  So maybe the issue is whether
the previous operation was 'n'?  Or there's some parity issue
with the count of operations? 

Regardless, I think autoscaling in the current HEAD now works
correctly for your test case.
I also think that it is incorrect for the unzoom operation to invoke
autoscaling at all, but that would require additional code changes.

	Ethan

Re: Strange autoscaling with rgbimage and inverted yranges

[Dima K. <gn...@di...>]

Dima Kogan <gn...@di...> writes:

> Ethan A Merritt <me...@uw...> writes:
>
>> And even after tracking it that far I still don't understand why
>> 'p' works and 'u' doesn't.     If you can figure that out I'd be 
>> happier with declaring it fixed.
>
> In my tests just now 'p' and 'u' both show this issue. Are you 100% sure
> that 'p' does NOT show the issue for you?

I just tried this with the x11, qt and wxt terminals, and in all 3 cases
both 'u' and 'p' show this issue. I'm at the git HEAD as off a few days
ago:

  commit 160d7511bcbdfd0d1943a624af5fc42504a9f84c
  Author: Ethan A Merritt <merritt@u.washington.edu>
  Date:   Thu Sep 3 20:20:23 2020 -0700

      new function (long)argb = rgbcolor("name")

The test is:

1. start gnuplot
2. "load" the script in the first email in this thread
3. interactively zoom
4. 'u' or 'p'

I'm on GNU/Linux. If I can reproduce the behavior you're seeing ('p' and
'u' work differently), then I'll be happy to debug it.

Re: Strange autoscaling with rgbimage and inverted yranges

[Dima K. <gn...@di...>]

Ethan A Merritt <me...@uw...> writes:

> And even after tracking it that far I still don't understand why
> 'p' works and 'u' doesn't.     If you can figure that out I'd be 
> happier with declaring it fixed.

In my tests just now 'p' and 'u' both show this issue. Are you 100% sure
that 'p' does NOT show the issue for you?

Re: Strange autoscaling with rgbimage and inverted yranges

[Ethan A M. <me...@uw...>]

On Friday, 11 September 2020 22:51:03 PDT Ethan A Merritt wrote:

> I _think_ that the problem involves this macro in axis.h
> 
> %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
> /* Simplest form of autoscaling (no check on autoscale constraints).
>  * Used by refresh_bounds() and refresh_3dbounds().
>  * Used also by autoscale_boxplot.
>  */
> #define autoscale_one_point(axis, x) do {\
>     if (axis->set_autoscale & AUTOSCALE_MIN && x < axis->min) \
>         axis->min = x; \
>     if (axis->set_autoscale & AUTOSCALE_MAX && x > axis->max) \
>         axis->max = x; \
>     } while (0);
> %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
> 
> as called from plot2d.c:refresh_bounds() when zooming volatile data.
> 
> Note that the macro does not consider the case of reversed axes.
> Adding a test for (axis->min < axis->max) seems to make your
> test case work.   See attached patch.

Better patch (but still not fully correct)

	Ethan

Re: Strange autoscaling with rgbimage and inverted yranges

[Ethan A M. <me...@uw...>]

On Friday, 11 September 2020 20:13:33 PDT Dima Kogan wrote:

> In any case, this worked in 5.2. I just did a bisection, and I can now
> see why this seemed familiar. The commit that broke it:
> 
>   https://sourceforge.net/p/gnuplot/gnuplot-main/ci/36a2407e50/
> 
> The person who broke it was me! But despite what git says, I have very
> little recollection of any of this. Ethan: do you have a good sense of
> what's happening here, and can you easily fix it now? If not, I can poke
> at it, and figure it out.
> 
> dima

I am puzzled because I do not understand the difference between
returning to the top level zoom state by pressing 'p' to step backwards
through the stack of zoom operations and pressing 'u' to jump back to the
top of the stack.  Either way it should be applying the same state
(confirmed in the debugger) but for some reason that I don't understand
the former works correcly and the latter doesn't.

I _think_ that the problem involves this macro in axis.h

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
/* Simplest form of autoscaling (no check on autoscale constraints).
 * Used by refresh_bounds() and refresh_3dbounds().
 * Used also by autoscale_boxplot.
 */
#define autoscale_one_point(axis, x) do {\
    if (axis->set_autoscale & AUTOSCALE_MIN && x < axis->min) \
        axis->min = x; \
    if (axis->set_autoscale & AUTOSCALE_MAX && x > axis->max) \
        axis->max = x; \
    } while (0);
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

as called from plot2d.c:refresh_bounds() when zooming volatile data.

Note that the macro does not consider the case of reversed axes.
Adding a test for (axis->min < axis->max) seems to make your
test case work.   See attached patch.

The unpatched code does the wrong thing for reversed axes.
The patched code does nothing at all for reversed axes,
is it probably wrong also but for some case other than yours.
A more complete fix for the macro is wanted.

And even after tracking it that far I still don't understand why
'p' works and 'u' doesn't.     If you can figure that out I'd be 
happier with declaring it fixed.

                 Ethan
-- 
Ethan A Merritt
Biomolecular Structure Center,  K-428 Health Sciences Bldg
MS 357742,   University of Washington, Seattle 98195-7742

Re: Strange autoscaling with rgbimage and inverted yranges

[Dima K. <gn...@di...>]

Ethan A Merritt <me...@uw...> writes:

> I think the issue is that because your example uses in-line data it
> cannot be re-read. The program tries to work around this by reusing
> the contents of the various internal data structures but in this case
> the workaround fails. Possibly fixable, but I think the primary lesson
> is that data blocks are much preferred to in-line data if you are
> going to be manipulating/re-drawing the plot.

Hi.

100% of my fairly heavy usage of gnuplot uses inline data. The
particular case in this report is fairly common too: when doing any sort
of computer vision that annotates an image, you want to display the
source image with a flipped y axis, since the origin pixel is at the
top-left.

In any case, this worked in 5.2. I just did a bisection, and I can now
see why this seemed familiar. The commit that broke it:

  https://sourceforge.net/p/gnuplot/gnuplot-main/ci/36a2407e50/

The person who broke it was me! But despite what git says, I have very
little recollection of any of this. Ethan: do you have a good sense of
what's happening here, and can you easily fix it now? If not, I can poke
at it, and figure it out.

dima

Re: [bug] plot bins does not clip output to plot area

[Achim G. <Str...@ne...>]

Ethan A Merritt writes:
>> I'm using it as a way to average over a large number of samples that are
>> sampled equidistantly (kernel density is way slower obviously). The
>> documentation doesn't really tell how it is implemented or supposed to
>> work, so I've experimented a bit to find out.
>
> I take the blame for any failures in either the implementation or
> the documentation of the "bins" option, since I wrote both.
> I would be happy to amend the code or the description to cover
> uses that I didn't anticipate.

No blame to go around… any feature is prone to be (mis)used in ways
never intended, once it's there.  :-)

> Since I envisioned it as a histogramming tool, it did not occur to
> me that users would want to plot "with lines" rather than using
> boxes or impulses.   Therefore I did not think about or test the
> clipping behaviour.

No good deed goes unpunished, as they say.

> As it happens, the routines that draw boxes or impulses do the clipping as
> they draw each one.  The routine that draws "with lines" assumes that the
> inrange/outrange/undefined status of each point has already been flagged,
> and only considers clipping if a particular line segment changes state.
> This is so that successive in range points are drawn as a single poly-line
> with smooth joins rather than being split into individual segments.
> Normally the inrange/outrange flag is set on data entry, but those flags
> apply to the original data points rather than to the binned totals.
> I have now added a separate pass to re-check the binned data against
> yrange so that clipping works as you expect it to.
> (New code added for both 5.4 and 5.5).

Great, I'll check it out later.

> I remain a little uneasy about the use of negative values in the weighting
> column, although I don't have a specific example in mind that will fail.

So far I have not yet encountered any problems with that and I have
likely given this part quite a bit of exercise as most of the data I'm
plotting is roughly centered about zero.

> Thanks for the explanation of what you are using it for.
> I agree that in that mode "bins" is similar to a kernel density model
> that uses a delta function rather than a Gaussian kernel.

I would suggest that it would be welcome if one could specify the kernel
function and/or provide a way to implement general FIR filtering on the
original data.  While the moving average trick is cute, it is also quite
slow and cumbersome once you need to average over a large number of
samples.

> I may not have phrased that well.  It weights the _contribution_ of each
> sample by the value in the seconds column.  If no second column is
> provided, the weight is 1.

That seems a more accessible description to me, thanks.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Factory and User Sound Singles for Waldorf Blofeld:
http://Synth.Stromeko.net/Downloads.html#WaldorfSounds

Re: [bug] plot bins does not clip output to plot area

[Ethan A M. <me...@uw...>]

> I'm using it as a way to average over a large number of samples that are
> sampled equidistantly (kernel density is way slower obviously). The
> documentation doesn't really tell how it is implemented or supposed to
> work, so I've experimented a bit to find out.

I take the blame for any failures in either the implementation or
the documentation of the "bins" option, since I wrote both.
I would be happy to amend the code or the description to cover
uses that I didn't anticipate.

Since I envisioned it as a histogramming tool, it did not occur to
me that users would want to plot "with lines" rather than using
boxes or impulses.   Therefore I did not think about or test the
clipping behaviour.

As it happens, the routines that draw boxes or impulses do the clipping as
they draw each one.  The routine that draws "with lines" assumes that the
inrange/outrange/undefined status of each point has already been flagged,
and only considers clipping if a particular line segment changes state.
This is so that successive in range points are drawn as a single poly-line
with smooth joins rather than being split into individual segments.
Normally the inrange/outrange flag is set on data entry, but those flags
apply to the original data points rather than to the binned totals.
I have now added a separate pass to re-check the binned data against
yrange so that clipping works as you expect it to.
(New code added for both 5.4 and 5.5).

I remain a little uneasy about the use of negative values in the weighting
column, although I don't have a specific example in mind that will fail.

Thanks for the explanation of what you are using it for.
I agree that in that mode "bins" is similar to a kernel density model
that uses a delta function rather than a Gaussian kernel.

	cheers,

		Ethan

On Friday, 11 September 2020 11:59:28 PDT ASSI wrote:
> Ethan A Merritt writes:
> 
> > If there is a second column of data this is interpreted as a weight.
> 
> Let's assume the sample spacing is 1 and no samples are missing, then
> you'll get the sum over the bin width.  Different sampling density just
> scales the result.  Dividing by the number of samples gives you the
> average.

Correct.

[snip]

> 
> > Your test script provides no "using" specifier, however, so the plot
> > command draws values from column 1 (essentially the numbers 0 to 100)
> > and weights each one by the value in the second column (sin(x)).
> 
> It doesn't do that either or the result would be a linearly rising
> function with a sin riding on top of it.  It reproduces the function if
> I arrange the binwidth to contain just a single sample, so clearly the x
> column isn't used directly.

I may not have phrased that well.  It weights the _contribution_ of each
sample by the value in the seconds column.  If no second column is
provided, the weight is 1.

Re: [bug] plot bins does not clip output to plot area

[ASSI <Str...@ne...>]

Ethan A Merritt writes:
> I am confused as to what exactly you are trying to plot.
> The intent of the "bins" option is similar to the "smooth frequency" option.
> It derives an approximation to the distribution of values read from
> a single column of data.

I'm using it as a way to average over a large number of samples that are
sampled equidistantly (kernel density is way slower obviously).  The
documentation doesn't really tell how it is implemented or supposed to
work, so I've experimented a bit to find out.

> If there is a second column of data this is interpreted as a weight.

Let's assume the sample spacing is 1 and no samples are missing, then
you'll get the sum over the bin width.  Different sampling density just
scales the result.  Dividing by the number of samples gives you the
average.

> To show the distribution of values for f(x) over a given range of x, 
> the plotting command could be
>
>       plot sample [xmin:xmax:increment] '+' using (f(x)) bins {binwidth=foo}
>
> If I rewrite your test case in this form, shown as a histogram, it becomes
>
>       plot sample [0:100:1] '+' using (sin(x)) bins binwidth=0.1 with boxes
>
> The autoscaling works correctly on both x and y, and remains correct if 
> I plot instead "with lines". Note that this is a single column of data, so
> the samples have uniform weights.

We've already established that "with boxes" and "with impulses" would
clip the plotted data correctly, but both "with points" and "with lines"
doesn't.  That's the only thing I think is a bug and needs fixing.

> Your test case creates this same set of samples but stores it in a 
> data block with two columns   x and f(x).   To reproduce the plot from
> the stored samples the command would be
>
>      plot $data using 2 bins binwidth=0.1

That's different from what I'm trying to get, though; and it doesn't
reproduce the data either.  It does produce a distribution of the
samples, resolved into 0.1 wide buckets.

> Your test script provides no "using" specifier, however, so the plot
> command draws values from column 1 (essentially the numbers 0 to 100)
> and weights each one by the value in the second column (sin(x)).

It doesn't do that either or the result would be a linearly rising
function with a sin riding on top of it.  It reproduces the function if
I arrange the binwidth to contain just a single sample, so clearly the x
column isn't used directly.  It integrates over the bin if I arrange the
bin to contain a larger number of samples (10 in my example).  So if I
divide the result by the number of samples I get an average over the
bin.

> The presence of negative weights in particular confuses the program.
> I don't rule out the possibility that there is a legitimate use for such
> but the code assumes positive weights, typically either 1 or a set of
> fractional values summing to 1 so that the distribution is normalized.
> Also since the range of values is [0:100], using a bin width of 0.1 means
> that the vast majority of bins are empty.
> This produces the strange-looking plot you saw.

The plot is the result of most of the bins having an integral either
far above 0.1 or below -0.1, so it should have been clipped at the axis
borders, but it wasn't.

> If your test case is intended as a simple stand-in for some real world
> data that does in fact use negative weights, please clarify the
> expected properties of the resulting distribution and I will try to
> modify the program's expectations to allow it.  As it stands, the 
> programs expectations about y values lead it to totally mis-assign
> all the points as "inrange" and therefore not clipped.

Again, I'm not using it this way and the actual code doesn't seem to
work the way you described it either.  Now that you've described how to
get a binned distribution/histogram I do have uses for that too, but that's
not what I was having problems with.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Wavetables for the Waldorf Blofeld:
http://Synth.Stromeko.net/Downloads.html#BlofeldUserWavetables

Re: [bug] plot bins does not clip output to plot area

[Ethan A M. <me...@uw...>]

I am confused as to what exactly you are trying to plot.
The intent of the "bins" option is similar to the "smooth frequency" option.
It derives an approximation to the distribution of values read from
a single column of data.
If there is a second column of data this is interpreted as a weight.

To show the distribution of values for f(x) over a given range of x,
the plotting command could be

      plot sample [xmin:xmax:increment] '+' using (f(x)) bins {binwidth=foo}

If I rewrite your test case in this form, shown as a histogram, it becomes

      plot sample [0:100:1] '+' using (sin(x)) bins binwidth=0.1 with boxes

The autoscaling works correctly on both x and y, and remains correct if
I plot instead "with lines". Note that this is a single column of data, so
the samples have uniform weights.

Your test case creates this same set of samples but stores it in a
data block with two columns   x and f(x).   To reproduce the plot from
the stored samples the command would be

     plot $data using 2 bins binwidth=0.1

Your test script provides no "using" specifier, however, so the plot
command draws values from column 1 (essentially the numbers 0 to 100)
and weights each one by the value in the second column (sin(x)).
The presence of negative weights in particular confuses the program.
I don't rule out the possibility that there is a legitimate use for such
but the code assumes positive weights, typically either 1 or a set of
fractional values summing to 1 so that the distribution is normalized.
Also since the range of values is [0:100], using a bin width of 0.1 means
that the vast majority of bins are empty.
This produces the strange-looking plot you saw.

If your test case is intended as a simple stand-in for some real world
data that does in fact use negative weights, please clarify the
expected properties of the resulting distribution and I will try to
modify the program's expectations to allow it.  As it stands, the
programs expectations about y values lead it to totally mis-assign
all the points as "inrange" and therefore not clipped.

       Ethan

Re: [bug] plot bins does not clip output to plot area

[Achim G. <Str...@ne...>]

Ethan A Merritt writes:
> The y clipping seems to fail specifically when plotting "with lines".
> It clips properly for "with impulses" or "with boxes", right?

I haven't tried these two specifically, but "with points" also plots
outside the boundary.  OK, impulses and boxes are correctly clipped.
Things aer more clearly visible if you additionally

 set tmargin at screen 0.8
 set bmargin at screen 0.2

in my reproducer before the plot command.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptations for KORG EX-800 and Poly-800MkII V0.9:
http://Synth.Stromeko.net/Downloads.html#KorgSDada

Re: [bug] plot bins does not clip output to plot area

[Ethan A M. <me...@uw...>]

On Wednesday, 9 September 2020 22:36:56 PDT ASSI wrote:
> Ethan A Merritt writes:
> > The program extends auto-scaled ranges to the next axis tic unless
> > you tell it not to.  If you do not want the range to be extended, tell it
> 
> The y axis is not autoscaled in my example and there is no clipping at
> all, which gets obvious if you do multiplots.

Ah, sorry.  I misunderstood.

The y clipping seems to fail specifically when plotting "with lines".
It clips properly for "with impulses" or "with boxes", right?

	Ethan

> 
> 
> Regards,
> Achim.
> 

Re: [bug] plot bins does not clip output to plot area

[ASSI <Str...@ne...>]

Ethan A Merritt writes:
> The program extends auto-scaled ranges to the next axis tic unless
> you tell it not to.  If you do not want the range to be extended, tell it

The y axis is not autoscaled in my example and there is no clipping at
all, which gets obvious if you do multiplots.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptations for Waldorf Q V3.00R3 and Q+ V3.54R2:
http://Synth.Stromeko.net/Downloads.html#WaldorfSDada

Re: [bug] plot bins does not clip output to plot area

[Ethan A M. <me...@uw...>]

On Wednesday, 9 September 2020 11:05:16 PDT Achim Gratz wrote:
> 
> To reproduce:
> 
> gnuplot> set table $data
> gnuplot> plot [0:100] sin(x)
> gnuplot> unset table
> gnuplot> plot [*:*][-0.1:0.1] $data w lines bins binwidth=0.1
> 
> 
> Regards,
> Achim.

The program extends auto-scaled ranges to the next axis tic unless
you tell it not to.  If you do not want the range to be extended, tell it

    set xrange noextend       #version 5.4
or
    set xrange [*:*] noextend #version 5.2

	cheers,

		Ethan
 

[bug] plot bins does not clip output to plot area

[Achim G. <Str...@ne...>]

To reproduce:

gnuplot> set table $data
gnuplot> plot [0:100] sin(x)
gnuplot> unset table
gnuplot> plot [*:*][-0.1:0.1] $data w lines bins binwidth=0.1


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Factory and User Sound Singles for Waldorf rackAttack:
http://Synth.Stromeko.net/Downloads.html#WaldorfSounds

Re: Strange autoscaling with rgbimage and inverted yranges

[Ethan A M. <me...@uw...>]

On Wednesday, 9 September 2020 02:07:50 PDT Dima Kogan wrote:
> Hi. I vaguely recall mentioning this already, but I don't see it in my
> email. Apologies if this has already been reported. I have this script:
> 
>     set yrange [:] reverse
>     set size ratio -1
>     plot "/tmp/gray.jpg" binary filetype=auto flipy with rgbimage,'-'   using 1:2 notitle with points pt 3 ps 3 
[snip]
>     e
> 
> I.e. we plot some points on top of the attached image. The contents of
> the image don't matter. Here we just have a gray field. I'm using the
> latest master branch, but I belive 5.4 does this too.
> 
> 1. Running that script in an interactive terminal (I tried x11 and qt),
>    the plot pops up just fine: the autoscale considered the image AND the
>    points
> 
> 2. We interactively zoom on any area with the mouse. This works too
> 
> 3. We press 'u' to zoom out. The expectation is that we get back to the
>    full view after step 1. Instead we get to a new view, zoomed-in on
>    the points only, NOT on the image.

Hmm.  
But if you press 'p' for "previous" the unzoom works correctly.
That is probably a clue.

Also, if the inline data is replaced by a datablock the problem goes away.

I think the issue is that because your example uses in-line data it
cannot be re-read.  The program tries to work around this by reusing
the contents of the various internal data structures but in this case the
workaround fails.  Possibly fixable, but I think the primary lesson is
that data blocks are much preferred to in-line data if you are going to be
manipulating/re-drawing the plot.

	Ethan

> Something about the reverse yrange is confusing I think. Have I reported
> this already?
> 
> Thanks.