gnuplot-beta — Developers and beta testers of new gnuplot code

Strange autoscaling with rgbimage and inverted yranges

[Dima K. <gn...@di...>]

Hi. I vaguely recall mentioning this already, but I don't see it in my
email. Apologies if this has already been reported. I have this script:

    set yrange [:] reverse
    set size ratio -1
    plot "/tmp/gray.jpg" binary filetype=auto flipy with rgbimage,'-'   using 1:2 notitle with points pt 3 ps 3 
    1854.0 1599.0
    1596.0 1568.0
    2170.0 1626.0
    3649.0 1762.0
    1937.0 1604.0
    1243.0 1544.0
    2412.0 1648.0
    886.0 1508.0
    141.0 1436.0
    3088.0 1709.0
    292.0 1455.0
    2311.0 1639.0
    647.0 1484.0
    2662.0 1672.0
    1105.0 1529.0
    803.0 1499.0
    4218.0 1811.0
    731.0 1498.0
    3743.0 1769.0
    1371.0 1555.0
    4100.0 1800.0
    1446.0 1544.0
    2092.0 1587.0
    2863.0 1669.0
    2930.0 1681.0
    3197.0 1702.0
    3341.0 1709.0
    3483.0 1733.0
    e

I.e. we plot some points on top of the attached image. The contents of
the image don't matter. Here we just have a gray field. I'm using the
latest master branch, but I belive 5.4 does this too.

1. Running that script in an interactive terminal (I tried x11 and qt),
   the plot pops up just fine: the autoscale considered the image AND the
   points

2. We interactively zoom on any area with the mouse. This works too

3. We press 'u' to zoom out. The expectation is that we get back to the
   full view after step 1. Instead we get to a new view, zoomed-in on
   the points only, NOT on the image.

Something about the reverse yrange is confusing I think. Have I reported
this already?

Thanks.

Re: 5.4.rc2 second release candidate for gnuplot version 5.4

[Shigeharu T. <sh...@ie...>]

shige 08/28 2020
----------------

Tait wrote at 2020-06-09: 

| I ran gp54rc1-win64-mingw.exe downloaded from SF.
...
| I wanted to try the Windows terminal, too, but it hangs for a
| few seconds and then crashes, seemingly no matter what I try to
| plot.

Currently, I can't download the binary from SF. 

Well, I build windows binary from gnuplot source version 5.4.0 by
msys2-mingw64, and I ran it with wxt terminal as 
  gnuplot.exe -e "set term wxt; cd 'demo' ; load 'all.dem'"
then it works fine. 
For windows terminal, certainly
  gnuplot.exe -e "set term win; cd 'demo' ; load 'all.dem'"
crashes as reported. 
But, when I specify the window size explicitly as
  gnuplot.exe -e "set term win size 640,400; cd 'demo' ; load 'all.dem'"
it works fine. Please try.

+========================================================+
 Shigeharu TAKENO     NIigata Institute of Technology
                       kashiwazaki,Niigata 945-1195 JAPAN
 sh...@ie...   TEL(&FAX): +81-257-22-8161
+========================================================+

Re: [PATCH] cairolatex: respect transparency setting

[Henri M. <hen...@gm...>]

On 20/08/20, 21:31, Ethan A Merritt wrote:
> On Thursday, 20 August 2020 20:17:44 PDT Henri Menke wrote:
> > On 20/08/20, 20:01, Ethan A Merritt wrote:
> > > On Thursday, 20 August 2020 18:34:54 PDT Henri Menke wrote:
> > > > Currently the cairolatex terminal does not respect transparency options
> > > > given in the terminal setting.  In this minimal example the
> > > > notransparent option is ignored:
> > > > 
> > > >     set terminal cairolatex png notransparent
> > > >     set output "test.tex"
> > > >     plot sin(x)
> > > > 
> > > > Running this through gnuplot and then using `file' reveals that the
> > > > alpha channel is still present:
> > > > 
> > > >     $ gnuplot test.gnuplot && file test.png
> > > >     test.png: PNG image data, 1500 x 900, 8-bit/color RGBA, non-interlace
> > > 
> > > I think the raionale for the current code is that the png image created for
> > > inclusion in a LaTeX document must have a transparent background in order
> > > to support the "back" and "behind" attributes for text elements, since they
> > > will be drawn by the latex engine and then covered by inclusion of the
> > > graphics image.  The structure of the latex processing is:
> > > 
> > >     render text elements that are "behind" or "back"
> > >     \includegraphics{gnuplot_figure}
> > >     render text elements that are "front"
> > > 
> > > If you use a solid color background in the graphics image, it will hide
> > > any previously rendered text elements.
> > 
> > The reason why I'm looking into this is that currently it is impossible
> > to generate PDF/A when using plots generated by gnuplot.  PDF/A doesn't
> > allow transparency, so I'm trying to remove all of it (or at least make
> > it optional).  As noted in the patch, the default behaviour of using
> > transparency remains, so this is entirely optional.
> 
> I am researching this as I type, so please bear with me.
> 
> From Wikipedia I see
> 	Transparent objects and layers (Optional Content Groups) are 
> 	forbidden in PDF/A-1, but are allowed in PDF/A-2.
> PDF/A-1 was published in 2005; PDF/A-2 in 2011; PDF/A-3 in 2012;
> So maybe this is no longer a limitation?
> 
> From my own knowledge, there are commonly available tools that 
> render the transparency in a PDF document once, then save the result
> as an embedded image that no longer contains transparency.
> For example if you use gnuplot to create a PostScript file directly it cannot
> use transparency because PostScript itself does not support transparency.
> But if you use gnuplot to create a PDF file with transparency and then
> convert it
> 	pdf2ps input.pdf output.ps
> you get a PostScript file in which the transparency is "baked in" rather
> than interpreted.   Logically then you could take that PostScript file
> and convert it back to PDF, keeping the pre-rendered image that no
> longer uses an alpha channel. 
> 	ps2pdf output.ps new.pdf
> Whether that particular pair of tools produces a PDF/A compliant
> file I do not know.

This is actually a pretty good proposal.  In my actual plot I now have
added

    set terminal cairolatex pdf standalone header '\usepackage[a-1b]{pdfx}'
    set output "test.tex"
    # ...
    unset output
    !pdf2ps test-inc.pdf test-inc.ps
    !ps2pdf -dPDFA=1 test-inc.ps test-inc.pdf
    !pdflatex --interaction=nonstopmode test.tex

and the resulting document validates correctly with VeraPDF.  It's a bit
ugly and I don't know whether the ghostscript roundtrip would correctly
preserve fonts, but since the background image doesn't contain fonts, I
don't care.  Note that adding the -dPDFA=1 flag will make the PDF
structurally PDF/A compliant, but will not add metadata or color
profiles, so test-inc.pdf does not validate.  These extras are handled
by the pdfx LaTeX package.

To get my plots with PNG background compliant I use the patch from this
thread together with your suggestion `set tics front'.  Another option
would be to run mogrify to remove the transparency.

    set terminal cairolatex png standalone header '\usepackage[a-1b]{pdfx}'
    set output "test.tex"
    # ...
    unset output
    !mogrify -alpha remove test-inc.png
    !pdflatex --interaction=nonstopmode test.tex

So bottomlined my thesis now validates as PDF/A-1b, which is really
nice.

Cheers, Henri

> 
>    best,
> 
> 	Ethan
> 
> > 
> > Kind regards,
> > Henri
> > 
> > > 
> > > 	Ethan
> > > 
> > > > 
> > > > With this patch the transparency setting will be honoured.  The current
> > > > default setting of implicitly assuming transparency is unaltered.
> > > > ---
> > > >  term/cairo.trm | 2 +-
> > > >  1 file changed, 1 insertion(+), 1 deletion(-)
> > > > 
> > > > diff --git a/term/cairo.trm b/term/cairo.trm
> > > > index ee748dfe2..3ef76ecc6 100644
> > > > --- a/term/cairo.trm
> > > > +++ b/term/cairo.trm
> > > > @@ -849,7 +849,7 @@ void cairotrm_graphics()
> > > >  	plot.background.g = cairo_params->background.g;
> > > >  	plot.background.b = cairo_params->background.b;
> > > >  	gp_cairo_set_background(cairo_params->background);
> > > > -	if (ISCAIROLATEX || cairo_params->transparent)
> > > > +	if (cairo_params->transparent)
> > > >  		gp_cairo_clear_background(&plot);
> > > >  	else
> > > >  		gp_cairo_solid_background(&plot);
> > > > 
> > > 
> > > 
> > > 
> > > 
> 
> 
> 

Re: [BUG] pdfcairo notransparent has transparency

[Ethan A M. <me...@uw...>]

On Thursday, 20 August 2020 21:42:51 PDT Henri Menke wrote:
> Dear all,
> 
> With regards to my recent patch to optionally disable transparency on
> cairolatex, I stumbled upon the fact that the PDF output of the cairo
> terminal does not properly respect the notransparent option.  Consider
> the following minimal example:
> 
>     set terminal pdfcairo notransparent
>     set output "test.pdf"
>     plot sin(x)
> 
> When running this, the page is still encapsulated in a transparency
> group.  

The "{no}transparent" option used by several gnuplot terminals
specifically refers to the plot background.  It does not, and was never
intended to, disable transparency in the plot proper.   The two things
are orthogonal.   Consider for example
 
    set term png notruecolor transparent 

This sets the background to be transparent but the plot contains no
alpha channel because it uses indexed color. 
Contrast this with

    set term png truecolor backgound "grey"

This sets the background to be opaque (=notransparent),
but the plot itself uses 24bit RGB color + alpha channel.

	Ethan



> This breaks PDF/A compatibility.
> 
>     $ gnuplot test.gnuplot
>     $ grep -aC6 'Transparency' test.pdf
>     << /Type /Page % 1
>        /Parent 1 0 R
>        /MediaBox [ 0 0 360 216 ]
>        /Contents 4 0 R
>        /Group <<
>           /Type /Group
>           /S /Transparency
>           /I true
>           /CS /DeviceRGB
>        >>
>        /Resources 3 0 R
>     >>
>     endobj
> 
> I'm not entire sure and I haven't tested this yet, but the cairo
> terminal uses the gp_cairo_set_color function from
> src/wxterminal/gp_cairo.c which in turn calls cairo_set_source_rgba.
> This explicit call for transparency might be the cause for cairo
> inserting a transparency group, even if all alpha values are zero or
> one.
> 
> Cheers, Henri
> 

[BUG] pdfcairo notransparent has transparency

[Henri M. <hen...@gm...>]

Dear all,

With regards to my recent patch to optionally disable transparency on
cairolatex, I stumbled upon the fact that the PDF output of the cairo
terminal does not properly respect the notransparent option.  Consider
the following minimal example:

    set terminal pdfcairo notransparent
    set output "test.pdf"
    plot sin(x)

When running this, the page is still encapsulated in a transparency
group.  This breaks PDF/A compatibility.

    $ gnuplot test.gnuplot
    $ grep -aC6 'Transparency' test.pdf
    << /Type /Page % 1
       /Parent 1 0 R
       /MediaBox [ 0 0 360 216 ]
       /Contents 4 0 R
       /Group <<
          /Type /Group
          /S /Transparency
          /I true
          /CS /DeviceRGB
       >>
       /Resources 3 0 R
    >>
    endobj

I'm not entire sure and I haven't tested this yet, but the cairo
terminal uses the gp_cairo_set_color function from
src/wxterminal/gp_cairo.c which in turn calls cairo_set_source_rgba.
This explicit call for transparency might be the cause for cairo
inserting a transparency group, even if all alpha values are zero or
one.

Cheers, Henri

Re: [PATCH] cairolatex: respect transparency setting

[Ethan A M. <me...@uw...>]

On Thursday, 20 August 2020 20:17:44 PDT Henri Menke wrote:
> On 20/08/20, 20:01, Ethan A Merritt wrote:
> > On Thursday, 20 August 2020 18:34:54 PDT Henri Menke wrote:
> > > Currently the cairolatex terminal does not respect transparency options
> > > given in the terminal setting.  In this minimal example the
> > > notransparent option is ignored:
> > > 
> > >     set terminal cairolatex png notransparent
> > >     set output "test.tex"
> > >     plot sin(x)
> > > 
> > > Running this through gnuplot and then using `file' reveals that the
> > > alpha channel is still present:
> > > 
> > >     $ gnuplot test.gnuplot && file test.png
> > >     test.png: PNG image data, 1500 x 900, 8-bit/color RGBA, non-interlace
> > 
> > I think the raionale for the current code is that the png image created for
> > inclusion in a LaTeX document must have a transparent background in order
> > to support the "back" and "behind" attributes for text elements, since they
> > will be drawn by the latex engine and then covered by inclusion of the
> > graphics image.  The structure of the latex processing is:
> > 
> >     render text elements that are "behind" or "back"
> >     \includegraphics{gnuplot_figure}
> >     render text elements that are "front"
> > 
> > If you use a solid color background in the graphics image, it will hide
> > any previously rendered text elements.
> 
> The reason why I'm looking into this is that currently it is impossible
> to generate PDF/A when using plots generated by gnuplot.  PDF/A doesn't
> allow transparency, so I'm trying to remove all of it (or at least make
> it optional).  As noted in the patch, the default behaviour of using
> transparency remains, so this is entirely optional.

I am researching this as I type, so please bear with me.

>From Wikipedia I see
	Transparent objects and layers (Optional Content Groups) are 
	forbidden in PDF/A-1, but are allowed in PDF/A-2.
PDF/A-1 was published in 2005; PDF/A-2 in 2011; PDF/A-3 in 2012;
So maybe this is no longer a limitation?

>From my own knowledge, there are commonly available tools that 
render the transparency in a PDF document once, then save the result
as an embedded image that no longer contains transparency.
For example if you use gnuplot to create a PostScript file directly it cannot
use transparency because PostScript itself does not support transparency.
But if you use gnuplot to create a PDF file with transparency and then
convert it
	pdf2ps input.pdf output.ps
you get a PostScript file in which the transparency is "baked in" rather
than interpreted.   Logically then you could take that PostScript file
and convert it back to PDF, keeping the pre-rendered image that no
longer uses an alpha channel. 
	ps2pdf output.ps new.pdf
Whether that particular pair of tools produces a PDF/A compliant
file I do not know.

   best,

	Ethan

> 
> Kind regards,
> Henri
> 
> > 
> > 	Ethan
> > 
> > > 
> > > With this patch the transparency setting will be honoured.  The current
> > > default setting of implicitly assuming transparency is unaltered.
> > > ---
> > >  term/cairo.trm | 2 +-
> > >  1 file changed, 1 insertion(+), 1 deletion(-)
> > > 
> > > diff --git a/term/cairo.trm b/term/cairo.trm
> > > index ee748dfe2..3ef76ecc6 100644
> > > --- a/term/cairo.trm
> > > +++ b/term/cairo.trm
> > > @@ -849,7 +849,7 @@ void cairotrm_graphics()
> > >  	plot.background.g = cairo_params->background.g;
> > >  	plot.background.b = cairo_params->background.b;
> > >  	gp_cairo_set_background(cairo_params->background);
> > > -	if (ISCAIROLATEX || cairo_params->transparent)
> > > +	if (cairo_params->transparent)
> > >  		gp_cairo_clear_background(&plot);
> > >  	else
> > >  		gp_cairo_solid_background(&plot);
> > > 
> > 
> > 
> > 
> > 

Re: [PATCH] cairolatex: respect transparency setting

[Henri M. <hen...@gm...>]

On 20/08/20, 20:01, Ethan A Merritt wrote:
> On Thursday, 20 August 2020 18:34:54 PDT Henri Menke wrote:
> > Currently the cairolatex terminal does not respect transparency options
> > given in the terminal setting.  In this minimal example the
> > notransparent option is ignored:
> > 
> >     set terminal cairolatex png notransparent
> >     set output "test.tex"
> >     plot sin(x)
> > 
> > Running this through gnuplot and then using `file' reveals that the
> > alpha channel is still present:
> > 
> >     $ gnuplot test.gnuplot && file test.png
> >     test.png: PNG image data, 1500 x 900, 8-bit/color RGBA, non-interlace
> 
> I think the raionale for the current code is that the png image created for
> inclusion in a LaTeX document must have a transparent background in order
> to support the "back" and "behind" attributes for text elements, since they
> will be drawn by the latex engine and then covered by inclusion of the
> graphics image.  The structure of the latex processing is:
> 
>     render text elements that are "behind" or "back"
>     \includegraphics{gnuplot_figure}
>     render text elements that are "front"
> 
> If you use a solid color background in the graphics image, it will hide
> any previously rendered text elements.

The reason why I'm looking into this is that currently it is impossible
to generate PDF/A when using plots generated by gnuplot.  PDF/A doesn't
allow transparency, so I'm trying to remove all of it (or at least make
it optional).  As noted in the patch, the default behaviour of using
transparency remains, so this is entirely optional.

Kind regards,
Henri

> 
> 	Ethan
> 
> > 
> > With this patch the transparency setting will be honoured.  The current
> > default setting of implicitly assuming transparency is unaltered.
> > ---
> >  term/cairo.trm | 2 +-
> >  1 file changed, 1 insertion(+), 1 deletion(-)
> > 
> > diff --git a/term/cairo.trm b/term/cairo.trm
> > index ee748dfe2..3ef76ecc6 100644
> > --- a/term/cairo.trm
> > +++ b/term/cairo.trm
> > @@ -849,7 +849,7 @@ void cairotrm_graphics()
> >  	plot.background.g = cairo_params->background.g;
> >  	plot.background.b = cairo_params->background.b;
> >  	gp_cairo_set_background(cairo_params->background);
> > -	if (ISCAIROLATEX || cairo_params->transparent)
> > +	if (cairo_params->transparent)
> >  		gp_cairo_clear_background(&plot);
> >  	else
> >  		gp_cairo_solid_background(&plot);
> > 
> 
> 
> 
> 

Re: [PATCH] cairolatex: respect transparency setting

[Ethan A M. <me...@uw...>]

On Thursday, 20 August 2020 19:34:40 PDT Henri Menke wrote:
> Now there seems to a problem with ordering of the elements in the
> resulting TeX file.  The ticklabels are behind the background image.
> Even if transparency is enabled, this is probably a bad idea.  I'll
> investigate.

That is controlled by a generic setting, not specific to the latex terminals:

    set tics {front|back}

The default is "back".

	Ethan


> 
> Cheers, Henri
> 
> On 21/08/20, 13:34, Henri Menke wrote:
> > Currently the cairolatex terminal does not respect transparency options
> > given in the terminal setting.  In this minimal example the
> > notransparent option is ignored:
> > 
> >     set terminal cairolatex png notransparent
> >     set output "test.tex"
> >     plot sin(x)
> > 
> > Running this through gnuplot and then using `file' reveals that the
> > alpha channel is still present:
> > 
> >     $ gnuplot test.gnuplot && file test.png
> >     test.png: PNG image data, 1500 x 900, 8-bit/color RGBA, non-interlace
> > 
> > With this patch the transparency setting will be honoured.  The current
> > default setting of implicitly assuming transparency is unaltered.
> > ---
> >  term/cairo.trm | 2 +-
> >  1 file changed, 1 insertion(+), 1 deletion(-)
> > 
> > diff --git a/term/cairo.trm b/term/cairo.trm
> > index ee748dfe2..3ef76ecc6 100644
> > --- a/term/cairo.trm
> > +++ b/term/cairo.trm
> > @@ -849,7 +849,7 @@ void cairotrm_graphics()
> >  	plot.background.g = cairo_params->background.g;
> >  	plot.background.b = cairo_params->background.b;
> >  	gp_cairo_set_background(cairo_params->background);
> > -	if (ISCAIROLATEX || cairo_params->transparent)
> > +	if (cairo_params->transparent)
> >  		gp_cairo_clear_background(&plot);
> >  	else
> >  		gp_cairo_solid_background(&plot);
> 

Re: [PATCH] cairolatex: respect transparency setting

[Ethan A M. <me...@uw...>]

On Thursday, 20 August 2020 18:34:54 PDT Henri Menke wrote:
> Currently the cairolatex terminal does not respect transparency options
> given in the terminal setting.  In this minimal example the
> notransparent option is ignored:
> 
>     set terminal cairolatex png notransparent
>     set output "test.tex"
>     plot sin(x)
> 
> Running this through gnuplot and then using `file' reveals that the
> alpha channel is still present:
> 
>     $ gnuplot test.gnuplot && file test.png
>     test.png: PNG image data, 1500 x 900, 8-bit/color RGBA, non-interlace

I think the raionale for the current code is that the png image created for
inclusion in a LaTeX document must have a transparent background in order
to support the "back" and "behind" attributes for text elements, since they
will be drawn by the latex engine and then covered by inclusion of the
graphics image.  The structure of the latex processing is:

    render text elements that are "behind" or "back"
    \includegraphics{gnuplot_figure}
    render text elements that are "front"

If you use a solid color background in the graphics image, it will hide
any previously rendered text elements.

	Ethan

> 
> With this patch the transparency setting will be honoured.  The current
> default setting of implicitly assuming transparency is unaltered.
> ---
>  term/cairo.trm | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/term/cairo.trm b/term/cairo.trm
> index ee748dfe2..3ef76ecc6 100644
> --- a/term/cairo.trm
> +++ b/term/cairo.trm
> @@ -849,7 +849,7 @@ void cairotrm_graphics()
>  	plot.background.g = cairo_params->background.g;
>  	plot.background.b = cairo_params->background.b;
>  	gp_cairo_set_background(cairo_params->background);
> -	if (ISCAIROLATEX || cairo_params->transparent)
> +	if (cairo_params->transparent)
>  		gp_cairo_clear_background(&plot);
>  	else
>  		gp_cairo_solid_background(&plot);
> 

Re: [PATCH] cairolatex: respect transparency setting

[Henri M. <hen...@gm...>]

Now there seems to a problem with ordering of the elements in the
resulting TeX file.  The ticklabels are behind the background image.
Even if transparency is enabled, this is probably a bad idea.  I'll
investigate.

Cheers, Henri

On 21/08/20, 13:34, Henri Menke wrote:
> Currently the cairolatex terminal does not respect transparency options
> given in the terminal setting.  In this minimal example the
> notransparent option is ignored:
> 
>     set terminal cairolatex png notransparent
>     set output "test.tex"
>     plot sin(x)
> 
> Running this through gnuplot and then using `file' reveals that the
> alpha channel is still present:
> 
>     $ gnuplot test.gnuplot && file test.png
>     test.png: PNG image data, 1500 x 900, 8-bit/color RGBA, non-interlace
> 
> With this patch the transparency setting will be honoured.  The current
> default setting of implicitly assuming transparency is unaltered.
> ---
>  term/cairo.trm | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/term/cairo.trm b/term/cairo.trm
> index ee748dfe2..3ef76ecc6 100644
> --- a/term/cairo.trm
> +++ b/term/cairo.trm
> @@ -849,7 +849,7 @@ void cairotrm_graphics()
>  	plot.background.g = cairo_params->background.g;
>  	plot.background.b = cairo_params->background.b;
>  	gp_cairo_set_background(cairo_params->background);
> -	if (ISCAIROLATEX || cairo_params->transparent)
> +	if (cairo_params->transparent)
>  		gp_cairo_clear_background(&plot);
>  	else
>  		gp_cairo_solid_background(&plot);
> -- 
> 2.28.0
> 

[PATCH] cairolatex: respect transparency setting

[Henri M. <hen...@gm...>]

Currently the cairolatex terminal does not respect transparency options
given in the terminal setting.  In this minimal example the
notransparent option is ignored:

    set terminal cairolatex png notransparent
    set output "test.tex"
    plot sin(x)

Running this through gnuplot and then using `file' reveals that the
alpha channel is still present:

    $ gnuplot test.gnuplot && file test.png
    test.png: PNG image data, 1500 x 900, 8-bit/color RGBA, non-interlace

With this patch the transparency setting will be honoured.  The current
default setting of implicitly assuming transparency is unaltered.
---
 term/cairo.trm | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/term/cairo.trm b/term/cairo.trm
index ee748dfe2..3ef76ecc6 100644
--- a/term/cairo.trm
+++ b/term/cairo.trm
@@ -849,7 +849,7 @@ void cairotrm_graphics()
 	plot.background.g = cairo_params->background.g;
 	plot.background.b = cairo_params->background.b;
 	gp_cairo_set_background(cairo_params->background);
-	if (ISCAIROLATEX || cairo_params->transparent)
+	if (cairo_params->transparent)
 		gp_cairo_clear_background(&plot);
 	else
 		gp_cairo_solid_background(&plot);
-- 
2.28.0

Re: Sideways violin plots in one plot cmd?

[Dima K. <gn...@di...>]

Ethan A Merritt <me...@uw...> writes:

> I guess I still don't get it.
> Since the wrapper can pass in any necessary set of commands,
> why not just pass in the commands that do the double plot?

Because my tools would then need special-case logic for this one case. I
would need some sort of "feedgnuplot --violin" and logic internally to
handle it. At the very start I decided that explicitly supporting
everything gnuplot supports with such special-case logic would be
extremely effortful and extremely error-prone and maintenance-full. And
I avoided the issue entirely by passing user-specified strings to
gnuplot transparently. Which works great for most styles.

If (as a random example) I have 3-column input that I want to plot as
errorbars I can do this:

  < 3columns_datafile feedgnuplot --domain --tuplesizeall 3 --with yerrorbars

feegnuplot then knows to be dealing with 3-values-per-point data, and
tells gnuplot to plot with "yerrorbars". There's no special-case logic
for "yerrorbars". And I don't want to create a separate path for violin
plots.

Full disclosure: there IS one special-case path to deal with histograms.
But those are extremely useful, and worth violating the general design.
violin plots aren't nearly as ubiquitous, and I dont want to add a big
special-case path for them.

Re: Sideways violin plots in one plot cmd?

[Juhász P. <pet...@gm...>]

On Sat, 2020-08-15 at 13:17 -0700, Ethan A Merritt wrote:
> On Friday, 14 August 2020 10:34:52 PDT Juhász Péter wrote:
> > On Wed, 2020-08-05 at 23:46 -0700, Dima Kogan wrote:
> > > A question. I'm looking at the violin plot demo page:
> > > 
> > >   http://gnuplot.sourceforge.net/demo/violinplot.html
> > > 
> > > It shows how to make a horizontal density plot using 'smooth
> > > kdensity'.
> > > 
> > > Then it shows to to make a vertical density plot by plotting
> > > horizontally into a table, and then plotting the table, with
> > > "using"
> > > directives used to swap the axes. This works clearly, but is
> > > there a
> > > way
> > > to do this (vertical density plots) with a single plot command?
> > > That
> > > would make my life much easier.
> > > 
> > > Thanks!
> > > 
> > > 
> > > 
> > 
> > Well, recently at $work I used gnuplot to produce a sideways
> > histogram.
> > Because it had to be sideways, I had to resort to manually
> > assembling
> > the plot with the boxxyerror style, with an awkward preprocessing
> > step.
> > 
> > This, and your question, gives me the idea that it might be worth
> > adding a general 'transpose' option to some of the "clever" plot
> > styles
> > that produce non-trivial vertical plots (histograms and boxplots,
> > mostly).
> > 
> > best regards,
> > Peter Juhasz
> 
> "How to make a sideways plot" does seem to be a very
> frequently asked question.  There may be some clever way to select
> an x/y axis transposition in the general case, but if so I have not
> been
> able to find it.  
> 

I might have mislead the discussion with my suggestion, because as it
turns out, the only common thing it has with the original violin plot
question is "would there be a way to do X in a single command". Sorry
for the confusion.

That said, I don't think bringing in 3D (s)plot styles is a good
direction towards a general solution for sideways plots. To me, it
sounds like looking for our lost car keys in the corner only because
that's where there is light.

> The program was designed from the start to have a single 2D plot mode
> with x always horizontal, [...]

Yes, originally it was designed like that, but then more complex plot
styles like histograms and boxplots were added, which do non-trivial
preprocessing and aggregating steps before plotting the data. For
boxplots (and less acutely, even for histograms) the x axis of the plot
is no longer directly maps to any column of the original data.

> Peter - would the plot you constructed
> by hand be a good example to adapt?  If the combination
>     set view projection xz
>     splot ... with boxerror
> doesn't handle it adequately, that would highlight where it would
> be worth some additional work on the code. 

No, not really, unless I misunderstand you - my point was that I wanted
to produce (IIRC) a columnstacked histogram, and if it could have been
a vertical plot, I could have used `set style histogram columnstacked;
set style data histogram`, with all the nice intuitive data layout and
all the automatic box placement that style provides. But the
requirements called for a horizontal plot, so I had to do all that by
hand. Doing the same in 3D wouldn't have helped.

Peter

> 
> 	Ethan
> 
> 

Re: Sideways violin plots in one plot cmd?

[Ethan A M. <me...@uw...>]

On Saturday, 15 August 2020 22:06:34 PDT Dima Kogan wrote:
> Ethan A Merritt <me...@uw...> writes:
> >
> > I don't get it. What is it about the shell interface that would prevent you
> > from doing exactly what is in violinplot.dem?
> >   set table $datablock 
> >   plot ...
> >   unset table
> >   plot $datablock ...
> 
> Both gnuplotlib and feedgnuplot work as very thin wrappers around
> gnuplot, where they pass strings directly to gnuplot without knowing
> what the strings mean. This allows those interfaces to support lots and
> lots of styles, while containing no code that makes those styles work:
> all the logic is inside gnuplot itself. The double-plotting in
> violinplot.dem is qualitatively different than pretty much every other
> gnuplot style, so I would need to add special-case support for it. I
> COULD do that, but I really don't want to: the really simple API of
> feedgnuplot and gnuplotlib is a feature, and this special-case path
> would dilute that feature.
> 
> In a perfect world we'd be able to make violin plots like all others:
> with some "set" commands (that don't have the data in it) and a single
> plot/splot command that is given data. That is sounding like a big
> project inside gnuplot, though.


I guess I still don't get it.
Since the wrapper can pass in any necessary set of commands,
why not just pass in the commands that do the double plot?

Would it help to have a library of helper scripts for gnuplot?
Here's one that creates a vertical violin plot from a single gnuplot
command.  You could obviously add a lot of customization either
by passing in additional parameters or defining them globally.

   gnuplot> set term pngcairo
   gnuplot> set output 'violin.png'
   gnuplot> # unit width violin plot centered at x=10., linetype 3
   gnuplot> call 'violin.gp' 'filename.dat' 10. 3 

violin.gp script follows
%%%%%%%%%%%%%%%%%%%%%%%%
#
# invoke this script from inside gnuplot using
#       call violin.gp filename xcenter linetype
#
filename = ARGV[1]
xcenter = ARGV[2]
LT = ARGV[3]
#
set xrange [*:*]
set table $kdensity
plot filename using 2:(1) smooth kdensity with filledcurves above x1
unset table
stats $kdensity using 2 noout
plot [0:20][0:*]\
     $kdensity using (xcenter + column(2)/STATS_max):1 with filledcurves x=xcenter lt LT, \
     $kdensity using (xcenter - column(2)/STATS_max):1 with filledcurves x=xcenter lt LT
#
%%%%%%%%%%%%%%%%%%%%%%%%%

Re: Sideways violin plots in one plot cmd?

[Dima K. <gn...@di...>]

Ethan A Merritt <me...@uw...> writes:
>
> I don't get it. What is it about the shell interface that would prevent you
> from doing exactly what is in violinplot.dem?
>   set table $datablock 
>   plot ...
>   unset table
>   plot $datablock ...

Both gnuplotlib and feedgnuplot work as very thin wrappers around
gnuplot, where they pass strings directly to gnuplot without knowing
what the strings mean. This allows those interfaces to support lots and
lots of styles, while containing no code that makes those styles work:
all the logic is inside gnuplot itself. The double-plotting in
violinplot.dem is qualitatively different than pretty much every other
gnuplot style, so I would need to add special-case support for it. I
COULD do that, but I really don't want to: the really simple API of
feedgnuplot and gnuplotlib is a feature, and this special-case path
would dilute that feature.

In a perfect world we'd be able to make violin plots like all others:
with some "set" commands (that don't have the data in it) and a single
plot/splot command that is given data. That is sounding like a big
project inside gnuplot, though.

Re: Sideways violin plots in one plot cmd?

[Ethan A M. <me...@uw...>]

On Saturday, 15 August 2020 16:37:35 PDT Dima Kogan wrote:
> Ethan A Merritt <me...@uw...> writes:
> 
> > Are you trying to avoid the extra step of saving to an array or
> > datablock and then replotting? That seems like a totally different
> > question from vertical/horizontal layout or coordinate
> > transformations.
> 
> Yes. Avoiding that extra step is the question in this thread; see the
> subject. I've never tried to plot sideways histograms, like Peter did; I
> was assuming that this is simple if we plot into a datablock and then
> replot, but maybe not.
> 
> In any case, here's my problem. I use gnuplot A LOT. But 99% of that
> usage is via one of two wrappers (I wrote both of these):
> 
>   feedgnuplot (a nice shell interface)
>   gnuplotlib (a nice python interface)
> 
> Both of these work as thin wrappers around gnuplot, the idea being that
> with some "set" commands and a fancy "plot" command you can do anything.
> As it currently stands, violin plots are apparently not a part of this
> "anything". One could say that my wrappers are silly and not
> flexible-enough. Which maybe is true, but realistically speaking, I
> don't see ever expanding these tools to use datablocks, since that would
> break all sorts of interface assumptions both of these tools are making.

I don't get it. What is it about the shell interface that would prevent you
from doing exactly what is in violinplot.dem?
  set table $datablock 
  plot ...
  unset table
  plot $datablock ...

	Ethan

Re: Sideways violin plots in one plot cmd?

[Dima K. <gn...@di...>]

Ethan A Merritt <me...@uw...> writes:

> Are you trying to avoid the extra step of saving to an array or
> datablock and then replotting? That seems like a totally different
> question from vertical/horizontal layout or coordinate
> transformations.

Yes. Avoiding that extra step is the question in this thread; see the
subject. I've never tried to plot sideways histograms, like Peter did; I
was assuming that this is simple if we plot into a datablock and then
replot, but maybe not.

In any case, here's my problem. I use gnuplot A LOT. But 99% of that
usage is via one of two wrappers (I wrote both of these):

  feedgnuplot (a nice shell interface)
  gnuplotlib (a nice python interface)

Both of these work as thin wrappers around gnuplot, the idea being that
with some "set" commands and a fancy "plot" command you can do anything.
As it currently stands, violin plots are apparently not a part of this
"anything". One could say that my wrappers are silly and not
flexible-enough. Which maybe is true, but realistically speaking, I
don't see ever expanding these tools to use datablocks, since that would
break all sorts of interface assumptions both of these tools are making.
So I can live without violin plots, or maybe go use matplotlib. Which
I'd rather not do.

Any suggestions appreciated.

Re: Sideways violin plots in one plot cmd?

[Ethan A M. <me...@uw...>]

On Saturday, 15 August 2020 13:51:32 PDT Dima Kogan wrote:
> If we add better support for these, I think supporting more general
> transformations than just a simple x/y transpose would be useful. While
> thinking of violin plots I had a workaround idea: instead of a set of
> horizontally-spaced vertical density plots, I'd be fine with a set of
> vertically-spaced horizontal density plots. But I think gnuplot
> currently can't do that either: the density plots must be horizontal AND
> they must start at y=0. Right? So being able to support translations
> would be nice.

I may not understand what you want.
kdensity smoothing is a sum of Gaussians, so by definition the baseline
is zero.   That is not a limitation of how gnuplot plots things, that is
the definition of the function being plotted.    You can of course
save the resulting curve and plot it with an offset on y, just as the 
violin plot demo replots it vertically with an offset on x.

Are you trying to avoid the extra step of saving to an array or
datablock and then replotting?   That seems like a totally different
question from vertical/horizontal layout or coordinate transformations.

	Ethan

Re: Sideways violin plots in one plot cmd?

[Dima K. <gn...@di...>]

If we add better support for these, I think supporting more general
transformations than just a simple x/y transpose would be useful. While
thinking of violin plots I had a workaround idea: instead of a set of
horizontally-spaced vertical density plots, I'd be fine with a set of
vertically-spaced horizontal density plots. But I think gnuplot
currently can't do that either: the density plots must be horizontal AND
they must start at y=0. Right? So being able to support translations
would be nice.

Re: Sideways violin plots in one plot cmd?

[Ethan A M. <me...@uw...>]

On Friday, 14 August 2020 10:34:52 PDT Juhász Péter wrote:
> On Wed, 2020-08-05 at 23:46 -0700, Dima Kogan wrote:
> > A question. I'm looking at the violin plot demo page:
> > 
> >   http://gnuplot.sourceforge.net/demo/violinplot.html
> > 
> > It shows how to make a horizontal density plot using 'smooth
> > kdensity'.
> > 
> > Then it shows to to make a vertical density plot by plotting
> > horizontally into a table, and then plotting the table, with "using"
> > directives used to swap the axes. This works clearly, but is there a
> > way
> > to do this (vertical density plots) with a single plot command? That
> > would make my life much easier.
> > 
> > Thanks!
> > 
> > 
> > 
> 
> Well, recently at $work I used gnuplot to produce a sideways histogram.
> Because it had to be sideways, I had to resort to manually assembling
> the plot with the boxxyerror style, with an awkward preprocessing step.
> 
> This, and your question, gives me the idea that it might be worth
> adding a general 'transpose' option to some of the "clever" plot styles
> that produce non-trivial vertical plots (histograms and boxplots,
> mostly).
> 
> best regards,
> Peter Juhasz

"How to make a sideways plot" does seem to be a very
frequently asked question.  There may be some clever way to select
an x/y axis transposition in the general case, but if so I have not been
able to find it.  

The program was designed from the start to have a single 2D plot mode
with x always horizontal, and a single 3D plot mode with z always
perpendicular to to horizontal axis. It would be possible to introduce
a third plot mode with x vertical and y horizontal, but that would have
to essentially duplicate all the existing code in plot2d.c and graphics.c,
which I think is far too much effort and code for only a little gain.

It is easier to relax the restrictions on the 3D plot mode. 
Over time the program has gained the options

- "set view map"   reproduces the x-horizontal y-vertical 2D layout.
  This allow use of 3D plot styles but does not address the current request.

- "set view azimuth" relaxes the requirement that the z axis of a 3D
  plot must be perpendicular to the horizontal.  This allows 3D view
  conventions such as altitude-azimuth that were not previously possible.

- "set view projection [xz|yz]".  These are new in version 5.4
 They do allow a "sideways" plot layout, with two drawbacks:
 1) it only supports the subset of styles in "splot" rather than "plot"
 2) the horizontal axis is z (not y), which is a bit confusing

I think there is room to build on this last idea in a couple of ways.

The 1st drawback could be addressed by extending more of the
2D plot styles to work with "splot".
Currently these styles work in projection:
	points lines vectors impulses 
	filledcurves/polygons (2D and 3D not quite the same)
	boxes 
	boxerrorbars (just added, conditional on #ifdef BOXERROR3D)

Smoothing is currently limited to csplines.
Adding kdensity smoothing to the 3D code would be easy, which 
might address Peter's current request.  But we will undoubtedly
continue to see requests for sideways histograms, boxplots, etc.
Those would not be so easy to extend to 3D.

The 2nd drawback may have a clever solution, although I do not
see immediately how it would work.  Would it be possible to hide
the x-is-now-y, z-is-now-x coordinate replacement at the input stage
so that the user doesn't have to worry about it?  How?

I have been thinking about adding a new demo to show that
"set view projection xz" can produce many "sideways" plot styles,
but haven't done much about it yet.   It needs to use some class
of data that legitimately benefits from the horizontal rather than
vertical layout.   Just turning an existing plot sideways is not
IMHO particularly useful.  Peter - would the plot you constructed
by hand be a good example to adapt?  If the combination
    set view projection xz
    splot ... with boxerror
doesn't handle it adequately, that would highlight where it would
be worth some additional work on the code. 

	Ethan

Re: Sideways violin plots in one plot cmd?

[Juhász P. <pet...@gm...>]

On Wed, 2020-08-05 at 23:46 -0700, Dima Kogan wrote:
> A question. I'm looking at the violin plot demo page:
> 
>   http://gnuplot.sourceforge.net/demo/violinplot.html
> 
> It shows how to make a horizontal density plot using 'smooth
> kdensity'.
> 
> Then it shows to to make a vertical density plot by plotting
> horizontally into a table, and then plotting the table, with "using"
> directives used to swap the axes. This works clearly, but is there a
> way
> to do this (vertical density plots) with a single plot command? That
> would make my life much easier.
> 
> Thanks!
> 
> 
> 

Well, recently at $work I used gnuplot to produce a sideways histogram.
Because it had to be sideways, I had to resort to manually assembling
the plot with the boxxyerror style, with an awkward preprocessing step.

This, and your question, gives me the idea that it might be worth
adding a general 'transpose' option to some of the "clever" plot styles
that produce non-trivial vertical plots (histograms and boxplots,
mostly).

best regards,
Peter Juhasz

Sideways violin plots in one plot cmd?

[Dima K. <gn...@di...>]

A question. I'm looking at the violin plot demo page:

  http://gnuplot.sourceforge.net/demo/violinplot.html

It shows how to make a horizontal density plot using 'smooth kdensity'.

Then it shows to to make a vertical density plot by plotting
horizontally into a table, and then plotting the table, with "using"
directives used to swap the axes. This works clearly, but is there a way
to do this (vertical density plots) with a single plot command? That
would make my life much easier.

Thanks!

Re: Minor correction to the demos

[Ethan A M. <me...@uw...>]

On Wednesday, 5 August 2020 15:02:22 PDT Dima Kogan wrote:
> Hi. I'm experimenting with violin plots, and just found a minor bug in
> the demos.
> 
> I'm looking at
> 
>   http://gnuplot.sourceforge.net/demo/violinplot.html
> 
> The "Same data - kernel density" section plot command has "with
> filledcurves above y". Shouldn't it be "above x"? I.e. we're using the x
> axis as one of the boundaries of the polygon we're filling in. The next
> section turns the plots 90%, so there "above y" is correct.

I agree.  Fixed now.

	Ethan

Minor correction to the demos

[Dima K. <gn...@di...>]

Hi. I'm experimenting with violin plots, and just found a minor bug in
the demos.

I'm looking at

  http://gnuplot.sourceforge.net/demo/violinplot.html

The "Same data - kernel density" section plot command has "with
filledcurves above y". Shouldn't it be "above x"? I.e. we're using the x
axis as one of the boundaries of the polygon we're filling in. The next
section turns the plots 90%, so there "above y" is correct.

dima

Re: weirdness and probable bug with set xdata/ydata time

[Ethan M. <eam...@gm...>]

On Saturday, 18 July 2020 15:51:04 PDT Juhász Péter wrote:
> Hi,
> 
> I've ran into an issue with time data that looks like a bug.
> 
> The script below works as expected:
> 
> ################
> $data << END
> 2020-07-01      1.1
> 2020-07-08      2.2
> 2020-07-15      3.3
> END
> 
> set xdata time
> set timefmt "%Y-%m-%d"
> #set ydata time
> plot $data u 1:2 w lp
> ###############
> 
> It produces a plot with correctly parsed and displayed dates on the x
> axis, and the x range is also sensible.
> 
> But uncomment the `set ydata time` line and it fails:
> 
> ################
>          "x.plt" line 10: warning: Skipping data file with no valid
> points
> 
> gnuplot> plot $data u 1:2 w lp
>                               ^
>          "x.plt" line 10: x range is invalid
> ##############
> 
> I can sort of understand that it fails because it can't parse the
> second column as date - but then why does it report the *x* range as
> invalid?

Since all of the points are invalid (because of y), it has no data left to
use for autoscaling x.  Therefore the range remains [VERYLARGE:-VERYLARGE],
which is an invalid range.

Perhaps easier to understand what's going on if you leave time data and
time formats out of it:

############
$NAN << EOD
NaN NaN
NaN NaN
EOD

plot $NAN using 1:2
         x range is invalid
show xrange
         set xrange [ * : * ] noreverse writeback  # (currently [8.98847e+307:-8.98847e+307] )
set xdata time
show xrange
         set xrange [ * : * ] noreverse writeback  # (currently ["         warning: time value out of range
32650/35/00,-468425088:00":"         warning: time value out of range
32650/35/00,-468425088:00"] )
############

> 
> Even more spooky is what it thinks about the x range after the failed
> plot:
> ##############
> gnuplot> sh xr
> 
> 	set xdata time
> 	set xrange [ * : * ] noreverse writeback  # (currently
> ["         warning: time value out of range
> 0000-35-32586":"         warning: time value out of range
> 0000-35-32586"] )
> #############
> 
> If I run gnuplot with valgrind, it barfs some "Conditional jump or move
> depends on uninitialised value(s)" errors, so some internal state is
> definitely corrupted.

Dunno. Valgrind doesn't complain here.

	cheers,

		Ethan


> 
> Best regards,
> Peter Juhasz
> 
> 
> 
> 
> 
> _______________________________________________
> gnuplot-beta mailing list
> gnu...@li...
> Membership management via: https://lists.sourceforge.net/lists/listinfo/gnuplot-beta
>