gnuplot-beta — Developers and beta testers of new gnuplot code

Re: [BUG] pdfcairo notransparent has transparency

[Ethan A M. <me...@uw...>]

On Thursday, 20 August 2020 21:42:51 PDT Henri Menke wrote:
> Dear all,
> 
> With regards to my recent patch to optionally disable transparency on
> cairolatex, I stumbled upon the fact that the PDF output of the cairo
> terminal does not properly respect the notransparent option.  Consider
> the following minimal example:
> 
>     set terminal pdfcairo notransparent
>     set output "test.pdf"
>     plot sin(x)
> 
> When running this, the page is still encapsulated in a transparency
> group.  

The "{no}transparent" option used by several gnuplot terminals
specifically refers to the plot background.  It does not, and was never
intended to, disable transparency in the plot proper.   The two things
are orthogonal.   Consider for example
 
    set term png notruecolor transparent 

This sets the background to be transparent but the plot contains no
alpha channel because it uses indexed color. 
Contrast this with

    set term png truecolor backgound "grey"

This sets the background to be opaque (=notransparent),
but the plot itself uses 24bit RGB color + alpha channel.

	Ethan



> This breaks PDF/A compatibility.
> 
>     $ gnuplot test.gnuplot
>     $ grep -aC6 'Transparency' test.pdf
>     << /Type /Page % 1
>        /Parent 1 0 R
>        /MediaBox [ 0 0 360 216 ]
>        /Contents 4 0 R
>        /Group <<
>           /Type /Group
>           /S /Transparency
>           /I true
>           /CS /DeviceRGB
>        >>
>        /Resources 3 0 R
>     >>
>     endobj
> 
> I'm not entire sure and I haven't tested this yet, but the cairo
> terminal uses the gp_cairo_set_color function from
> src/wxterminal/gp_cairo.c which in turn calls cairo_set_source_rgba.
> This explicit call for transparency might be the cause for cairo
> inserting a transparency group, even if all alpha values are zero or
> one.
> 
> Cheers, Henri
> 

[BUG] pdfcairo notransparent has transparency

[Henri M. <hen...@gm...>]

Dear all,

With regards to my recent patch to optionally disable transparency on
cairolatex, I stumbled upon the fact that the PDF output of the cairo
terminal does not properly respect the notransparent option.  Consider
the following minimal example:

    set terminal pdfcairo notransparent
    set output "test.pdf"
    plot sin(x)

When running this, the page is still encapsulated in a transparency
group.  This breaks PDF/A compatibility.

    $ gnuplot test.gnuplot
    $ grep -aC6 'Transparency' test.pdf
    << /Type /Page % 1
       /Parent 1 0 R
       /MediaBox [ 0 0 360 216 ]
       /Contents 4 0 R
       /Group <<
          /Type /Group
          /S /Transparency
          /I true
          /CS /DeviceRGB
       >>
       /Resources 3 0 R
    >>
    endobj

I'm not entire sure and I haven't tested this yet, but the cairo
terminal uses the gp_cairo_set_color function from
src/wxterminal/gp_cairo.c which in turn calls cairo_set_source_rgba.
This explicit call for transparency might be the cause for cairo
inserting a transparency group, even if all alpha values are zero or
one.

Cheers, Henri

Re: [PATCH] cairolatex: respect transparency setting

[Ethan A M. <me...@uw...>]

On Thursday, 20 August 2020 20:17:44 PDT Henri Menke wrote:
> On 20/08/20, 20:01, Ethan A Merritt wrote:
> > On Thursday, 20 August 2020 18:34:54 PDT Henri Menke wrote:
> > > Currently the cairolatex terminal does not respect transparency options
> > > given in the terminal setting.  In this minimal example the
> > > notransparent option is ignored:
> > > 
> > >     set terminal cairolatex png notransparent
> > >     set output "test.tex"
> > >     plot sin(x)
> > > 
> > > Running this through gnuplot and then using `file' reveals that the
> > > alpha channel is still present:
> > > 
> > >     $ gnuplot test.gnuplot && file test.png
> > >     test.png: PNG image data, 1500 x 900, 8-bit/color RGBA, non-interlace
> > 
> > I think the raionale for the current code is that the png image created for
> > inclusion in a LaTeX document must have a transparent background in order
> > to support the "back" and "behind" attributes for text elements, since they
> > will be drawn by the latex engine and then covered by inclusion of the
> > graphics image.  The structure of the latex processing is:
> > 
> >     render text elements that are "behind" or "back"
> >     \includegraphics{gnuplot_figure}
> >     render text elements that are "front"
> > 
> > If you use a solid color background in the graphics image, it will hide
> > any previously rendered text elements.
> 
> The reason why I'm looking into this is that currently it is impossible
> to generate PDF/A when using plots generated by gnuplot.  PDF/A doesn't
> allow transparency, so I'm trying to remove all of it (or at least make
> it optional).  As noted in the patch, the default behaviour of using
> transparency remains, so this is entirely optional.

I am researching this as I type, so please bear with me.

>From Wikipedia I see
	Transparent objects and layers (Optional Content Groups) are 
	forbidden in PDF/A-1, but are allowed in PDF/A-2.
PDF/A-1 was published in 2005; PDF/A-2 in 2011; PDF/A-3 in 2012;
So maybe this is no longer a limitation?

>From my own knowledge, there are commonly available tools that 
render the transparency in a PDF document once, then save the result
as an embedded image that no longer contains transparency.
For example if you use gnuplot to create a PostScript file directly it cannot
use transparency because PostScript itself does not support transparency.
But if you use gnuplot to create a PDF file with transparency and then
convert it
	pdf2ps input.pdf output.ps
you get a PostScript file in which the transparency is "baked in" rather
than interpreted.   Logically then you could take that PostScript file
and convert it back to PDF, keeping the pre-rendered image that no
longer uses an alpha channel. 
	ps2pdf output.ps new.pdf
Whether that particular pair of tools produces a PDF/A compliant
file I do not know.

   best,

	Ethan

> 
> Kind regards,
> Henri
> 
> > 
> > 	Ethan
> > 
> > > 
> > > With this patch the transparency setting will be honoured.  The current
> > > default setting of implicitly assuming transparency is unaltered.
> > > ---
> > >  term/cairo.trm | 2 +-
> > >  1 file changed, 1 insertion(+), 1 deletion(-)
> > > 
> > > diff --git a/term/cairo.trm b/term/cairo.trm
> > > index ee748dfe2..3ef76ecc6 100644
> > > --- a/term/cairo.trm
> > > +++ b/term/cairo.trm
> > > @@ -849,7 +849,7 @@ void cairotrm_graphics()
> > >  	plot.background.g = cairo_params->background.g;
> > >  	plot.background.b = cairo_params->background.b;
> > >  	gp_cairo_set_background(cairo_params->background);
> > > -	if (ISCAIROLATEX || cairo_params->transparent)
> > > +	if (cairo_params->transparent)
> > >  		gp_cairo_clear_background(&plot);
> > >  	else
> > >  		gp_cairo_solid_background(&plot);
> > > 
> > 
> > 
> > 
> > 

Re: [PATCH] cairolatex: respect transparency setting

[Henri M. <hen...@gm...>]

On 20/08/20, 20:01, Ethan A Merritt wrote:
> On Thursday, 20 August 2020 18:34:54 PDT Henri Menke wrote:
> > Currently the cairolatex terminal does not respect transparency options
> > given in the terminal setting.  In this minimal example the
> > notransparent option is ignored:
> > 
> >     set terminal cairolatex png notransparent
> >     set output "test.tex"
> >     plot sin(x)
> > 
> > Running this through gnuplot and then using `file' reveals that the
> > alpha channel is still present:
> > 
> >     $ gnuplot test.gnuplot && file test.png
> >     test.png: PNG image data, 1500 x 900, 8-bit/color RGBA, non-interlace
> 
> I think the raionale for the current code is that the png image created for
> inclusion in a LaTeX document must have a transparent background in order
> to support the "back" and "behind" attributes for text elements, since they
> will be drawn by the latex engine and then covered by inclusion of the
> graphics image.  The structure of the latex processing is:
> 
>     render text elements that are "behind" or "back"
>     \includegraphics{gnuplot_figure}
>     render text elements that are "front"
> 
> If you use a solid color background in the graphics image, it will hide
> any previously rendered text elements.

The reason why I'm looking into this is that currently it is impossible
to generate PDF/A when using plots generated by gnuplot.  PDF/A doesn't
allow transparency, so I'm trying to remove all of it (or at least make
it optional).  As noted in the patch, the default behaviour of using
transparency remains, so this is entirely optional.

Kind regards,
Henri

> 
> 	Ethan
> 
> > 
> > With this patch the transparency setting will be honoured.  The current
> > default setting of implicitly assuming transparency is unaltered.
> > ---
> >  term/cairo.trm | 2 +-
> >  1 file changed, 1 insertion(+), 1 deletion(-)
> > 
> > diff --git a/term/cairo.trm b/term/cairo.trm
> > index ee748dfe2..3ef76ecc6 100644
> > --- a/term/cairo.trm
> > +++ b/term/cairo.trm
> > @@ -849,7 +849,7 @@ void cairotrm_graphics()
> >  	plot.background.g = cairo_params->background.g;
> >  	plot.background.b = cairo_params->background.b;
> >  	gp_cairo_set_background(cairo_params->background);
> > -	if (ISCAIROLATEX || cairo_params->transparent)
> > +	if (cairo_params->transparent)
> >  		gp_cairo_clear_background(&plot);
> >  	else
> >  		gp_cairo_solid_background(&plot);
> > 
> 
> 
> 
> 

Re: [PATCH] cairolatex: respect transparency setting

[Ethan A M. <me...@uw...>]

On Thursday, 20 August 2020 19:34:40 PDT Henri Menke wrote:
> Now there seems to a problem with ordering of the elements in the
> resulting TeX file.  The ticklabels are behind the background image.
> Even if transparency is enabled, this is probably a bad idea.  I'll
> investigate.

That is controlled by a generic setting, not specific to the latex terminals:

    set tics {front|back}

The default is "back".

	Ethan


> 
> Cheers, Henri
> 
> On 21/08/20, 13:34, Henri Menke wrote:
> > Currently the cairolatex terminal does not respect transparency options
> > given in the terminal setting.  In this minimal example the
> > notransparent option is ignored:
> > 
> >     set terminal cairolatex png notransparent
> >     set output "test.tex"
> >     plot sin(x)
> > 
> > Running this through gnuplot and then using `file' reveals that the
> > alpha channel is still present:
> > 
> >     $ gnuplot test.gnuplot && file test.png
> >     test.png: PNG image data, 1500 x 900, 8-bit/color RGBA, non-interlace
> > 
> > With this patch the transparency setting will be honoured.  The current
> > default setting of implicitly assuming transparency is unaltered.
> > ---
> >  term/cairo.trm | 2 +-
> >  1 file changed, 1 insertion(+), 1 deletion(-)
> > 
> > diff --git a/term/cairo.trm b/term/cairo.trm
> > index ee748dfe2..3ef76ecc6 100644
> > --- a/term/cairo.trm
> > +++ b/term/cairo.trm
> > @@ -849,7 +849,7 @@ void cairotrm_graphics()
> >  	plot.background.g = cairo_params->background.g;
> >  	plot.background.b = cairo_params->background.b;
> >  	gp_cairo_set_background(cairo_params->background);
> > -	if (ISCAIROLATEX || cairo_params->transparent)
> > +	if (cairo_params->transparent)
> >  		gp_cairo_clear_background(&plot);
> >  	else
> >  		gp_cairo_solid_background(&plot);
> 

Re: [PATCH] cairolatex: respect transparency setting

[Ethan A M. <me...@uw...>]

On Thursday, 20 August 2020 18:34:54 PDT Henri Menke wrote:
> Currently the cairolatex terminal does not respect transparency options
> given in the terminal setting.  In this minimal example the
> notransparent option is ignored:
> 
>     set terminal cairolatex png notransparent
>     set output "test.tex"
>     plot sin(x)
> 
> Running this through gnuplot and then using `file' reveals that the
> alpha channel is still present:
> 
>     $ gnuplot test.gnuplot && file test.png
>     test.png: PNG image data, 1500 x 900, 8-bit/color RGBA, non-interlace

I think the raionale for the current code is that the png image created for
inclusion in a LaTeX document must have a transparent background in order
to support the "back" and "behind" attributes for text elements, since they
will be drawn by the latex engine and then covered by inclusion of the
graphics image.  The structure of the latex processing is:

    render text elements that are "behind" or "back"
    \includegraphics{gnuplot_figure}
    render text elements that are "front"

If you use a solid color background in the graphics image, it will hide
any previously rendered text elements.

	Ethan

> 
> With this patch the transparency setting will be honoured.  The current
> default setting of implicitly assuming transparency is unaltered.
> ---
>  term/cairo.trm | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/term/cairo.trm b/term/cairo.trm
> index ee748dfe2..3ef76ecc6 100644
> --- a/term/cairo.trm
> +++ b/term/cairo.trm
> @@ -849,7 +849,7 @@ void cairotrm_graphics()
>  	plot.background.g = cairo_params->background.g;
>  	plot.background.b = cairo_params->background.b;
>  	gp_cairo_set_background(cairo_params->background);
> -	if (ISCAIROLATEX || cairo_params->transparent)
> +	if (cairo_params->transparent)
>  		gp_cairo_clear_background(&plot);
>  	else
>  		gp_cairo_solid_background(&plot);
> 

Re: [PATCH] cairolatex: respect transparency setting

[Henri M. <hen...@gm...>]

Now there seems to a problem with ordering of the elements in the
resulting TeX file.  The ticklabels are behind the background image.
Even if transparency is enabled, this is probably a bad idea.  I'll
investigate.

Cheers, Henri

On 21/08/20, 13:34, Henri Menke wrote:
> Currently the cairolatex terminal does not respect transparency options
> given in the terminal setting.  In this minimal example the
> notransparent option is ignored:
> 
>     set terminal cairolatex png notransparent
>     set output "test.tex"
>     plot sin(x)
> 
> Running this through gnuplot and then using `file' reveals that the
> alpha channel is still present:
> 
>     $ gnuplot test.gnuplot && file test.png
>     test.png: PNG image data, 1500 x 900, 8-bit/color RGBA, non-interlace
> 
> With this patch the transparency setting will be honoured.  The current
> default setting of implicitly assuming transparency is unaltered.
> ---
>  term/cairo.trm | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/term/cairo.trm b/term/cairo.trm
> index ee748dfe2..3ef76ecc6 100644
> --- a/term/cairo.trm
> +++ b/term/cairo.trm
> @@ -849,7 +849,7 @@ void cairotrm_graphics()
>  	plot.background.g = cairo_params->background.g;
>  	plot.background.b = cairo_params->background.b;
>  	gp_cairo_set_background(cairo_params->background);
> -	if (ISCAIROLATEX || cairo_params->transparent)
> +	if (cairo_params->transparent)
>  		gp_cairo_clear_background(&plot);
>  	else
>  		gp_cairo_solid_background(&plot);
> -- 
> 2.28.0
> 

[PATCH] cairolatex: respect transparency setting

[Henri M. <hen...@gm...>]

Currently the cairolatex terminal does not respect transparency options
given in the terminal setting.  In this minimal example the
notransparent option is ignored:

    set terminal cairolatex png notransparent
    set output "test.tex"
    plot sin(x)

Running this through gnuplot and then using `file' reveals that the
alpha channel is still present:

    $ gnuplot test.gnuplot && file test.png
    test.png: PNG image data, 1500 x 900, 8-bit/color RGBA, non-interlace

With this patch the transparency setting will be honoured.  The current
default setting of implicitly assuming transparency is unaltered.
---
 term/cairo.trm | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/term/cairo.trm b/term/cairo.trm
index ee748dfe2..3ef76ecc6 100644
--- a/term/cairo.trm
+++ b/term/cairo.trm
@@ -849,7 +849,7 @@ void cairotrm_graphics()
 	plot.background.g = cairo_params->background.g;
 	plot.background.b = cairo_params->background.b;
 	gp_cairo_set_background(cairo_params->background);
-	if (ISCAIROLATEX || cairo_params->transparent)
+	if (cairo_params->transparent)
 		gp_cairo_clear_background(&plot);
 	else
 		gp_cairo_solid_background(&plot);
-- 
2.28.0

Re: Sideways violin plots in one plot cmd?

[Dima K. <gn...@di...>]

Ethan A Merritt <me...@uw...> writes:

> I guess I still don't get it.
> Since the wrapper can pass in any necessary set of commands,
> why not just pass in the commands that do the double plot?

Because my tools would then need special-case logic for this one case. I
would need some sort of "feedgnuplot --violin" and logic internally to
handle it. At the very start I decided that explicitly supporting
everything gnuplot supports with such special-case logic would be
extremely effortful and extremely error-prone and maintenance-full. And
I avoided the issue entirely by passing user-specified strings to
gnuplot transparently. Which works great for most styles.

If (as a random example) I have 3-column input that I want to plot as
errorbars I can do this:

  < 3columns_datafile feedgnuplot --domain --tuplesizeall 3 --with yerrorbars

feegnuplot then knows to be dealing with 3-values-per-point data, and
tells gnuplot to plot with "yerrorbars". There's no special-case logic
for "yerrorbars". And I don't want to create a separate path for violin
plots.

Full disclosure: there IS one special-case path to deal with histograms.
But those are extremely useful, and worth violating the general design.
violin plots aren't nearly as ubiquitous, and I dont want to add a big
special-case path for them.

Re: Sideways violin plots in one plot cmd?

[Juhász P. <pet...@gm...>]

On Sat, 2020-08-15 at 13:17 -0700, Ethan A Merritt wrote:
> On Friday, 14 August 2020 10:34:52 PDT Juhász Péter wrote:
> > On Wed, 2020-08-05 at 23:46 -0700, Dima Kogan wrote:
> > > A question. I'm looking at the violin plot demo page:
> > > 
> > >   http://gnuplot.sourceforge.net/demo/violinplot.html
> > > 
> > > It shows how to make a horizontal density plot using 'smooth
> > > kdensity'.
> > > 
> > > Then it shows to to make a vertical density plot by plotting
> > > horizontally into a table, and then plotting the table, with
> > > "using"
> > > directives used to swap the axes. This works clearly, but is
> > > there a
> > > way
> > > to do this (vertical density plots) with a single plot command?
> > > That
> > > would make my life much easier.
> > > 
> > > Thanks!
> > > 
> > > 
> > > 
> > 
> > Well, recently at $work I used gnuplot to produce a sideways
> > histogram.
> > Because it had to be sideways, I had to resort to manually
> > assembling
> > the plot with the boxxyerror style, with an awkward preprocessing
> > step.
> > 
> > This, and your question, gives me the idea that it might be worth
> > adding a general 'transpose' option to some of the "clever" plot
> > styles
> > that produce non-trivial vertical plots (histograms and boxplots,
> > mostly).
> > 
> > best regards,
> > Peter Juhasz
> 
> "How to make a sideways plot" does seem to be a very
> frequently asked question.  There may be some clever way to select
> an x/y axis transposition in the general case, but if so I have not
> been
> able to find it.  
> 

I might have mislead the discussion with my suggestion, because as it
turns out, the only common thing it has with the original violin plot
question is "would there be a way to do X in a single command". Sorry
for the confusion.

That said, I don't think bringing in 3D (s)plot styles is a good
direction towards a general solution for sideways plots. To me, it
sounds like looking for our lost car keys in the corner only because
that's where there is light.

> The program was designed from the start to have a single 2D plot mode
> with x always horizontal, [...]

Yes, originally it was designed like that, but then more complex plot
styles like histograms and boxplots were added, which do non-trivial
preprocessing and aggregating steps before plotting the data. For
boxplots (and less acutely, even for histograms) the x axis of the plot
is no longer directly maps to any column of the original data.

> Peter - would the plot you constructed
> by hand be a good example to adapt?  If the combination
>     set view projection xz
>     splot ... with boxerror
> doesn't handle it adequately, that would highlight where it would
> be worth some additional work on the code. 

No, not really, unless I misunderstand you - my point was that I wanted
to produce (IIRC) a columnstacked histogram, and if it could have been
a vertical plot, I could have used `set style histogram columnstacked;
set style data histogram`, with all the nice intuitive data layout and
all the automatic box placement that style provides. But the
requirements called for a horizontal plot, so I had to do all that by
hand. Doing the same in 3D wouldn't have helped.

Peter

> 
> 	Ethan
> 
> 

Re: Sideways violin plots in one plot cmd?

[Ethan A M. <me...@uw...>]

On Saturday, 15 August 2020 22:06:34 PDT Dima Kogan wrote:
> Ethan A Merritt <me...@uw...> writes:
> >
> > I don't get it. What is it about the shell interface that would prevent you
> > from doing exactly what is in violinplot.dem?
> >   set table $datablock 
> >   plot ...
> >   unset table
> >   plot $datablock ...
> 
> Both gnuplotlib and feedgnuplot work as very thin wrappers around
> gnuplot, where they pass strings directly to gnuplot without knowing
> what the strings mean. This allows those interfaces to support lots and
> lots of styles, while containing no code that makes those styles work:
> all the logic is inside gnuplot itself. The double-plotting in
> violinplot.dem is qualitatively different than pretty much every other
> gnuplot style, so I would need to add special-case support for it. I
> COULD do that, but I really don't want to: the really simple API of
> feedgnuplot and gnuplotlib is a feature, and this special-case path
> would dilute that feature.
> 
> In a perfect world we'd be able to make violin plots like all others:
> with some "set" commands (that don't have the data in it) and a single
> plot/splot command that is given data. That is sounding like a big
> project inside gnuplot, though.


I guess I still don't get it.
Since the wrapper can pass in any necessary set of commands,
why not just pass in the commands that do the double plot?

Would it help to have a library of helper scripts for gnuplot?
Here's one that creates a vertical violin plot from a single gnuplot
command.  You could obviously add a lot of customization either
by passing in additional parameters or defining them globally.

   gnuplot> set term pngcairo
   gnuplot> set output 'violin.png'
   gnuplot> # unit width violin plot centered at x=10., linetype 3
   gnuplot> call 'violin.gp' 'filename.dat' 10. 3 

violin.gp script follows
%%%%%%%%%%%%%%%%%%%%%%%%
#
# invoke this script from inside gnuplot using
#       call violin.gp filename xcenter linetype
#
filename = ARGV[1]
xcenter = ARGV[2]
LT = ARGV[3]
#
set xrange [*:*]
set table $kdensity
plot filename using 2:(1) smooth kdensity with filledcurves above x1
unset table
stats $kdensity using 2 noout
plot [0:20][0:*]\
     $kdensity using (xcenter + column(2)/STATS_max):1 with filledcurves x=xcenter lt LT, \
     $kdensity using (xcenter - column(2)/STATS_max):1 with filledcurves x=xcenter lt LT
#
%%%%%%%%%%%%%%%%%%%%%%%%%

Re: Sideways violin plots in one plot cmd?

[Dima K. <gn...@di...>]

Ethan A Merritt <me...@uw...> writes:
>
> I don't get it. What is it about the shell interface that would prevent you
> from doing exactly what is in violinplot.dem?
>   set table $datablock 
>   plot ...
>   unset table
>   plot $datablock ...

Both gnuplotlib and feedgnuplot work as very thin wrappers around
gnuplot, where they pass strings directly to gnuplot without knowing
what the strings mean. This allows those interfaces to support lots and
lots of styles, while containing no code that makes those styles work:
all the logic is inside gnuplot itself. The double-plotting in
violinplot.dem is qualitatively different than pretty much every other
gnuplot style, so I would need to add special-case support for it. I
COULD do that, but I really don't want to: the really simple API of
feedgnuplot and gnuplotlib is a feature, and this special-case path
would dilute that feature.

In a perfect world we'd be able to make violin plots like all others:
with some "set" commands (that don't have the data in it) and a single
plot/splot command that is given data. That is sounding like a big
project inside gnuplot, though.

Re: Sideways violin plots in one plot cmd?

[Ethan A M. <me...@uw...>]

On Saturday, 15 August 2020 16:37:35 PDT Dima Kogan wrote:
> Ethan A Merritt <me...@uw...> writes:
> 
> > Are you trying to avoid the extra step of saving to an array or
> > datablock and then replotting? That seems like a totally different
> > question from vertical/horizontal layout or coordinate
> > transformations.
> 
> Yes. Avoiding that extra step is the question in this thread; see the
> subject. I've never tried to plot sideways histograms, like Peter did; I
> was assuming that this is simple if we plot into a datablock and then
> replot, but maybe not.
> 
> In any case, here's my problem. I use gnuplot A LOT. But 99% of that
> usage is via one of two wrappers (I wrote both of these):
> 
>   feedgnuplot (a nice shell interface)
>   gnuplotlib (a nice python interface)
> 
> Both of these work as thin wrappers around gnuplot, the idea being that
> with some "set" commands and a fancy "plot" command you can do anything.
> As it currently stands, violin plots are apparently not a part of this
> "anything". One could say that my wrappers are silly and not
> flexible-enough. Which maybe is true, but realistically speaking, I
> don't see ever expanding these tools to use datablocks, since that would
> break all sorts of interface assumptions both of these tools are making.

I don't get it. What is it about the shell interface that would prevent you
from doing exactly what is in violinplot.dem?
  set table $datablock 
  plot ...
  unset table
  plot $datablock ...

	Ethan

Re: Sideways violin plots in one plot cmd?

[Dima K. <gn...@di...>]

Ethan A Merritt <me...@uw...> writes:

> Are you trying to avoid the extra step of saving to an array or
> datablock and then replotting? That seems like a totally different
> question from vertical/horizontal layout or coordinate
> transformations.

Yes. Avoiding that extra step is the question in this thread; see the
subject. I've never tried to plot sideways histograms, like Peter did; I
was assuming that this is simple if we plot into a datablock and then
replot, but maybe not.

In any case, here's my problem. I use gnuplot A LOT. But 99% of that
usage is via one of two wrappers (I wrote both of these):

  feedgnuplot (a nice shell interface)
  gnuplotlib (a nice python interface)

Both of these work as thin wrappers around gnuplot, the idea being that
with some "set" commands and a fancy "plot" command you can do anything.
As it currently stands, violin plots are apparently not a part of this
"anything". One could say that my wrappers are silly and not
flexible-enough. Which maybe is true, but realistically speaking, I
don't see ever expanding these tools to use datablocks, since that would
break all sorts of interface assumptions both of these tools are making.
So I can live without violin plots, or maybe go use matplotlib. Which
I'd rather not do.

Any suggestions appreciated.

Re: Sideways violin plots in one plot cmd?

[Ethan A M. <me...@uw...>]

On Saturday, 15 August 2020 13:51:32 PDT Dima Kogan wrote:
> If we add better support for these, I think supporting more general
> transformations than just a simple x/y transpose would be useful. While
> thinking of violin plots I had a workaround idea: instead of a set of
> horizontally-spaced vertical density plots, I'd be fine with a set of
> vertically-spaced horizontal density plots. But I think gnuplot
> currently can't do that either: the density plots must be horizontal AND
> they must start at y=0. Right? So being able to support translations
> would be nice.

I may not understand what you want.
kdensity smoothing is a sum of Gaussians, so by definition the baseline
is zero.   That is not a limitation of how gnuplot plots things, that is
the definition of the function being plotted.    You can of course
save the resulting curve and plot it with an offset on y, just as the 
violin plot demo replots it vertically with an offset on x.

Are you trying to avoid the extra step of saving to an array or
datablock and then replotting?   That seems like a totally different
question from vertical/horizontal layout or coordinate transformations.

	Ethan

Re: Sideways violin plots in one plot cmd?

[Dima K. <gn...@di...>]

If we add better support for these, I think supporting more general
transformations than just a simple x/y transpose would be useful. While
thinking of violin plots I had a workaround idea: instead of a set of
horizontally-spaced vertical density plots, I'd be fine with a set of
vertically-spaced horizontal density plots. But I think gnuplot
currently can't do that either: the density plots must be horizontal AND
they must start at y=0. Right? So being able to support translations
would be nice.

Re: Sideways violin plots in one plot cmd?

[Ethan A M. <me...@uw...>]

On Friday, 14 August 2020 10:34:52 PDT Juhász Péter wrote:
> On Wed, 2020-08-05 at 23:46 -0700, Dima Kogan wrote:
> > A question. I'm looking at the violin plot demo page:
> > 
> >   http://gnuplot.sourceforge.net/demo/violinplot.html
> > 
> > It shows how to make a horizontal density plot using 'smooth
> > kdensity'.
> > 
> > Then it shows to to make a vertical density plot by plotting
> > horizontally into a table, and then plotting the table, with "using"
> > directives used to swap the axes. This works clearly, but is there a
> > way
> > to do this (vertical density plots) with a single plot command? That
> > would make my life much easier.
> > 
> > Thanks!
> > 
> > 
> > 
> 
> Well, recently at $work I used gnuplot to produce a sideways histogram.
> Because it had to be sideways, I had to resort to manually assembling
> the plot with the boxxyerror style, with an awkward preprocessing step.
> 
> This, and your question, gives me the idea that it might be worth
> adding a general 'transpose' option to some of the "clever" plot styles
> that produce non-trivial vertical plots (histograms and boxplots,
> mostly).
> 
> best regards,
> Peter Juhasz

"How to make a sideways plot" does seem to be a very
frequently asked question.  There may be some clever way to select
an x/y axis transposition in the general case, but if so I have not been
able to find it.  

The program was designed from the start to have a single 2D plot mode
with x always horizontal, and a single 3D plot mode with z always
perpendicular to to horizontal axis. It would be possible to introduce
a third plot mode with x vertical and y horizontal, but that would have
to essentially duplicate all the existing code in plot2d.c and graphics.c,
which I think is far too much effort and code for only a little gain.

It is easier to relax the restrictions on the 3D plot mode. 
Over time the program has gained the options

- "set view map"   reproduces the x-horizontal y-vertical 2D layout.
  This allow use of 3D plot styles but does not address the current request.

- "set view azimuth" relaxes the requirement that the z axis of a 3D
  plot must be perpendicular to the horizontal.  This allows 3D view
  conventions such as altitude-azimuth that were not previously possible.

- "set view projection [xz|yz]".  These are new in version 5.4
 They do allow a "sideways" plot layout, with two drawbacks:
 1) it only supports the subset of styles in "splot" rather than "plot"
 2) the horizontal axis is z (not y), which is a bit confusing

I think there is room to build on this last idea in a couple of ways.

The 1st drawback could be addressed by extending more of the
2D plot styles to work with "splot".
Currently these styles work in projection:
	points lines vectors impulses 
	filledcurves/polygons (2D and 3D not quite the same)
	boxes 
	boxerrorbars (just added, conditional on #ifdef BOXERROR3D)

Smoothing is currently limited to csplines.
Adding kdensity smoothing to the 3D code would be easy, which 
might address Peter's current request.  But we will undoubtedly
continue to see requests for sideways histograms, boxplots, etc.
Those would not be so easy to extend to 3D.

The 2nd drawback may have a clever solution, although I do not
see immediately how it would work.  Would it be possible to hide
the x-is-now-y, z-is-now-x coordinate replacement at the input stage
so that the user doesn't have to worry about it?  How?

I have been thinking about adding a new demo to show that
"set view projection xz" can produce many "sideways" plot styles,
but haven't done much about it yet.   It needs to use some class
of data that legitimately benefits from the horizontal rather than
vertical layout.   Just turning an existing plot sideways is not
IMHO particularly useful.  Peter - would the plot you constructed
by hand be a good example to adapt?  If the combination
    set view projection xz
    splot ... with boxerror
doesn't handle it adequately, that would highlight where it would
be worth some additional work on the code. 

	Ethan

Re: Sideways violin plots in one plot cmd?

[Juhász P. <pet...@gm...>]

On Wed, 2020-08-05 at 23:46 -0700, Dima Kogan wrote:
> A question. I'm looking at the violin plot demo page:
> 
>   http://gnuplot.sourceforge.net/demo/violinplot.html
> 
> It shows how to make a horizontal density plot using 'smooth
> kdensity'.
> 
> Then it shows to to make a vertical density plot by plotting
> horizontally into a table, and then plotting the table, with "using"
> directives used to swap the axes. This works clearly, but is there a
> way
> to do this (vertical density plots) with a single plot command? That
> would make my life much easier.
> 
> Thanks!
> 
> 
> 

Well, recently at $work I used gnuplot to produce a sideways histogram.
Because it had to be sideways, I had to resort to manually assembling
the plot with the boxxyerror style, with an awkward preprocessing step.

This, and your question, gives me the idea that it might be worth
adding a general 'transpose' option to some of the "clever" plot styles
that produce non-trivial vertical plots (histograms and boxplots,
mostly).

best regards,
Peter Juhasz

Sideways violin plots in one plot cmd?

[Dima K. <gn...@di...>]

A question. I'm looking at the violin plot demo page:

  http://gnuplot.sourceforge.net/demo/violinplot.html

It shows how to make a horizontal density plot using 'smooth kdensity'.

Then it shows to to make a vertical density plot by plotting
horizontally into a table, and then plotting the table, with "using"
directives used to swap the axes. This works clearly, but is there a way
to do this (vertical density plots) with a single plot command? That
would make my life much easier.

Thanks!

Re: Minor correction to the demos

[Ethan A M. <me...@uw...>]

On Wednesday, 5 August 2020 15:02:22 PDT Dima Kogan wrote:
> Hi. I'm experimenting with violin plots, and just found a minor bug in
> the demos.
> 
> I'm looking at
> 
>   http://gnuplot.sourceforge.net/demo/violinplot.html
> 
> The "Same data - kernel density" section plot command has "with
> filledcurves above y". Shouldn't it be "above x"? I.e. we're using the x
> axis as one of the boundaries of the polygon we're filling in. The next
> section turns the plots 90%, so there "above y" is correct.

I agree.  Fixed now.

	Ethan

Minor correction to the demos

[Dima K. <gn...@di...>]

Hi. I'm experimenting with violin plots, and just found a minor bug in
the demos.

I'm looking at

  http://gnuplot.sourceforge.net/demo/violinplot.html

The "Same data - kernel density" section plot command has "with
filledcurves above y". Shouldn't it be "above x"? I.e. we're using the x
axis as one of the boundaries of the polygon we're filling in. The next
section turns the plots 90%, so there "above y" is correct.

dima

Re: weirdness and probable bug with set xdata/ydata time

[Ethan M. <eam...@gm...>]

On Saturday, 18 July 2020 15:51:04 PDT Juhász Péter wrote:
> Hi,
> 
> I've ran into an issue with time data that looks like a bug.
> 
> The script below works as expected:
> 
> ################
> $data << END
> 2020-07-01      1.1
> 2020-07-08      2.2
> 2020-07-15      3.3
> END
> 
> set xdata time
> set timefmt "%Y-%m-%d"
> #set ydata time
> plot $data u 1:2 w lp
> ###############
> 
> It produces a plot with correctly parsed and displayed dates on the x
> axis, and the x range is also sensible.
> 
> But uncomment the `set ydata time` line and it fails:
> 
> ################
>          "x.plt" line 10: warning: Skipping data file with no valid
> points
> 
> gnuplot> plot $data u 1:2 w lp
>                               ^
>          "x.plt" line 10: x range is invalid
> ##############
> 
> I can sort of understand that it fails because it can't parse the
> second column as date - but then why does it report the *x* range as
> invalid?

Since all of the points are invalid (because of y), it has no data left to
use for autoscaling x.  Therefore the range remains [VERYLARGE:-VERYLARGE],
which is an invalid range.

Perhaps easier to understand what's going on if you leave time data and
time formats out of it:

############
$NAN << EOD
NaN NaN
NaN NaN
EOD

plot $NAN using 1:2
         x range is invalid
show xrange
         set xrange [ * : * ] noreverse writeback  # (currently [8.98847e+307:-8.98847e+307] )
set xdata time
show xrange
         set xrange [ * : * ] noreverse writeback  # (currently ["         warning: time value out of range
32650/35/00,-468425088:00":"         warning: time value out of range
32650/35/00,-468425088:00"] )
############

> 
> Even more spooky is what it thinks about the x range after the failed
> plot:
> ##############
> gnuplot> sh xr
> 
> 	set xdata time
> 	set xrange [ * : * ] noreverse writeback  # (currently
> ["         warning: time value out of range
> 0000-35-32586":"         warning: time value out of range
> 0000-35-32586"] )
> #############
> 
> If I run gnuplot with valgrind, it barfs some "Conditional jump or move
> depends on uninitialised value(s)" errors, so some internal state is
> definitely corrupted.

Dunno. Valgrind doesn't complain here.

	cheers,

		Ethan


> 
> Best regards,
> Peter Juhasz
> 
> 
> 
> 
> 
> _______________________________________________
> gnuplot-beta mailing list
> gnu...@li...
> Membership management via: https://lists.sourceforge.net/lists/listinfo/gnuplot-beta
> 

weirdness and probable bug with set xdata/ydata time

[Juhász P. <pet...@gm...>]

Hi,

I've ran into an issue with time data that looks like a bug.

The script below works as expected:

################
$data << END
2020-07-01      1.1
2020-07-08      2.2
2020-07-15      3.3
END

set xdata time
set timefmt "%Y-%m-%d"
#set ydata time
plot $data u 1:2 w lp
###############

It produces a plot with correctly parsed and displayed dates on the x
axis, and the x range is also sensible.

But uncomment the `set ydata time` line and it fails:

################
         "x.plt" line 10: warning: Skipping data file with no valid
points

gnuplot> plot $data u 1:2 w lp
                              ^
         "x.plt" line 10: x range is invalid
##############

I can sort of understand that it fails because it can't parse the
second column as date - but then why does it report the *x* range as
invalid?

Even more spooky is what it thinks about the x range after the failed
plot:
##############
gnuplot> sh xr

	set xdata time
	set xrange [ * : * ] noreverse writeback  # (currently
["         warning: time value out of range
0000-35-32586":"         warning: time value out of range
0000-35-32586"] )
#############

If I run gnuplot with valgrind, it barfs some "Conditional jump or move
depends on uninitialised value(s)" errors, so some internal state is
definitely corrupted.

Best regards,
Peter Juhasz

New Release package - gnuplot version 5.4

[Ethan A M. <me...@uw...>]

Gnuplot version 5.4 - first release

I have placed a source tarball for 5.4 on SourceForge

   https://sf.net/projects/gnuplot/files/gnuplot/5.4.0/gnuplot-5.4.0.tar.gz

Release Notes with new feature list

   http://gnuplot.info/ReleaseNotes_5_4.html

Version 5.4.0 is the start of a new "major release" series.
It supersedes version 5.2, first released in September 2017.

I realize that there may be unresolved issues with building Windows binaries
for Windows 10, but I decided that it is not reasonable to hold up release any
longer for issues that appear to be more related to system configuration than to
the gnuplot source.  If necessary we can put out a Windows-specific patchlevel 1.
As always, I am totally reliant on and extremely grateful to the volunteers
who put together Windows binaries from a source release package.

        cheers, and happy plotting

                Ethan

Re: [PATCH v3] Support building development version without git

[Ethan A M. <me...@uw...>]

On Wednesday, 1 July 2020 17:31:03 PDT Henri Menke wrote:
> ---
> I've changed the commit format from %cs to %cd which is supported in
> older git versions as well.  I'd like to avoid using %ci because this
> requires an extra pipe to strip off the unnecessary time of day and
> timezone information which makes the code even more ugly and unreadable.

Looks good to me.

	Ethan

 
>  src/Makefile.am | 6 +++---
>  1 file changed, 3 insertions(+), 3 deletions(-)
> 
> diff --git a/src/Makefile.am b/src/Makefile.am
> index 1b0d8136e..0b261ea47 100644
> --- a/src/Makefile.am
> +++ b/src/Makefile.am
> @@ -107,12 +107,12 @@ endif
>  
>  DISTCLEANFILES = timestamp.h
>  BUILT_SOURCES = timestamp.h
> -git_current := $(shell cut -c6- $(top_srcdir)/.git/HEAD)
> -timestamp.h: $(top_srcdir)/.git/${git_current} Makefile
> +.PHONY: timestamp.h
> +timestamp.h: Makefile
>  	@echo Making $@
>  	@echo "#ifndef GNUPLOT_TIMEBASE_H_INCLUDED" >$@t
>  	@echo "#define GNUPLOT_TIMEBASE_H_INCLUDED" >>$@t
> -	@echo "const char gnuplot_date[] = \"`git --git-dir '$(top_srcdir)/.git' log -1 --format=%ci | cut -b-11`\";" >>$@t
> +	@echo "const char gnuplot_date[] = \"`git --git-dir '$(top_srcdir)/.git' log -1 --date=short --format=%cd || stat -c '%.10y' Makefile.am`\";" >>$@t
>  	@echo "#endif /* GNUPLOT_TIMEBASE_H_INCLUDED */" >> $@t
>  	@if cmp -s $@ $@t; then rm -f $@t; else mv $@t $@; fi