gnuplot-beta — Developers and beta testers of new gnuplot code

Re: Strange autoscaling with rgbimage and inverted yranges

[Ethan A M. <me...@uw...>]

On Saturday, 12 September 2020 21:07:58 PDT Dima Kogan wrote:
> Dima Kogan <gn...@di...> writes:
> 
> > Ethan A Merritt <me...@uw...> writes:
> >
> >> And even after tracking it that far I still don't understand why
> >> 'p' works and 'u' doesn't.     If you can figure that out I'd be 
> >> happier with declaring it fixed.
> >
> > In my tests just now 'p' and 'u' both show this issue. Are you 100% sure
> > that 'p' does NOT show the issue for you?
> 
> I just tried this with the x11, qt and wxt terminals, and in all 3 cases
> both 'u' and 'p' show this issue. I'm at the git HEAD as off a few days
> ago:
> 
>   commit 160d7511bcbdfd0d1943a624af5fc42504a9f84c
>   Author: Ethan A Merritt <merritt@u.washington.edu>
>   Date:   Thu Sep 3 20:20:23 2020 -0700
> 
>       new function (long)argb = rgbcolor("name")
> 
> The test is:
> 
> 1. start gnuplot
> 2. "load" the script in the first email in this thread
> 3. interactively zoom
> 4. 'u' or 'p'
> 
> I'm on GNU/Linux. If I can reproduce the behavior you're seeing ('p' and
> 'u' work differently), then I'll be happy to debug it.

Try this:

1) start gnuplot
2) load script
3) zoom
4) 'u'
   plot is now incorrectly scaled
5) 'n'
   shows zoom state from (3)
6) 'p'
   correctly unzooms to original plot

Now that I try more sequences I notice that replacing 'p' with 'u'
in step (6) also works correctly.  So maybe the issue is whether
the previous operation was 'n'?  Or there's some parity issue
with the count of operations? 

Regardless, I think autoscaling in the current HEAD now works
correctly for your test case.
I also think that it is incorrect for the unzoom operation to invoke
autoscaling at all, but that would require additional code changes.

	Ethan

Re: Strange autoscaling with rgbimage and inverted yranges

[Dima K. <gn...@di...>]

Dima Kogan <gn...@di...> writes:

> Ethan A Merritt <me...@uw...> writes:
>
>> And even after tracking it that far I still don't understand why
>> 'p' works and 'u' doesn't.     If you can figure that out I'd be 
>> happier with declaring it fixed.
>
> In my tests just now 'p' and 'u' both show this issue. Are you 100% sure
> that 'p' does NOT show the issue for you?

I just tried this with the x11, qt and wxt terminals, and in all 3 cases
both 'u' and 'p' show this issue. I'm at the git HEAD as off a few days
ago:

  commit 160d7511bcbdfd0d1943a624af5fc42504a9f84c
  Author: Ethan A Merritt <merritt@u.washington.edu>
  Date:   Thu Sep 3 20:20:23 2020 -0700

      new function (long)argb = rgbcolor("name")

The test is:

1. start gnuplot
2. "load" the script in the first email in this thread
3. interactively zoom
4. 'u' or 'p'

I'm on GNU/Linux. If I can reproduce the behavior you're seeing ('p' and
'u' work differently), then I'll be happy to debug it.

Re: Strange autoscaling with rgbimage and inverted yranges

[Dima K. <gn...@di...>]

Ethan A Merritt <me...@uw...> writes:

> And even after tracking it that far I still don't understand why
> 'p' works and 'u' doesn't.     If you can figure that out I'd be 
> happier with declaring it fixed.

In my tests just now 'p' and 'u' both show this issue. Are you 100% sure
that 'p' does NOT show the issue for you?

Re: Strange autoscaling with rgbimage and inverted yranges

[Ethan A M. <me...@uw...>]

On Friday, 11 September 2020 22:51:03 PDT Ethan A Merritt wrote:

> I _think_ that the problem involves this macro in axis.h
> 
> %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
> /* Simplest form of autoscaling (no check on autoscale constraints).
>  * Used by refresh_bounds() and refresh_3dbounds().
>  * Used also by autoscale_boxplot.
>  */
> #define autoscale_one_point(axis, x) do {\
>     if (axis->set_autoscale & AUTOSCALE_MIN && x < axis->min) \
>         axis->min = x; \
>     if (axis->set_autoscale & AUTOSCALE_MAX && x > axis->max) \
>         axis->max = x; \
>     } while (0);
> %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
> 
> as called from plot2d.c:refresh_bounds() when zooming volatile data.
> 
> Note that the macro does not consider the case of reversed axes.
> Adding a test for (axis->min < axis->max) seems to make your
> test case work.   See attached patch.

Better patch (but still not fully correct)

	Ethan

Re: Strange autoscaling with rgbimage and inverted yranges

[Ethan A M. <me...@uw...>]

On Friday, 11 September 2020 20:13:33 PDT Dima Kogan wrote:

> In any case, this worked in 5.2. I just did a bisection, and I can now
> see why this seemed familiar. The commit that broke it:
> 
>   https://sourceforge.net/p/gnuplot/gnuplot-main/ci/36a2407e50/
> 
> The person who broke it was me! But despite what git says, I have very
> little recollection of any of this. Ethan: do you have a good sense of
> what's happening here, and can you easily fix it now? If not, I can poke
> at it, and figure it out.
> 
> dima

I am puzzled because I do not understand the difference between
returning to the top level zoom state by pressing 'p' to step backwards
through the stack of zoom operations and pressing 'u' to jump back to the
top of the stack.  Either way it should be applying the same state
(confirmed in the debugger) but for some reason that I don't understand
the former works correcly and the latter doesn't.

I _think_ that the problem involves this macro in axis.h

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
/* Simplest form of autoscaling (no check on autoscale constraints).
 * Used by refresh_bounds() and refresh_3dbounds().
 * Used also by autoscale_boxplot.
 */
#define autoscale_one_point(axis, x) do {\
    if (axis->set_autoscale & AUTOSCALE_MIN && x < axis->min) \
        axis->min = x; \
    if (axis->set_autoscale & AUTOSCALE_MAX && x > axis->max) \
        axis->max = x; \
    } while (0);
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

as called from plot2d.c:refresh_bounds() when zooming volatile data.

Note that the macro does not consider the case of reversed axes.
Adding a test for (axis->min < axis->max) seems to make your
test case work.   See attached patch.

The unpatched code does the wrong thing for reversed axes.
The patched code does nothing at all for reversed axes,
is it probably wrong also but for some case other than yours.
A more complete fix for the macro is wanted.

And even after tracking it that far I still don't understand why
'p' works and 'u' doesn't.     If you can figure that out I'd be 
happier with declaring it fixed.

                 Ethan
-- 
Ethan A Merritt
Biomolecular Structure Center,  K-428 Health Sciences Bldg
MS 357742,   University of Washington, Seattle 98195-7742

Re: Strange autoscaling with rgbimage and inverted yranges

[Dima K. <gn...@di...>]

Ethan A Merritt <me...@uw...> writes:

> I think the issue is that because your example uses in-line data it
> cannot be re-read. The program tries to work around this by reusing
> the contents of the various internal data structures but in this case
> the workaround fails. Possibly fixable, but I think the primary lesson
> is that data blocks are much preferred to in-line data if you are
> going to be manipulating/re-drawing the plot.

Hi.

100% of my fairly heavy usage of gnuplot uses inline data. The
particular case in this report is fairly common too: when doing any sort
of computer vision that annotates an image, you want to display the
source image with a flipped y axis, since the origin pixel is at the
top-left.

In any case, this worked in 5.2. I just did a bisection, and I can now
see why this seemed familiar. The commit that broke it:

  https://sourceforge.net/p/gnuplot/gnuplot-main/ci/36a2407e50/

The person who broke it was me! But despite what git says, I have very
little recollection of any of this. Ethan: do you have a good sense of
what's happening here, and can you easily fix it now? If not, I can poke
at it, and figure it out.

dima

Re: [bug] plot bins does not clip output to plot area

[Achim G. <Str...@ne...>]

Ethan A Merritt writes:
>> I'm using it as a way to average over a large number of samples that are
>> sampled equidistantly (kernel density is way slower obviously). The
>> documentation doesn't really tell how it is implemented or supposed to
>> work, so I've experimented a bit to find out.
>
> I take the blame for any failures in either the implementation or
> the documentation of the "bins" option, since I wrote both.
> I would be happy to amend the code or the description to cover
> uses that I didn't anticipate.

No blame to go around… any feature is prone to be (mis)used in ways
never intended, once it's there.  :-)

> Since I envisioned it as a histogramming tool, it did not occur to
> me that users would want to plot "with lines" rather than using
> boxes or impulses.   Therefore I did not think about or test the
> clipping behaviour.

No good deed goes unpunished, as they say.

> As it happens, the routines that draw boxes or impulses do the clipping as
> they draw each one.  The routine that draws "with lines" assumes that the
> inrange/outrange/undefined status of each point has already been flagged,
> and only considers clipping if a particular line segment changes state.
> This is so that successive in range points are drawn as a single poly-line
> with smooth joins rather than being split into individual segments.
> Normally the inrange/outrange flag is set on data entry, but those flags
> apply to the original data points rather than to the binned totals.
> I have now added a separate pass to re-check the binned data against
> yrange so that clipping works as you expect it to.
> (New code added for both 5.4 and 5.5).

Great, I'll check it out later.

> I remain a little uneasy about the use of negative values in the weighting
> column, although I don't have a specific example in mind that will fail.

So far I have not yet encountered any problems with that and I have
likely given this part quite a bit of exercise as most of the data I'm
plotting is roughly centered about zero.

> Thanks for the explanation of what you are using it for.
> I agree that in that mode "bins" is similar to a kernel density model
> that uses a delta function rather than a Gaussian kernel.

I would suggest that it would be welcome if one could specify the kernel
function and/or provide a way to implement general FIR filtering on the
original data.  While the moving average trick is cute, it is also quite
slow and cumbersome once you need to average over a large number of
samples.

> I may not have phrased that well.  It weights the _contribution_ of each
> sample by the value in the seconds column.  If no second column is
> provided, the weight is 1.

That seems a more accessible description to me, thanks.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Factory and User Sound Singles for Waldorf Blofeld:
http://Synth.Stromeko.net/Downloads.html#WaldorfSounds

Re: [bug] plot bins does not clip output to plot area

[Ethan A M. <me...@uw...>]

> I'm using it as a way to average over a large number of samples that are
> sampled equidistantly (kernel density is way slower obviously). The
> documentation doesn't really tell how it is implemented or supposed to
> work, so I've experimented a bit to find out.

I take the blame for any failures in either the implementation or
the documentation of the "bins" option, since I wrote both.
I would be happy to amend the code or the description to cover
uses that I didn't anticipate.

Since I envisioned it as a histogramming tool, it did not occur to
me that users would want to plot "with lines" rather than using
boxes or impulses.   Therefore I did not think about or test the
clipping behaviour.

As it happens, the routines that draw boxes or impulses do the clipping as
they draw each one.  The routine that draws "with lines" assumes that the
inrange/outrange/undefined status of each point has already been flagged,
and only considers clipping if a particular line segment changes state.
This is so that successive in range points are drawn as a single poly-line
with smooth joins rather than being split into individual segments.
Normally the inrange/outrange flag is set on data entry, but those flags
apply to the original data points rather than to the binned totals.
I have now added a separate pass to re-check the binned data against
yrange so that clipping works as you expect it to.
(New code added for both 5.4 and 5.5).

I remain a little uneasy about the use of negative values in the weighting
column, although I don't have a specific example in mind that will fail.

Thanks for the explanation of what you are using it for.
I agree that in that mode "bins" is similar to a kernel density model
that uses a delta function rather than a Gaussian kernel.

	cheers,

		Ethan

On Friday, 11 September 2020 11:59:28 PDT ASSI wrote:
> Ethan A Merritt writes:
> 
> > If there is a second column of data this is interpreted as a weight.
> 
> Let's assume the sample spacing is 1 and no samples are missing, then
> you'll get the sum over the bin width.  Different sampling density just
> scales the result.  Dividing by the number of samples gives you the
> average.

Correct.

[snip]

> 
> > Your test script provides no "using" specifier, however, so the plot
> > command draws values from column 1 (essentially the numbers 0 to 100)
> > and weights each one by the value in the second column (sin(x)).
> 
> It doesn't do that either or the result would be a linearly rising
> function with a sin riding on top of it.  It reproduces the function if
> I arrange the binwidth to contain just a single sample, so clearly the x
> column isn't used directly.

I may not have phrased that well.  It weights the _contribution_ of each
sample by the value in the seconds column.  If no second column is
provided, the weight is 1.

Re: [bug] plot bins does not clip output to plot area

[ASSI <Str...@ne...>]

Ethan A Merritt writes:
> I am confused as to what exactly you are trying to plot.
> The intent of the "bins" option is similar to the "smooth frequency" option.
> It derives an approximation to the distribution of values read from
> a single column of data.

I'm using it as a way to average over a large number of samples that are
sampled equidistantly (kernel density is way slower obviously).  The
documentation doesn't really tell how it is implemented or supposed to
work, so I've experimented a bit to find out.

> If there is a second column of data this is interpreted as a weight.

Let's assume the sample spacing is 1 and no samples are missing, then
you'll get the sum over the bin width.  Different sampling density just
scales the result.  Dividing by the number of samples gives you the
average.

> To show the distribution of values for f(x) over a given range of x, 
> the plotting command could be
>
>       plot sample [xmin:xmax:increment] '+' using (f(x)) bins {binwidth=foo}
>
> If I rewrite your test case in this form, shown as a histogram, it becomes
>
>       plot sample [0:100:1] '+' using (sin(x)) bins binwidth=0.1 with boxes
>
> The autoscaling works correctly on both x and y, and remains correct if 
> I plot instead "with lines". Note that this is a single column of data, so
> the samples have uniform weights.

We've already established that "with boxes" and "with impulses" would
clip the plotted data correctly, but both "with points" and "with lines"
doesn't.  That's the only thing I think is a bug and needs fixing.

> Your test case creates this same set of samples but stores it in a 
> data block with two columns   x and f(x).   To reproduce the plot from
> the stored samples the command would be
>
>      plot $data using 2 bins binwidth=0.1

That's different from what I'm trying to get, though; and it doesn't
reproduce the data either.  It does produce a distribution of the
samples, resolved into 0.1 wide buckets.

> Your test script provides no "using" specifier, however, so the plot
> command draws values from column 1 (essentially the numbers 0 to 100)
> and weights each one by the value in the second column (sin(x)).

It doesn't do that either or the result would be a linearly rising
function with a sin riding on top of it.  It reproduces the function if
I arrange the binwidth to contain just a single sample, so clearly the x
column isn't used directly.  It integrates over the bin if I arrange the
bin to contain a larger number of samples (10 in my example).  So if I
divide the result by the number of samples I get an average over the
bin.

> The presence of negative weights in particular confuses the program.
> I don't rule out the possibility that there is a legitimate use for such
> but the code assumes positive weights, typically either 1 or a set of
> fractional values summing to 1 so that the distribution is normalized.
> Also since the range of values is [0:100], using a bin width of 0.1 means
> that the vast majority of bins are empty.
> This produces the strange-looking plot you saw.

The plot is the result of most of the bins having an integral either
far above 0.1 or below -0.1, so it should have been clipped at the axis
borders, but it wasn't.

> If your test case is intended as a simple stand-in for some real world
> data that does in fact use negative weights, please clarify the
> expected properties of the resulting distribution and I will try to
> modify the program's expectations to allow it.  As it stands, the 
> programs expectations about y values lead it to totally mis-assign
> all the points as "inrange" and therefore not clipped.

Again, I'm not using it this way and the actual code doesn't seem to
work the way you described it either.  Now that you've described how to
get a binned distribution/histogram I do have uses for that too, but that's
not what I was having problems with.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Wavetables for the Waldorf Blofeld:
http://Synth.Stromeko.net/Downloads.html#BlofeldUserWavetables

Re: [bug] plot bins does not clip output to plot area

[Ethan A M. <me...@uw...>]

I am confused as to what exactly you are trying to plot.
The intent of the "bins" option is similar to the "smooth frequency" option.
It derives an approximation to the distribution of values read from
a single column of data.
If there is a second column of data this is interpreted as a weight.

To show the distribution of values for f(x) over a given range of x,
the plotting command could be

      plot sample [xmin:xmax:increment] '+' using (f(x)) bins {binwidth=foo}

If I rewrite your test case in this form, shown as a histogram, it becomes

      plot sample [0:100:1] '+' using (sin(x)) bins binwidth=0.1 with boxes

The autoscaling works correctly on both x and y, and remains correct if
I plot instead "with lines". Note that this is a single column of data, so
the samples have uniform weights.

Your test case creates this same set of samples but stores it in a
data block with two columns   x and f(x).   To reproduce the plot from
the stored samples the command would be

     plot $data using 2 bins binwidth=0.1

Your test script provides no "using" specifier, however, so the plot
command draws values from column 1 (essentially the numbers 0 to 100)
and weights each one by the value in the second column (sin(x)).
The presence of negative weights in particular confuses the program.
I don't rule out the possibility that there is a legitimate use for such
but the code assumes positive weights, typically either 1 or a set of
fractional values summing to 1 so that the distribution is normalized.
Also since the range of values is [0:100], using a bin width of 0.1 means
that the vast majority of bins are empty.
This produces the strange-looking plot you saw.

If your test case is intended as a simple stand-in for some real world
data that does in fact use negative weights, please clarify the
expected properties of the resulting distribution and I will try to
modify the program's expectations to allow it.  As it stands, the
programs expectations about y values lead it to totally mis-assign
all the points as "inrange" and therefore not clipped.

       Ethan

Re: [bug] plot bins does not clip output to plot area

[Achim G. <Str...@ne...>]

Ethan A Merritt writes:
> The y clipping seems to fail specifically when plotting "with lines".
> It clips properly for "with impulses" or "with boxes", right?

I haven't tried these two specifically, but "with points" also plots
outside the boundary.  OK, impulses and boxes are correctly clipped.
Things aer more clearly visible if you additionally

 set tmargin at screen 0.8
 set bmargin at screen 0.2

in my reproducer before the plot command.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptations for KORG EX-800 and Poly-800MkII V0.9:
http://Synth.Stromeko.net/Downloads.html#KorgSDada

Re: [bug] plot bins does not clip output to plot area

[Ethan A M. <me...@uw...>]

On Wednesday, 9 September 2020 22:36:56 PDT ASSI wrote:
> Ethan A Merritt writes:
> > The program extends auto-scaled ranges to the next axis tic unless
> > you tell it not to.  If you do not want the range to be extended, tell it
> 
> The y axis is not autoscaled in my example and there is no clipping at
> all, which gets obvious if you do multiplots.

Ah, sorry.  I misunderstood.

The y clipping seems to fail specifically when plotting "with lines".
It clips properly for "with impulses" or "with boxes", right?

	Ethan

> 
> 
> Regards,
> Achim.
> 

Re: [bug] plot bins does not clip output to plot area

[ASSI <Str...@ne...>]

Ethan A Merritt writes:
> The program extends auto-scaled ranges to the next axis tic unless
> you tell it not to.  If you do not want the range to be extended, tell it

The y axis is not autoscaled in my example and there is no clipping at
all, which gets obvious if you do multiplots.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptations for Waldorf Q V3.00R3 and Q+ V3.54R2:
http://Synth.Stromeko.net/Downloads.html#WaldorfSDada

Re: [bug] plot bins does not clip output to plot area

[Ethan A M. <me...@uw...>]

On Wednesday, 9 September 2020 11:05:16 PDT Achim Gratz wrote:
> 
> To reproduce:
> 
> gnuplot> set table $data
> gnuplot> plot [0:100] sin(x)
> gnuplot> unset table
> gnuplot> plot [*:*][-0.1:0.1] $data w lines bins binwidth=0.1
> 
> 
> Regards,
> Achim.

The program extends auto-scaled ranges to the next axis tic unless
you tell it not to.  If you do not want the range to be extended, tell it

    set xrange noextend       #version 5.4
or
    set xrange [*:*] noextend #version 5.2

	cheers,

		Ethan
 

[bug] plot bins does not clip output to plot area

[Achim G. <Str...@ne...>]

To reproduce:

gnuplot> set table $data
gnuplot> plot [0:100] sin(x)
gnuplot> unset table
gnuplot> plot [*:*][-0.1:0.1] $data w lines bins binwidth=0.1


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Factory and User Sound Singles for Waldorf rackAttack:
http://Synth.Stromeko.net/Downloads.html#WaldorfSounds

Re: Strange autoscaling with rgbimage and inverted yranges

[Ethan A M. <me...@uw...>]

On Wednesday, 9 September 2020 02:07:50 PDT Dima Kogan wrote:
> Hi. I vaguely recall mentioning this already, but I don't see it in my
> email. Apologies if this has already been reported. I have this script:
> 
>     set yrange [:] reverse
>     set size ratio -1
>     plot "/tmp/gray.jpg" binary filetype=auto flipy with rgbimage,'-'   using 1:2 notitle with points pt 3 ps 3 
[snip]
>     e
> 
> I.e. we plot some points on top of the attached image. The contents of
> the image don't matter. Here we just have a gray field. I'm using the
> latest master branch, but I belive 5.4 does this too.
> 
> 1. Running that script in an interactive terminal (I tried x11 and qt),
>    the plot pops up just fine: the autoscale considered the image AND the
>    points
> 
> 2. We interactively zoom on any area with the mouse. This works too
> 
> 3. We press 'u' to zoom out. The expectation is that we get back to the
>    full view after step 1. Instead we get to a new view, zoomed-in on
>    the points only, NOT on the image.

Hmm.  
But if you press 'p' for "previous" the unzoom works correctly.
That is probably a clue.

Also, if the inline data is replaced by a datablock the problem goes away.

I think the issue is that because your example uses in-line data it
cannot be re-read.  The program tries to work around this by reusing
the contents of the various internal data structures but in this case the
workaround fails.  Possibly fixable, but I think the primary lesson is
that data blocks are much preferred to in-line data if you are going to be
manipulating/re-drawing the plot.

	Ethan

> Something about the reverse yrange is confusing I think. Have I reported
> this already?
> 
> Thanks.

Strange autoscaling with rgbimage and inverted yranges

[Dima K. <gn...@di...>]

Hi. I vaguely recall mentioning this already, but I don't see it in my
email. Apologies if this has already been reported. I have this script:

    set yrange [:] reverse
    set size ratio -1
    plot "/tmp/gray.jpg" binary filetype=auto flipy with rgbimage,'-'   using 1:2 notitle with points pt 3 ps 3 
    1854.0 1599.0
    1596.0 1568.0
    2170.0 1626.0
    3649.0 1762.0
    1937.0 1604.0
    1243.0 1544.0
    2412.0 1648.0
    886.0 1508.0
    141.0 1436.0
    3088.0 1709.0
    292.0 1455.0
    2311.0 1639.0
    647.0 1484.0
    2662.0 1672.0
    1105.0 1529.0
    803.0 1499.0
    4218.0 1811.0
    731.0 1498.0
    3743.0 1769.0
    1371.0 1555.0
    4100.0 1800.0
    1446.0 1544.0
    2092.0 1587.0
    2863.0 1669.0
    2930.0 1681.0
    3197.0 1702.0
    3341.0 1709.0
    3483.0 1733.0
    e

I.e. we plot some points on top of the attached image. The contents of
the image don't matter. Here we just have a gray field. I'm using the
latest master branch, but I belive 5.4 does this too.

1. Running that script in an interactive terminal (I tried x11 and qt),
   the plot pops up just fine: the autoscale considered the image AND the
   points

2. We interactively zoom on any area with the mouse. This works too

3. We press 'u' to zoom out. The expectation is that we get back to the
   full view after step 1. Instead we get to a new view, zoomed-in on
   the points only, NOT on the image.

Something about the reverse yrange is confusing I think. Have I reported
this already?

Thanks.

Re: 5.4.rc2 second release candidate for gnuplot version 5.4

[Shigeharu T. <sh...@ie...>]

shige 08/28 2020
----------------

Tait wrote at 2020-06-09: 

| I ran gp54rc1-win64-mingw.exe downloaded from SF.
...
| I wanted to try the Windows terminal, too, but it hangs for a
| few seconds and then crashes, seemingly no matter what I try to
| plot.

Currently, I can't download the binary from SF. 

Well, I build windows binary from gnuplot source version 5.4.0 by
msys2-mingw64, and I ran it with wxt terminal as 
  gnuplot.exe -e "set term wxt; cd 'demo' ; load 'all.dem'"
then it works fine. 
For windows terminal, certainly
  gnuplot.exe -e "set term win; cd 'demo' ; load 'all.dem'"
crashes as reported. 
But, when I specify the window size explicitly as
  gnuplot.exe -e "set term win size 640,400; cd 'demo' ; load 'all.dem'"
it works fine. Please try.

+========================================================+
 Shigeharu TAKENO     NIigata Institute of Technology
                       kashiwazaki,Niigata 945-1195 JAPAN
 sh...@ie...   TEL(&FAX): +81-257-22-8161
+========================================================+

Re: [PATCH] cairolatex: respect transparency setting

[Henri M. <hen...@gm...>]

On 20/08/20, 21:31, Ethan A Merritt wrote:
> On Thursday, 20 August 2020 20:17:44 PDT Henri Menke wrote:
> > On 20/08/20, 20:01, Ethan A Merritt wrote:
> > > On Thursday, 20 August 2020 18:34:54 PDT Henri Menke wrote:
> > > > Currently the cairolatex terminal does not respect transparency options
> > > > given in the terminal setting.  In this minimal example the
> > > > notransparent option is ignored:
> > > > 
> > > >     set terminal cairolatex png notransparent
> > > >     set output "test.tex"
> > > >     plot sin(x)
> > > > 
> > > > Running this through gnuplot and then using `file' reveals that the
> > > > alpha channel is still present:
> > > > 
> > > >     $ gnuplot test.gnuplot && file test.png
> > > >     test.png: PNG image data, 1500 x 900, 8-bit/color RGBA, non-interlace
> > > 
> > > I think the raionale for the current code is that the png image created for
> > > inclusion in a LaTeX document must have a transparent background in order
> > > to support the "back" and "behind" attributes for text elements, since they
> > > will be drawn by the latex engine and then covered by inclusion of the
> > > graphics image.  The structure of the latex processing is:
> > > 
> > >     render text elements that are "behind" or "back"
> > >     \includegraphics{gnuplot_figure}
> > >     render text elements that are "front"
> > > 
> > > If you use a solid color background in the graphics image, it will hide
> > > any previously rendered text elements.
> > 
> > The reason why I'm looking into this is that currently it is impossible
> > to generate PDF/A when using plots generated by gnuplot.  PDF/A doesn't
> > allow transparency, so I'm trying to remove all of it (or at least make
> > it optional).  As noted in the patch, the default behaviour of using
> > transparency remains, so this is entirely optional.
> 
> I am researching this as I type, so please bear with me.
> 
> From Wikipedia I see
> 	Transparent objects and layers (Optional Content Groups) are 
> 	forbidden in PDF/A-1, but are allowed in PDF/A-2.
> PDF/A-1 was published in 2005; PDF/A-2 in 2011; PDF/A-3 in 2012;
> So maybe this is no longer a limitation?
> 
> From my own knowledge, there are commonly available tools that 
> render the transparency in a PDF document once, then save the result
> as an embedded image that no longer contains transparency.
> For example if you use gnuplot to create a PostScript file directly it cannot
> use transparency because PostScript itself does not support transparency.
> But if you use gnuplot to create a PDF file with transparency and then
> convert it
> 	pdf2ps input.pdf output.ps
> you get a PostScript file in which the transparency is "baked in" rather
> than interpreted.   Logically then you could take that PostScript file
> and convert it back to PDF, keeping the pre-rendered image that no
> longer uses an alpha channel. 
> 	ps2pdf output.ps new.pdf
> Whether that particular pair of tools produces a PDF/A compliant
> file I do not know.

This is actually a pretty good proposal.  In my actual plot I now have
added

    set terminal cairolatex pdf standalone header '\usepackage[a-1b]{pdfx}'
    set output "test.tex"
    # ...
    unset output
    !pdf2ps test-inc.pdf test-inc.ps
    !ps2pdf -dPDFA=1 test-inc.ps test-inc.pdf
    !pdflatex --interaction=nonstopmode test.tex

and the resulting document validates correctly with VeraPDF.  It's a bit
ugly and I don't know whether the ghostscript roundtrip would correctly
preserve fonts, but since the background image doesn't contain fonts, I
don't care.  Note that adding the -dPDFA=1 flag will make the PDF
structurally PDF/A compliant, but will not add metadata or color
profiles, so test-inc.pdf does not validate.  These extras are handled
by the pdfx LaTeX package.

To get my plots with PNG background compliant I use the patch from this
thread together with your suggestion `set tics front'.  Another option
would be to run mogrify to remove the transparency.

    set terminal cairolatex png standalone header '\usepackage[a-1b]{pdfx}'
    set output "test.tex"
    # ...
    unset output
    !mogrify -alpha remove test-inc.png
    !pdflatex --interaction=nonstopmode test.tex

So bottomlined my thesis now validates as PDF/A-1b, which is really
nice.

Cheers, Henri

> 
>    best,
> 
> 	Ethan
> 
> > 
> > Kind regards,
> > Henri
> > 
> > > 
> > > 	Ethan
> > > 
> > > > 
> > > > With this patch the transparency setting will be honoured.  The current
> > > > default setting of implicitly assuming transparency is unaltered.
> > > > ---
> > > >  term/cairo.trm | 2 +-
> > > >  1 file changed, 1 insertion(+), 1 deletion(-)
> > > > 
> > > > diff --git a/term/cairo.trm b/term/cairo.trm
> > > > index ee748dfe2..3ef76ecc6 100644
> > > > --- a/term/cairo.trm
> > > > +++ b/term/cairo.trm
> > > > @@ -849,7 +849,7 @@ void cairotrm_graphics()
> > > >  	plot.background.g = cairo_params->background.g;
> > > >  	plot.background.b = cairo_params->background.b;
> > > >  	gp_cairo_set_background(cairo_params->background);
> > > > -	if (ISCAIROLATEX || cairo_params->transparent)
> > > > +	if (cairo_params->transparent)
> > > >  		gp_cairo_clear_background(&plot);
> > > >  	else
> > > >  		gp_cairo_solid_background(&plot);
> > > > 
> > > 
> > > 
> > > 
> > > 
> 
> 
> 

Re: [BUG] pdfcairo notransparent has transparency

[Ethan A M. <me...@uw...>]

On Thursday, 20 August 2020 21:42:51 PDT Henri Menke wrote:
> Dear all,
> 
> With regards to my recent patch to optionally disable transparency on
> cairolatex, I stumbled upon the fact that the PDF output of the cairo
> terminal does not properly respect the notransparent option.  Consider
> the following minimal example:
> 
>     set terminal pdfcairo notransparent
>     set output "test.pdf"
>     plot sin(x)
> 
> When running this, the page is still encapsulated in a transparency
> group.  

The "{no}transparent" option used by several gnuplot terminals
specifically refers to the plot background.  It does not, and was never
intended to, disable transparency in the plot proper.   The two things
are orthogonal.   Consider for example
 
    set term png notruecolor transparent 

This sets the background to be transparent but the plot contains no
alpha channel because it uses indexed color. 
Contrast this with

    set term png truecolor backgound "grey"

This sets the background to be opaque (=notransparent),
but the plot itself uses 24bit RGB color + alpha channel.

	Ethan



> This breaks PDF/A compatibility.
> 
>     $ gnuplot test.gnuplot
>     $ grep -aC6 'Transparency' test.pdf
>     << /Type /Page % 1
>        /Parent 1 0 R
>        /MediaBox [ 0 0 360 216 ]
>        /Contents 4 0 R
>        /Group <<
>           /Type /Group
>           /S /Transparency
>           /I true
>           /CS /DeviceRGB
>        >>
>        /Resources 3 0 R
>     >>
>     endobj
> 
> I'm not entire sure and I haven't tested this yet, but the cairo
> terminal uses the gp_cairo_set_color function from
> src/wxterminal/gp_cairo.c which in turn calls cairo_set_source_rgba.
> This explicit call for transparency might be the cause for cairo
> inserting a transparency group, even if all alpha values are zero or
> one.
> 
> Cheers, Henri
> 

[BUG] pdfcairo notransparent has transparency

[Henri M. <hen...@gm...>]

Dear all,

With regards to my recent patch to optionally disable transparency on
cairolatex, I stumbled upon the fact that the PDF output of the cairo
terminal does not properly respect the notransparent option.  Consider
the following minimal example:

    set terminal pdfcairo notransparent
    set output "test.pdf"
    plot sin(x)

When running this, the page is still encapsulated in a transparency
group.  This breaks PDF/A compatibility.

    $ gnuplot test.gnuplot
    $ grep -aC6 'Transparency' test.pdf
    << /Type /Page % 1
       /Parent 1 0 R
       /MediaBox [ 0 0 360 216 ]
       /Contents 4 0 R
       /Group <<
          /Type /Group
          /S /Transparency
          /I true
          /CS /DeviceRGB
       >>
       /Resources 3 0 R
    >>
    endobj

I'm not entire sure and I haven't tested this yet, but the cairo
terminal uses the gp_cairo_set_color function from
src/wxterminal/gp_cairo.c which in turn calls cairo_set_source_rgba.
This explicit call for transparency might be the cause for cairo
inserting a transparency group, even if all alpha values are zero or
one.

Cheers, Henri

Re: [PATCH] cairolatex: respect transparency setting

[Ethan A M. <me...@uw...>]

On Thursday, 20 August 2020 20:17:44 PDT Henri Menke wrote:
> On 20/08/20, 20:01, Ethan A Merritt wrote:
> > On Thursday, 20 August 2020 18:34:54 PDT Henri Menke wrote:
> > > Currently the cairolatex terminal does not respect transparency options
> > > given in the terminal setting.  In this minimal example the
> > > notransparent option is ignored:
> > > 
> > >     set terminal cairolatex png notransparent
> > >     set output "test.tex"
> > >     plot sin(x)
> > > 
> > > Running this through gnuplot and then using `file' reveals that the
> > > alpha channel is still present:
> > > 
> > >     $ gnuplot test.gnuplot && file test.png
> > >     test.png: PNG image data, 1500 x 900, 8-bit/color RGBA, non-interlace
> > 
> > I think the raionale for the current code is that the png image created for
> > inclusion in a LaTeX document must have a transparent background in order
> > to support the "back" and "behind" attributes for text elements, since they
> > will be drawn by the latex engine and then covered by inclusion of the
> > graphics image.  The structure of the latex processing is:
> > 
> >     render text elements that are "behind" or "back"
> >     \includegraphics{gnuplot_figure}
> >     render text elements that are "front"
> > 
> > If you use a solid color background in the graphics image, it will hide
> > any previously rendered text elements.
> 
> The reason why I'm looking into this is that currently it is impossible
> to generate PDF/A when using plots generated by gnuplot.  PDF/A doesn't
> allow transparency, so I'm trying to remove all of it (or at least make
> it optional).  As noted in the patch, the default behaviour of using
> transparency remains, so this is entirely optional.

I am researching this as I type, so please bear with me.

>From Wikipedia I see
	Transparent objects and layers (Optional Content Groups) are 
	forbidden in PDF/A-1, but are allowed in PDF/A-2.
PDF/A-1 was published in 2005; PDF/A-2 in 2011; PDF/A-3 in 2012;
So maybe this is no longer a limitation?

>From my own knowledge, there are commonly available tools that 
render the transparency in a PDF document once, then save the result
as an embedded image that no longer contains transparency.
For example if you use gnuplot to create a PostScript file directly it cannot
use transparency because PostScript itself does not support transparency.
But if you use gnuplot to create a PDF file with transparency and then
convert it
	pdf2ps input.pdf output.ps
you get a PostScript file in which the transparency is "baked in" rather
than interpreted.   Logically then you could take that PostScript file
and convert it back to PDF, keeping the pre-rendered image that no
longer uses an alpha channel. 
	ps2pdf output.ps new.pdf
Whether that particular pair of tools produces a PDF/A compliant
file I do not know.

   best,

	Ethan

> 
> Kind regards,
> Henri
> 
> > 
> > 	Ethan
> > 
> > > 
> > > With this patch the transparency setting will be honoured.  The current
> > > default setting of implicitly assuming transparency is unaltered.
> > > ---
> > >  term/cairo.trm | 2 +-
> > >  1 file changed, 1 insertion(+), 1 deletion(-)
> > > 
> > > diff --git a/term/cairo.trm b/term/cairo.trm
> > > index ee748dfe2..3ef76ecc6 100644
> > > --- a/term/cairo.trm
> > > +++ b/term/cairo.trm
> > > @@ -849,7 +849,7 @@ void cairotrm_graphics()
> > >  	plot.background.g = cairo_params->background.g;
> > >  	plot.background.b = cairo_params->background.b;
> > >  	gp_cairo_set_background(cairo_params->background);
> > > -	if (ISCAIROLATEX || cairo_params->transparent)
> > > +	if (cairo_params->transparent)
> > >  		gp_cairo_clear_background(&plot);
> > >  	else
> > >  		gp_cairo_solid_background(&plot);
> > > 
> > 
> > 
> > 
> > 

Re: [PATCH] cairolatex: respect transparency setting

[Henri M. <hen...@gm...>]

On 20/08/20, 20:01, Ethan A Merritt wrote:
> On Thursday, 20 August 2020 18:34:54 PDT Henri Menke wrote:
> > Currently the cairolatex terminal does not respect transparency options
> > given in the terminal setting.  In this minimal example the
> > notransparent option is ignored:
> > 
> >     set terminal cairolatex png notransparent
> >     set output "test.tex"
> >     plot sin(x)
> > 
> > Running this through gnuplot and then using `file' reveals that the
> > alpha channel is still present:
> > 
> >     $ gnuplot test.gnuplot && file test.png
> >     test.png: PNG image data, 1500 x 900, 8-bit/color RGBA, non-interlace
> 
> I think the raionale for the current code is that the png image created for
> inclusion in a LaTeX document must have a transparent background in order
> to support the "back" and "behind" attributes for text elements, since they
> will be drawn by the latex engine and then covered by inclusion of the
> graphics image.  The structure of the latex processing is:
> 
>     render text elements that are "behind" or "back"
>     \includegraphics{gnuplot_figure}
>     render text elements that are "front"
> 
> If you use a solid color background in the graphics image, it will hide
> any previously rendered text elements.

The reason why I'm looking into this is that currently it is impossible
to generate PDF/A when using plots generated by gnuplot.  PDF/A doesn't
allow transparency, so I'm trying to remove all of it (or at least make
it optional).  As noted in the patch, the default behaviour of using
transparency remains, so this is entirely optional.

Kind regards,
Henri

> 
> 	Ethan
> 
> > 
> > With this patch the transparency setting will be honoured.  The current
> > default setting of implicitly assuming transparency is unaltered.
> > ---
> >  term/cairo.trm | 2 +-
> >  1 file changed, 1 insertion(+), 1 deletion(-)
> > 
> > diff --git a/term/cairo.trm b/term/cairo.trm
> > index ee748dfe2..3ef76ecc6 100644
> > --- a/term/cairo.trm
> > +++ b/term/cairo.trm
> > @@ -849,7 +849,7 @@ void cairotrm_graphics()
> >  	plot.background.g = cairo_params->background.g;
> >  	plot.background.b = cairo_params->background.b;
> >  	gp_cairo_set_background(cairo_params->background);
> > -	if (ISCAIROLATEX || cairo_params->transparent)
> > +	if (cairo_params->transparent)
> >  		gp_cairo_clear_background(&plot);
> >  	else
> >  		gp_cairo_solid_background(&plot);
> > 
> 
> 
> 
> 

Re: [PATCH] cairolatex: respect transparency setting

[Ethan A M. <me...@uw...>]

On Thursday, 20 August 2020 19:34:40 PDT Henri Menke wrote:
> Now there seems to a problem with ordering of the elements in the
> resulting TeX file.  The ticklabels are behind the background image.
> Even if transparency is enabled, this is probably a bad idea.  I'll
> investigate.

That is controlled by a generic setting, not specific to the latex terminals:

    set tics {front|back}

The default is "back".

	Ethan


> 
> Cheers, Henri
> 
> On 21/08/20, 13:34, Henri Menke wrote:
> > Currently the cairolatex terminal does not respect transparency options
> > given in the terminal setting.  In this minimal example the
> > notransparent option is ignored:
> > 
> >     set terminal cairolatex png notransparent
> >     set output "test.tex"
> >     plot sin(x)
> > 
> > Running this through gnuplot and then using `file' reveals that the
> > alpha channel is still present:
> > 
> >     $ gnuplot test.gnuplot && file test.png
> >     test.png: PNG image data, 1500 x 900, 8-bit/color RGBA, non-interlace
> > 
> > With this patch the transparency setting will be honoured.  The current
> > default setting of implicitly assuming transparency is unaltered.
> > ---
> >  term/cairo.trm | 2 +-
> >  1 file changed, 1 insertion(+), 1 deletion(-)
> > 
> > diff --git a/term/cairo.trm b/term/cairo.trm
> > index ee748dfe2..3ef76ecc6 100644
> > --- a/term/cairo.trm
> > +++ b/term/cairo.trm
> > @@ -849,7 +849,7 @@ void cairotrm_graphics()
> >  	plot.background.g = cairo_params->background.g;
> >  	plot.background.b = cairo_params->background.b;
> >  	gp_cairo_set_background(cairo_params->background);
> > -	if (ISCAIROLATEX || cairo_params->transparent)
> > +	if (cairo_params->transparent)
> >  		gp_cairo_clear_background(&plot);
> >  	else
> >  		gp_cairo_solid_background(&plot);
> 

Re: [PATCH] cairolatex: respect transparency setting

[Ethan A M. <me...@uw...>]

On Thursday, 20 August 2020 18:34:54 PDT Henri Menke wrote:
> Currently the cairolatex terminal does not respect transparency options
> given in the terminal setting.  In this minimal example the
> notransparent option is ignored:
> 
>     set terminal cairolatex png notransparent
>     set output "test.tex"
>     plot sin(x)
> 
> Running this through gnuplot and then using `file' reveals that the
> alpha channel is still present:
> 
>     $ gnuplot test.gnuplot && file test.png
>     test.png: PNG image data, 1500 x 900, 8-bit/color RGBA, non-interlace

I think the raionale for the current code is that the png image created for
inclusion in a LaTeX document must have a transparent background in order
to support the "back" and "behind" attributes for text elements, since they
will be drawn by the latex engine and then covered by inclusion of the
graphics image.  The structure of the latex processing is:

    render text elements that are "behind" or "back"
    \includegraphics{gnuplot_figure}
    render text elements that are "front"

If you use a solid color background in the graphics image, it will hide
any previously rendered text elements.

	Ethan

> 
> With this patch the transparency setting will be honoured.  The current
> default setting of implicitly assuming transparency is unaltered.
> ---
>  term/cairo.trm | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/term/cairo.trm b/term/cairo.trm
> index ee748dfe2..3ef76ecc6 100644
> --- a/term/cairo.trm
> +++ b/term/cairo.trm
> @@ -849,7 +849,7 @@ void cairotrm_graphics()
>  	plot.background.g = cairo_params->background.g;
>  	plot.background.b = cairo_params->background.b;
>  	gp_cairo_set_background(cairo_params->background);
> -	if (ISCAIROLATEX || cairo_params->transparent)
> +	if (cairo_params->transparent)
>  		gp_cairo_clear_background(&plot);
>  	else
>  		gp_cairo_solid_background(&plot);
>