gnuplot-beta

Plot failure: "Visible pixel grid has a scan line longer than previous scan lines"

[Dima K. <gn...@di...>]

Can somebody tell me what this means?

I'm plotting a grid of data as an image. And I have two identical plot
commands, differing only in the data being plotted (attached). One
works, and the other doesn't, throwing the error in the subject. Does
anybody know what this means? It should work...

Re: Plot failure: "Visible pixel grid has a scan line longer than previous scan lines"

[Ethan A M. <me...@uw...>]

On Monday, 29 June 2020 16:37:12 PDT Dima Kogan wrote:
> Can somebody tell me what this means?
> 
> I'm plotting a grid of data as an image. And I have two identical plot
> commands, differing only in the data being plotted (attached). One
> works, and the other doesn't, throwing the error in the subject. Does
> anybody know what this means? It should work...

I have no idea what is happening.

"with image pixels" makes it work.
set xrange [0:4000] also makes it work.
But the two images show a different cbrange, which is worrisome.

	Ethan

Re: Plot failure: "Visible pixel grid has a scan line longer than previous scan lines"

[Dima K. <gn...@di...>]

Alright, I debugged this (rr was invaluable), and I know mostly what's
going on. The flow is roughly this:

1. The data is read here:

   https://sourceforge.net/p/gnuplot/gnuplot-main/ci/branch-5-4-stable/tree/src/plot3d.c#l1037

   At the end of the first line, x is 3999.0000000000005, which is just outside the
   xrange [0:3999]. This isn't unexpected due to the "using" expression

2. Later we evaluate the x axis for that point (first row, last col):

   https://sourceforge.net/p/gnuplot/gnuplot-main/ci/branch-5-4-stable/tree/src/axis.c#l2782

   3999.0000000000005>3999, so we set this to OUTRANGE

   In a perfect world we should be looking at the "with image" pixel
   edges instead of just the center; like we do in step 5 below.

3. Later we evaluate the z axis for that same point, but this is already
   OUTRANGE, so we don't bother to use that point to set axis->min or
   axis->max

4. As a result the view_port_z[] range we set later is narrower than it
   should be:

   https://sourceforge.net/p/gnuplot/gnuplot-main/ci/branch-5-4-stable/tree/src/graphics.c#l4822

5. And then, further on in in that function, that corner pixel is deemed
   invisible. So the first row then has only 15 pixels, but the next row
   has all 16, so we complain

The logic in step 5 (what we do for !visible pixels) looks suspicious,
but I want to say the fundamental problem is that step 2 and step 5 have
different logic for inrange/outrange. The top-right corner of the
failing plot is OUT in step 2, but IN in step 5.

Any obvious fixes?

Thanks

Re: Plot failure: "Visible pixel grid has a scan line longer than previous scan lines"

[Ethan A M. <me...@uw...>]

On Monday, 29 June 2020 19:51:04 PDT Dima Kogan wrote:
> Alright, I debugged this (rr was invaluable), and I know mostly what's
> going on. The flow is roughly this:
> 
> 1. The data is read here:
> 
>    https://sourceforge.net/p/gnuplot/gnuplot-main/ci/branch-5-4-stable/tree/src/plot3d.c#l1037
> 
>    At the end of the first line, x is 3999.0000000000005, which is just outside the
>    xrange [0:3999]. This isn't unexpected due to the "using" expression
> 
> 2. Later we evaluate the x axis for that point (first row, last col):
> 
>    https://sourceforge.net/p/gnuplot/gnuplot-main/ci/branch-5-4-stable/tree/src/axis.c#l2782
> 
>    3999.0000000000005>3999, so we set this to OUTRANGE
> 
>    In a perfect world we should be looking at the "with image" pixel
>    edges instead of just the center; like we do in step 5 below.
> 
> 3. Later we evaluate the z axis for that same point, but this is already
>    OUTRANGE, so we don't bother to use that point to set axis->min or
>    axis->max
> 
> 4. As a result the view_port_z[] range we set later is narrower than it
>    should be:
> 
>    https://sourceforge.net/p/gnuplot/gnuplot-main/ci/branch-5-4-stable/tree/src/graphics.c#l4822
> 
> 5. And then, further on in in that function, that corner pixel is deemed
>    invisible. So the first row then has only 15 pixels, but the next row
>    has all 16, so we complain
> 
> The logic in step 5 (what we do for !visible pixels) looks suspicious,
> but I want to say the fundamental problem is that step 2 and step 5 have
> different logic for inrange/outrange. The top-right corner of the
> failing plot is OUT in step 2, but IN in step 5.
> 
> Any obvious fixes?

In the end the fix was trivial, but I wouldn't call it "obvious".

The clipping in proces_image() at step 5 correctly accounts for the pixel
extent on x and y regardless of what flag is in pixel->type.  So nothing is
gained by  setting it to OUTRANGE in step 2.  If instead we force all
pixels to be flagged INRANGE at that stage, the incorrect zrange
determination at step 3 doesn't happen and all pixels are kept. 
It wasn't obvious to me that marking all pixels INRANGE would preserve
both autoscaling and clipping, but it does.

fix commited to tip, queued for 5.4

	Ethan

Re: Plot failure: "Visible pixel grid has a scan line longer than previous scan lines"

[Ethan A M. <me...@uw...>]

On Monday, 29 June 2020 19:51:04 PDT Dima Kogan wrote:
> Alright, I debugged this (rr was invaluable), and I know mostly what's
> going on. The flow is roughly this:
> 
> 1. The data is read here:
> 
>    https://sourceforge.net/p/gnuplot/gnuplot-main/ci/branch-5-4-stable/tree/src/plot3d.c#l1037
> 
>    At the end of the first line, x is 3999.0000000000005, which is just outside the
>    xrange [0:3999]. This isn't unexpected due to the "using" expression
> 
> 2. Later we evaluate the x axis for that point (first row, last col):
> 
>    https://sourceforge.net/p/gnuplot/gnuplot-main/ci/branch-5-4-stable/tree/src/axis.c#l2782
> 
>    3999.0000000000005>3999, so we set this to OUTRANGE
> 
>    In a perfect world we should be looking at the "with image" pixel
>    edges instead of just the center; like we do in step 5 below.
> 
> 3. Later we evaluate the z axis for that same point, but this is already
>    OUTRANGE, so we don't bother to use that point to set axis->min or
>    axis->max
> 
> 4. As a result the view_port_z[] range we set later is narrower than it
>    should be:
> 
>    https://sourceforge.net/p/gnuplot/gnuplot-main/ci/branch-5-4-stable/tree/src/graphics.c#l4822
> 
> 5. And then, further on in in that function, that corner pixel is deemed
>    invisible. So the first row then has only 15 pixels, but the next row
>    has all 16, so we complain
> 
> The logic in step 5 (what we do for !visible pixels) looks suspicious,
> but I want to say the fundamental problem is that step 2 and step 5 have
> different logic for inrange/outrange. The top-right corner of the
> failing plot is OUT in step 2, but IN in step 5.
> 
> Any obvious fixes?

I agree with your analysis.

I have some reservations about whether it ever makes sense to clip
the contents of a plot `with image`, since in the general case of
rotation+translation, clipping will not leave a rectangle and thus the
optimized image-data protocols that pass the image as a single chunk
of data won't work.
The current gnuplot code seems to assume that clipping will cleanly
remove a constant width slice from each edge.  Perhaps one fix is to first
detect that clipping exists, and if so fall back to 'with image pixels' so
that each pixel is considered individually.

Your case, however, triggers uneven clipping not because of rotation
but simply due to round-off error.  Two fixes occur to me.

A) The one implied by your point 2 above: auto-scaling and clipping
should take into account the full width of the pixel.
That can probably be done but I will put that on hold for now.

B) For any given axis (z in this case) the program should never try to
apply both autoscaling and clipping!   I don't know whether this glitch
can happen for other plot styles, but I will look for the most generic
possible place to prevent it.

	Ethan

Re: Plot failure: "Visible pixel grid has a scan line longer than previous scan lines"

[Hans-Bernhard B. <HBB...@t-...>]

Am 30.06.2020 um 20:49 schrieb Ethan A Merritt:
> 
> I have some reservations about whether it ever makes sense to clip
> the contents of a plot `with image`, since in the general case of
> rotation+translation, clipping will not leave a rectangle and thus the
> optimized image-data protocols that pass the image as a single chunk
> of data won't work.

OTOH, the alternative would be that the boundaries set up for clipping 
willl be disrespected completely.  That cannot really be considered correct.

> The current gnuplot code seems to assume that clipping will cleanly
> remove a constant width slice from each edge.  Perhaps one fix is to first
> detect that clipping exists, and if so fall back to 'with image pixels' so
> that each pixel is considered individually.

That does make more sense, IMHO.

> B) For any given axis (z in this case) the program should never try to
> apply both autoscaling and clipping!   

Not so fast.  It's still possible for one end of an axis to be clipped, 
while the other is autoscaled.

Re: Plot failure: "Visible pixel grid has a scan line longer than previous scan lines"

[Dima K. <gn...@di...>]

Ethan A Merritt <me...@uw...> writes:

> In the end the fix was trivial, but I wouldn't call it "obvious".
>
> The clipping in proces_image() at step 5 correctly accounts for the pixel
> extent on x and y regardless of what flag is in pixel->type.  So nothing is
> gained by  setting it to OUTRANGE in step 2.  If instead we force all
> pixels to be flagged INRANGE at that stage, the incorrect zrange
> determination at step 3 doesn't happen and all pixels are kept.
> It wasn't obvious to me that marking all pixels INRANGE would preserve
> both autoscaling and clipping, but it does.
>
> fix commited to tip, queued for 5.4

I tested it out, and it appears to work. Thanks a lot! I would've had a
hard time not breaking stuff if I tried to fix this myself.

Re: Plot failure: "Visible pixel grid has a scan line longer than previous scan lines"

[Ethan A M. <me...@uw...>]

On Monday, 29 June 2020 19:51:04 PDT Dima Kogan wrote:
> Alright, I debugged this (rr was invaluable), and I know mostly what's
> going on. The flow is roughly this:
> 
> 1. The data is read here:
> 
>    https://sourceforge.net/p/gnuplot/gnuplot-main/ci/branch-5-4-stable/tree/src/plot3d.c#l1037
> 
>    At the end of the first line, x is 3999.0000000000005, which is just outside the
>    xrange [0:3999]. This isn't unexpected due to the "using" expression

But the x coordinates should come out identically the same in the "good" and "bad"
cases, right?  So how does this explain a different between them?

	Ethan

> 
> 2. Later we evaluate the x axis for that point (first row, last col):
> 
>    https://sourceforge.net/p/gnuplot/gnuplot-main/ci/branch-5-4-stable/tree/src/axis.c#l2782
> 
>    3999.0000000000005>3999, so we set this to OUTRANGE
> 
>    In a perfect world we should be looking at the "with image" pixel
>    edges instead of just the center; like we do in step 5 below.
> 
> 3. Later we evaluate the z axis for that same point, but this is already
>    OUTRANGE, so we don't bother to use that point to set axis->min or
>    axis->max
> 
> 4. As a result the view_port_z[] range we set later is narrower than it
>    should be:
> 
>    https://sourceforge.net/p/gnuplot/gnuplot-main/ci/branch-5-4-stable/tree/src/graphics.c#l4822
> 
> 5. And then, further on in in that function, that corner pixel is deemed
>    invisible. So the first row then has only 15 pixels, but the next row
>    has all 16, so we complain
> 
> The logic in step 5 (what we do for !visible pixels) looks suspicious,
> but I want to say the fundamental problem is that step 2 and step 5 have
> different logic for inrange/outrange. The top-right corner of the
> failing plot is OUT in step 2, but IN in step 5.
> 
> Any obvious fixes?
> 
> Thanks
> 
> 
> _______________________________________________
> gnuplot-beta mailing list
> gnu...@li...
> Membership management via: https://lists.sourceforge.net/lists/listinfo/gnuplot-beta
> 


-- 
Ethan A Merritt
Biomolecular Structure Center,  K-428 Health Sciences Bldg
MS 357742,   University of Washington, Seattle 98195-7742

Re: Plot failure: "Visible pixel grid has a scan line longer than previous scan lines"

[Dima K. <gn...@di...>]

Ethan A Merritt <me...@uw...> writes:

> On Monday, 29 June 2020 19:51:04 PDT Dima Kogan wrote:
>> Alright, I debugged this (rr was invaluable), and I know mostly what's
>> going on. The flow is roughly this:
>>
>> 1. The data is read here:
>>
>>    https://sourceforge.net/p/gnuplot/gnuplot-main/ci/branch-5-4-stable/tree/src/plot3d.c#l1037
>>
>>    At the end of the first line, x is 3999.0000000000005, which is just outside the
>>    xrange [0:3999]. This isn't unexpected due to the "using" expression
>
> But the x coordinates should come out identically the same in the "good" and "bad"
> cases, right?  So how does this explain a different between them?

In the "good" case we're getting lucky in step 5. The corners (which
were out-of-bounds in step 2, and were thus ignored for computing
view_port_z[]) still end up in-bounds in step 5. There's an extra step
that I didn't mention in the previous email:

3.5. We expand the autoranged axis extents to nice, round numbers:

     https://sourceforge.net/p/gnuplot/gnuplot-main/ci/branch-5-4-stable/tree/src/axis.c#l960

This provides some margin so that getting lucky in this way is possible.

Re: Plot failure: "Visible pixel grid has a scan line longer than previous scan lines"

[Ethan A M. <me...@uw...>]

On Monday, 29 June 2020 20:55:26 PDT Dima Kogan wrote:
> Ethan A Merritt <me...@uw...> writes:
> 
> > On Monday, 29 June 2020 19:51:04 PDT Dima Kogan wrote:
> >> Alright, I debugged this (rr was invaluable), and I know mostly what's
> >> going on. The flow is roughly this:
> >>
> >> 1. The data is read here:
> >>
> >>    https://sourceforge.net/p/gnuplot/gnuplot-main/ci/branch-5-4-stable/tree/src/plot3d.c#l1037
> >>
> >>    At the end of the first line, x is 3999.0000000000005, which is just outside the
> >>    xrange [0:3999]. This isn't unexpected due to the "using" expression
> >
> > But the x coordinates should come out identically the same in the "good" and "bad"
> > cases, right?  So how does this explain a different between them?
> 
> In the "good" case we're getting lucky in step 5. The corners (which
> were out-of-bounds in step 2, and were thus ignored for computing
> view_port_z[]) still end up in-bounds in step 5. There's an extra step
> that I didn't mention in the previous email:

Very strange.  The z range should be totally irrelevant to "set view map".
z as a coordinate doesn't matter because it's in projection.
z as an image pixel value doesn't matter because the image grid doesn't
care about the pixel value, only the number of pixels.

But you're right, it does seem to make a difference in this case
For instance:
    gnuplot> splot $MATRIX matrix  using ($1*266.6):($2*199.9090909090909):3 title "" with image
         warning: Visible pixel grid has a scan line longer than previous scan lines.
    gnuplot> splot $MATRIX matrix  using ($1*266.6):($2*199.9090909090909):(0):3 title "" with image
         works with no problem

But opening up the z range explicitly does not fix the problem.
So I still don't understand exactly what's going on here. 

	I'll look harder tomorrow,

		Ethan


> 3.5. We expand the autoranged axis extents to nice, round numbers:
> 
>      https://sourceforge.net/p/gnuplot/gnuplot-main/ci/branch-5-4-stable/tree/src/axis.c#l960
> 
> This provides some margin so that getting lucky in this way is possible.