gnuplot-beta

Fast image markup?

[Dima K. <gn...@di...>]

Hi.

Can somebody chime in on how hard/trivial a new feature would be?

A common use case for me is to mark-up images. So gnuplot would plot an
image, and render stuff on top of it. Something like this:

  plot "image.jpg" binary filetype=auto flipy with rgbimage, ...

This works, but it is always really slow, and if I'm on a machine that's
not ridiculously overpowered, it runs out of memory.

My images are several 1000 pixels per side. So big, but not giant. I
THINK gnuplot is loading each pixel in the image into a separate point
to be plotted, which makes sense given how flexible this is. But 99% of
the time for ME, I'm just plotting the image exactly as it is, with
maybe some independent scaling in x and y. Would it be massively
effortful to detect that the user is trying to do the simple thing, and
to run a different code path that treats the image as an image, and is
thus fast?

Implementation suggestions? Would we want another terminal command for
this?

Thanks

Re: Fast image markup?

[Ethan A M. <me...@uw...>]

On Thursday, 9 April 2020 11:58:52 PDT Dima Kogan wrote:
> Hi.
> 
> Can somebody chime in on how hard/trivial a new feature would be?
> 
> A common use case for me is to mark-up images. So gnuplot would plot an
> image, and render stuff on top of it. Something like this:
> 
>   plot "image.jpg" binary filetype=auto flipy with rgbimage, ...
> 
> This works, but it is always really slow, and if I'm on a machine that's
> not ridiculously overpowered, it runs out of memory.
> 
> My images are several 1000 pixels per side. So big, but not giant. I
> THINK gnuplot is loading each pixel in the image into a separate point
> to be plotted, which makes sense given how flexible this is. But 99% of
> the time for ME, I'm just plotting the image exactly as it is, with
> maybe some independent scaling in x and y. Would it be massively
> effortful to detect that the user is trying to do the simple thing, and
> to run a different code path that treats the image as an image, and is
> thus fast?
> 
> Implementation suggestions? Would we want another terminal command for
> this?

Already there in 5.4 (please test release candidate!)
See

    http://gnuplot.sourceforge.net/demo_5.4/pixmap.html

But I have not benchmarked it or tried to establish what the limits might be
on a memory-limited machine.



gnuplot> help pixmap

 Syntax:
       set pixmap <index> "filename" at <position>
                  {width <w> | height <h> | size <w> <h>}
                  {front|behind} {center}
       show pixmaps
       unset pixmaps

 The `set pixmap` command is similar to `set object` in that it defines an
 object that will appear on subsequent plots. The rectangular array of
 red/green/blue/alpha values making up the pixmap are read from a png, jpeg,
 or gif file.  The position and extent occupied by the pixmap in the gnuplot
 output may be specified in any coordinate system (see `coordinates`).
 The coordinates given by `at <position>` refer to the lower left
 corner of the pixmap unless keyword `center` is present.

[rest of documentation snipped]

	Ethan

Re: Fast image markup?

[Dima K. <gn...@di...>]

Ethan A Merritt <me...@uw...> writes:

> On Thursday, 9 April 2020 11:58:52 PDT Dima Kogan wrote:
>> A common use case for me is to mark-up images. So gnuplot would plot an
>> image, and render stuff on top of it.
>
> Already there in 5.4 (please test release candidate!)

What? Wow! OK.

First off, I've been running 5.4-rc1 (built from source, new
x11_waitforinput patch cherry-picked) for a few days now. No surprises
yet.

I just tried out 'set pixmap'. For me it only appears to work in the
most basic cases, and I can't tell if it's any faster. I think there're
several issues at play.

I tried small images first. Using http://gnuplot.info/figs/front2.png
from the gnuplot site.

This works:

  set terminal x11
  set pixmap 1 "/tmp/front2.png" at 0,0
  plot x

This doesn't:

  set terminal x11
  set pixmap 1 "/tmp/front2.png" at 0,0 width first 300 behind
  plot x

The plot pops up, and goes away immediately. Might be x11_waitforinput()
still needs work, because in the qt terminal it does work. x11 is weird
in lots of cases here, so I'm using qt for the rest of this email. Will
revisit x11 later.

You need to zoom out manually because the pixmap isn't a part of the
autoscaling logic. Should it be? It is when using "with rgbimage" I
think.

It'd also be really nice to use the native size of the image somehow.
The default "set pixmap" uses is "native image size in pixel
coordinates". But what I'd want usually is "native image size in plot
axis coordinates", and this isn't available. I have to know the size,
and to communicate it in the command with something like "first 300".

Usually when talking about images, the y axis is flipped: the (0,0)
pixel is the TOP-left not the BOTTOM-left. So "set pixmap at" should
maybe place the top-left pixel of the image? In my usage I generally do
"set yrange [:] reverse". But even in that usage "set pixmap at" places
the bottom-left pixel. Among the same lines, it should be possible for
the image pixel axes to follow the plot axes. In that mode "set yrange
[:] reverse" would plot the image right side up, and with the default
axis orientation, it would be upside down.

"set pixel height" and "set pixel size" don't play nicely with "set
yrange [:] reverse", so for instance this doesn't draw an image at all:

  set yrange [:] reverse
  set pixmap 1 "/tmp/front2.png" at 0,0 height 300
  plot x

"set pixel size" doesn't follow the docs, I think. The docs say it wants
"set pixel size A B", while it actually wants "set pixel size A,B". Note
the extra comma.

I tried big images too. I THINK they work, but it's still really slow
compared to any image viewer. Is that expected? Should it be faster than
what "with rgbimage" does?

Thanks for the feature!

Re: Fast image markup?

[Ethan A M. <me...@uw...>]

On Thursday, 9 April 2020 14:06:01 PDT Dima Kogan wrote:
> Ethan A Merritt <me...@uw...> writes:
> 
> > On Thursday, 9 April 2020 11:58:52 PDT Dima Kogan wrote:
> >> A common use case for me is to mark-up images. So gnuplot would plot an
> >> image, and render stuff on top of it.
> >
> > Already there in 5.4 (please test release candidate!)
> 
> What? Wow! OK.
> 
> First off, I've been running 5.4-rc1 (built from source, new
> x11_waitforinput patch cherry-picked) for a few days now. No surprises
> yet.
> 
> I just tried out 'set pixmap'. For me it only appears to work in the
> most basic cases, and I can't tell if it's any faster. I think there're
> several issues at play.
> 
> I tried small images first. Using http://gnuplot.info/figs/front2.png
> from the gnuplot site.
> 
> This works:
> 
>   set terminal x11
>   set pixmap 1 "/tmp/front2.png" at 0,0
>   plot x
> 
> This doesn't:
> 
>   set terminal x11
>   set pixmap 1 "/tmp/front2.png" at 0,0 width first 300 behind
>   plot x

I think you misunderstood the meaning of "first" in that context.
The default xrange is [-10:10], if you place an image with its lower
corner at 0,0 then the maximum size that will fit in the plot is
   set pixmap 1 "/tmp/front2.png" at 0,0 width first 10 behind
i.e. 10 units in the first axis coordinate system

> The plot pops up, and goes away immediately. Might be x11_waitforinput()
> still needs work, because in the qt terminal it does work. x11 is weird
> in lots of cases here, so I'm using qt for the rest of this email. Will
> revisit x11 later.

I have no idea what that's all about, but possibly it is a result of
trying to draw a bitmap that extends waaaay off screen,
since "width first 300" is 30 times too big to fit on screen.

> You need to zoom out manually because the pixmap isn't a part of the
> autoscaling logic. Should it be? It is when using "with rgbimage" I
> think.

Again a misunderstanding?
If you use axis coordinates (e.g. "first 10") then it autoscales.
If you stay in pixel coordinates then it doesn't scale.

> It'd also be really nice to use the native size of the image somehow.
> The default "set pixmap" uses is "native image size in pixel
> coordinates". But what I'd want usually is "native image size in plot
> axis coordinates", and this isn't available.

What does that even mean?

> I have to know the size,
> and to communicate it in the command with something like "first 300".

No, that's wrong.  If you use plot coordinates you *don't* need to know
the original size in pixels because it will be scaled to match the plot.
You are specifying a width in plot coordinates.

> Usually when talking about images, the y axis is flipped: the (0,0)
> pixel is the TOP-left not the BOTTOM-left. So "set pixmap at" should
> maybe place the top-left pixel of the image? In my usage I generally do
> "set yrange [:] reverse". But even in that usage "set pixmap at" places
> the bottom-left pixel. Among the same lines, it should be possible for
> the image pixel axes to follow the plot axes. In that mode "set yrange
> [:] reverse" would plot the image right side up, and with the default
> axis orientation, it would be upside down.

Pixmaps are not intended as a replacement for "with image".

I think of it more like placing a complicated character glyph within
a page of text.  You can specify where to put it, and you can make
it larger or smaller, but in the end it is still the same character.
Rotation, inversion, or other substantial transformation is left to
images=pictures rather than to pixmaps=characters.

> "set pixel height" and "set pixel size" don't play nicely with "set
> yrange [:] reverse", so for instance this doesn't draw an image at all:
> 
>   set yrange [:] reverse
>   set pixmap 1 "/tmp/front2.png" at 0,0 height 300
>   plot x

Huh. That indeed looks like a bug of some sort.
The requested height is still wrong there (should be <10) but yes setting
yrange reverse messes up the placement.  I will have a look at that.

> "set pixel size" doesn't follow the docs, I think. The docs say it wants
> "set pixel size A B", while it actually wants "set pixel size A,B". Note
> the extra comma.

yes.

> I tried big images too. I THINK they work, but it's still really slow
> compared to any image viewer. Is that expected? Should it be faster than
> what "with rgbimage" does?

"slow" when doing what?  rotating in 3D? 
Can you do that in a normal image viewer?

The difference from rgbimage is that all the "with image" commands treat the
individual pixels as independent data points.  They are stored and manipulated
separately, taking up a lot of memory.  The "pixmap" mechanism stores and
dumps a whole png image, which is much more compact, possibly with rescaling.
The internal storage is persistant, so it doesn't have to be read in and re-processed
as new data on each plot/replot etc.   

So yeah, it should be faster so far as the gnuplot code is concerned.
What happens after you send it to an X-server is another matter.
Once it is in the x11 display list it probably doesn't make any difference
how it got there.

	Ethan

Re: Fast image markup?

[Dima K. <gn...@di...>]

Ethan A Merritt <me...@uw...> writes:

> On Thursday, 9 April 2020 14:06:01 PDT Dima Kogan wrote:
>
>> You need to zoom out manually because the pixmap isn't a part of the
>> autoscaling logic. Should it be? It is when using "with rgbimage" I
>> think.
>
> I think you misunderstood the meaning of "first" in that context.
> The default xrange is [-10:10], if you place an image with its lower
> corner at 0,0 then the maximum size that will fit in the plot is
>    set pixmap 1 "/tmp/front2.png" at 0,0 width first 10 behind
> i.e. 10 units in the first axis coordinate system

By default gnuplot scales the axes to fit all the data being plotted.
But the images plotted with "set pixmap" aren't included in this logic,
it looks like. I think they probably should be. So the default xrange of
[-10:10] would be expanded to include the image.


> If you use axis coordinates (e.g. "first 10") then it autoscales.
> If you stay in pixel coordinates then it doesn't scale.

I think this is how it should work, but it doesn't. I do this:

  set pixmap 1 "/tmp/front2.png" at 0,0 width first 300 behind
  plot x

This says "plot the image with a corner at 0,0 scaled such the whole
image is 300 wide along the x axis." The plot I get uses the initial
x scaling of [-10:10], which includes only a tiny corner of the image. I
used axis coordinates and it did not autoscale.

>> It'd also be really nice to use the native size of the image somehow.
>> The default "set pixmap" uses is "native image size in pixel
>> coordinates". But what I'd want usually is "native image size in plot
>> axis coordinates", and this isn't available.
>
> What does that even mean?
>
>> I have to know the size,
>> and to communicate it in the command with something like "first 300".
>
> No, that's wrong.  If you use plot coordinates you *don't* need to know
> the original size in pixels because it will be scaled to match the plot.
> You are specifying a width in plot coordinates.

I'm being loose with terminology; sorry. I'm trying to emulate "with
rgbimage". So if I have an image that's 500 pixels wide I want the x
mapping I get with

  set pixmap 1 "/tmp/front2.png" at 0,0 width first 500

But I don't want to specify "500" in the command. Should we maybe have a
width of "0" mean "original image width", or something? Currently
"set pixmap ... width first 0" is equivalent to "set pixmap ..."


> Pixmaps are not intended as a replacement for "with image".
>
> I think of it more like placing a complicated character glyph within
> a page of text.  You can specify where to put it, and you can make
> it larger or smaller, but in the end it is still the same character.
> Rotation, inversion, or other substantial transformation is left to
> images=pictures rather than to pixmaps=characters.

OK, so maybe pixmaps aren't what I actually want then. It's just a
different use case.


>> I tried big images too. I THINK they work, but it's still really slow
>> compared to any image viewer. Is that expected? Should it be faster than
>> what "with rgbimage" does?
>
> "slow" when doing what?  rotating in 3D?
> Can you do that in a normal image viewer?

Just the simplest usage: plot an image with some annotations. No 3d or
any transformations of anything. Compared to image viewers the speed
difference isn't subtle.


> The difference from rgbimage is that all the "with image" commands treat the
> individual pixels as independent data points.  They are stored and manipulated
> separately, taking up a lot of memory.  The "pixmap" mechanism stores and
> dumps a whole png image, which is much more compact, possibly with rescaling.
> The internal storage is persistant, so it doesn't have to be read in and re-processed
> as new data on each plot/replot etc.
>
> So yeah, it should be faster so far as the gnuplot code is concerned.
> What happens after you send it to an X-server is another matter.
> Once it is in the x11 display list it probably doesn't make any difference
> how it got there.

OK. I'm guessing there's both a gnuplot and an x component in the
performance difference. I'll take a look at some point.

Thanks for explaining

dima

Re: Fast image markup?

[Ethan A M. <me...@uw...>]

On Monday, 13 April 2020 11:08:01 PDT Dima Kogan wrote:

> >> I tried big images too. I THINK they work, but it's still really slow
> >> compared to any image viewer. Is that expected? Should it be faster than
> >> what "with rgbimage" does?
> >
> > "slow" when doing what?  rotating in 3D?
> > Can you do that in a normal image viewer?
> 
> Just the simplest usage: plot an image with some annotations. No 3d or
> any transformations of anything. Compared to image viewers the speed
> difference isn't subtle.

I may be misunderstanding what you are doing (quite possible).

For me using a large image (say a 1365x1024 jpg "wallpaper" photo)
as a background introduces no noticeable delay at all using the wxt
terminal.  Even for 3D rotation the response is instantaneous:

    set term wxt size 1500,1000
    set pixmap 1 at screen 0,0 size screen 1,1 behind '~/images/wallpaper/p1040193.jpg'
    splot x*y with pm3d
    [rotate/zoom/translate with mouse as normal]

For x11 the initial display is very quick, but the response to 3D rotation
is somewhat laggy.  Not bad, though.  

The strange one is actually qt, where the response to 3D rotation is horribly slow.
Zoom and pan in 2D is quick however. I'll bet there is some way to speed up the qt 
3D rendering also, but it will require some investigation. 

	Ethan

Re: Fast image markup?

[Dima K. <gn...@di...>]

Ethan A Merritt <me...@uw...> writes:

> On Monday, 13 April 2020 11:08:01 PDT Dima Kogan wrote:
>
>> >> I tried big images too. I THINK they work, but it's still really slow
>> >> compared to any image viewer. Is that expected? Should it be faster than
>> >> what "with rgbimage" does?
>> >
>> > "slow" when doing what?  rotating in 3D?
>> > Can you do that in a normal image viewer?
>>
>> Just the simplest usage: plot an image with some annotations. No 3d or
>> any transformations of anything. Compared to image viewers the speed
>> difference isn't subtle.
>
> I may be misunderstanding what you are doing (quite possible).
>
> For me using a large image (say a 1365x1024 jpg "wallpaper" photo)
> as a background introduces no noticeable delay at all using the wxt
> terminal.  Even for 3D rotation the response is instantaneous:
>
>     set term wxt size 1500,1000
>     set pixmap 1 at screen 0,0 size screen 1,1 behind '~/images/wallpaper/p1040193.jpg'
>     splot x*y with pm3d
>     [rotate/zoom/translate with mouse as normal]

That's a really good data point. Thanks. I got side-tracked into
something else, but let me come back to this later with some benchmarks,
and we can compare. It'd be great if there was something specific about
my setup that's slowing it down.