gnuplot-beta

Plotting contours together with 'with image' data

[Dima K. <gn...@di...>]

Hi. I have a matrix data set, and I'd like to plot it using colors (with
image), AND I'd like to overlay contours on top. This apparently does
not work.

It looks like 3d 'with image' plots use the data values for colors, but
NOT for z. I.e. you get a colored image at z=0. Since it looks like
contours read the z coordinate to do their thing, this doesn't work.

If I plot 'with lines', I get contours. I can also plot 'with pm3d',
which produces a color AND a z-coordinate, so contours work then.

Questions:

It makes no sense to me that 'with image' 3d plots are stuck at z=0.
Does ANYBODY want that? Can we apply the values to color AND z in this
case? Is this hard, or is there some specific reason we're doing it the
way we're doing it?

pm3d is intended for irregularly-sampled data (as I understand it), so
is there overhead for using pm3d in this case? Size? Speed? If there's
no overhead, can 'splot with image' simply map to 'splot with pm3d'?

If nothing else, can we barf instead of silently not rendering the
contours in the 'with image' case?

I'm attaching a sample data file.

Thanks!

Re: Plotting contours together with 'with image' data

[Dima K. <gn...@di...>]

Dima Kogan <gn...@di...> writes:

> It makes no sense to me that 'with image' 3d plots are stuck at z=0.
> Does ANYBODY want that? Can we apply the values to color AND z in this
> case? Is this hard, or is there some specific reason we're doing it the
> way we're doing it?
>
> pm3d is intended for irregularly-sampled data (as I understand it), so
> is there overhead for using pm3d in this case? Size? Speed? If there's
> no overhead, can 'splot with image' simply map to 'splot with pm3d'?

I just answered some of this. Plotting to a pdf, the pmd3d files are 70%
larger. And it looks like pm3d re-interpolated the original data, which
I don't want, and which shouldn't be needed, since I already have
regularly-spaced matrix data.

So the question that remains is the one about making 'with image'
produce colors AND z

Re: Plotting contours together with 'with image' data

[Ethan A M. <sf...@us...>]

On Wednesday, July 18, 2018 12:37:17 PM PDT Dima Kogan wrote:
> Hi. I have a matrix data set, and I'd like to plot it using colors (with
> image), AND I'd like to overlay contours on top. This apparently does
> not work.
> 
> It looks like 3d 'with image' plots use the data values for colors, but
> NOT for z. I.e. you get a colored image at z=0. Since it looks like
> contours read the z coordinate to do their thing, this doesn't work.

I will try to explain why "with image" is not a plot style that can be
contoured.

Consider a typical image file: PNG or GIF or JPEG or whatever.
It contains red/green/blue and maybe alpha values for each pixel.
But no z.  Where would you get a z value from?

It is true that PNG or GIF can also encode indexed color.
But the sequentially indexed color values do not necessarily correspond
to a continuous palette like the ones that gnuplot can generate.
So the matrix of index values is not generally something that can
be meaningfully contoured. 

Now it is true that gnuplot's "with image" mode can be used for 
gridded data where the color information is generated by
index lookup from gnuplot's own continuous color palette.
I guess that's what you are doing, but it is certainly not the general case.

> If I plot 'with lines', I get contours. I can also plot 'with pm3d',
> which produces a color AND a z-coordinate, so contours work then.
> 
> Questions:
> 
> It makes no sense to me that 'with image' 3d plots are stuck at z=0.
> Does ANYBODY want that? 

They are not stuck at z=0.  See for example plots 6-8 of the image2 demo:

      http://gnuplot.sourceforge.net/demo_5.2/image2.html
    
> Can we apply the values to color AND z in this
> case? Is this hard, or is there some specific reason we're doing it the
> way we're doing it?
> 
> pm3d is intended for irregularly-sampled data (as I understand it)

Not really. pm3d is a coloring algorithm that can be applied to anything
constructured from quadrangles.  Typically this is a regularly gridded surface.
There are also options for interpolating an irregularly sampled data set to
generate a regular grid that can then be colored (see "help dgrid3d"). 
But it sounds like your case doesn't need any extra step to regularize the grid.

> is there overhead for using pm3d in this case? Size? Speed? If there's
> no overhead, can 'splot with image' simply map to 'splot with pm3d'?

In gnuplot version 5 you can tell pm3d to use a pre-calculated
RGB color for each grid point.  This RGB value must be in the form of
a 24-bit packed RGB integer.   I don't know exactly what  the nature of
the values in your matrix so I cannot provide exact instructions,
but in essence the command sequence would be

  set pm3d border lc rgb variable
  set style fill solid 1.0
  splot $DATA using 1:2:3:(rgbfunc($3)) with pm3d

It is up to you to define rgbfunc() appropriately.
Here is an example using a regularly sampled grid and a constant
color value:

  set xrange [-2:2]
  set yrange [-2:2]
  set sample 20
  set isosample 20

  splot '++' using 1:2:(cos(y)*sin(x)):(int(0xdd77aa)) with pm3d

But your special case may not require any of that.
If the "z" values on a regular grid are to be used also as a  palette
index then indeed the default pm3d settings may be sufficient.

> If nothing else, can we barf instead of silently not rendering the
> contours in the 'with image' case?

The more common complaint is that too many error messages
are printed for things that are not fatal errors.
Consider a hypothetical plot that includes some contours, some image data,
and some other stuff:

    set contour
    splot A with lines, B with image, C with vectors

The result is a plot with contours for A and no contours for B or C.  
What would be the benefit of spewing errors or failing just because
B and C are plotted using styles that cannot be contoured?

	Ethan


> Thanks!
> 
>

Re: Plotting contours together with 'with image' data

[Dima K. <gn...@di...>]

sfeam <sf...@us...> writes:

> I think you are mis-interpreting the normal operation of PM3D as
> interpolation.  PM3D operates on quadrangles.  Each quadrangle has
> four corners.  Each corner has a color.   What color should be used for
> the quadrangle?   There are many choices and the command that selects one
> of the options is
>
>     set pm3d corners2color
> 		{ mean|geomean|harmean|rms|median|min|max|c1|c2|c3|c4 }
>
> The default option is "mean", and I guess that is what you are calling
> interpolation.  If you want each grid point in your original data to
> be the sole color source for a pm3d quadrangle ("pixel" if you are
> thinking of this as an image) then you want
>    set pm3d corners2color c1     (or c2 or c3 or c4)

Aha. Yes. That does sorta what I want. Except it shifts the image by 0.5
pixels in each direction. I understand why, but it's not ideal.


>> >> is there overhead for using pm3d in this case? Size? Speed? If there's
>> >> no overhead, can 'splot with image' simply map to 'splot with pm3d'?
>
> Many of the output devices can handle image data as a special case.
> The corresponding gnuplot driver can define a rectangular area and then
> pass only an array of color values to the device.  This is obviously a much
> smaller output stream than if the driver must output separate coordinates
> and color information for every pixel.  So in general the size of the
> output stream increases in the order
>
>    "with image" < "with pm3d" ~= "with image pixels"

OK. I forgot "with image pixels" existed. And just found an unrelated
(likely) bug: plotting the test data in this thread "with image" and
"with image pixels" produces slightly different images: the "with image"
version has each pixel centered not-quite at the correct locations. I
bet there's an off-by-one error in the "with image" code that makes the
coordinate mapping off slightly. I'll try to look at it when I get the
chance.


> I am thinking that we could use the command
>
>     set pm3d {explicit|implicit}
>
> as a model for opt-in contours.  As it is now, "set contours" causes
> all subsequent plots to use contours unless they cannot be contoured
> or they opt out with "nocontour".  This is the equivalent of the
> state produced by "set pm3d implicit".   It might work to add a command
> "set contours explicit" as we have for pm3d, meaning that subsequent
> plots are only contoured if they use an explicit plot style "contours".
>
> I don't think such a change would allow you to create any plot that you
> cannot already create, but perhaps making "set pm3d" and "set contours"
> more similar would be easier them both easier to understand.
> What do you think?

I think that extra bit of user control would definitely be welcome.

Thanks.
dima

Re: Plotting contours together with 'with image' data

[Dima K. <gn...@di...>]

Thanks Karl-Friedrich, Ethan for the suggestions.

These both sound like they work, but they also sound like workarounds
instead of solutions. I have already found an ugly workaround that
works: splot the same data twice: once 'with image' (for the colormap)
and again 'with lines nosurface' (for the contours).

Comments inline.


Ethan A Merritt <sf...@us...> writes:

> On Wednesday, July 18, 2018 12:37:17 PM PDT Dima Kogan wrote:
>> Hi. I have a matrix data set, and I'd like to plot it using colors (with
>> image), AND I'd like to overlay contours on top. This apparently does
>> not work.
>>
>> It looks like 3d 'with image' plots use the data values for colors, but
>> NOT for z. I.e. you get a colored image at z=0. Since it looks like
>> contours read the z coordinate to do their thing, this doesn't work.
>
> I will try to explain why "with image" is not a plot style that can be
> contoured.
>
> Consider a typical image file: PNG or GIF or JPEG or whatever.
> It contains red/green/blue and maybe alpha values for each pixel.
> But no z.  Where would you get a z value from?
>
> <snip>
>
> Now it is true that gnuplot's "with image" mode can be used for
> gridded data where the color information is generated by
> index lookup from gnuplot's own continuous color palette.
> I guess that's what you are doing, but it is certainly not the general case.

In all my experience, I've used 'with rgbimage' for image files (.png,
.jpg, ...) and 'with image' for gridded data. And 'help image' describes
the gridded data use case first, and at least this suggests that this
isn't some sort of side-feature.


>> If I plot 'with lines', I get contours. I can also plot 'with pm3d',
>> which produces a color AND a z-coordinate, so contours work then.
>>
>> Questions:
>>
>> It makes no sense to me that 'with image' 3d plots are stuck at z=0.
>> Does ANYBODY want that?
>
> They are not stuck at z=0.  See for example plots 6-8 of the image2 demo:
>
>       http://gnuplot.sourceforge.net/demo_5.2/image2.html

All right. If I specify something like "rotate", will I get contours? If
not, then I claim that this is surprising to the user, and is thus a
bug. My fundamental complaint is this:

  splot "something" with pm3d

gives me contours. 'with lines' and 'with points' give me contours.
'with image' does NOT give me contours. Silently. To a USER who doesn't
know or care about the gnuplot internals this is a bug.


>> Can we apply the values to color AND z in this
>> case? Is this hard, or is there some specific reason we're doing it the
>> way we're doing it?
>>
>> pm3d is intended for irregularly-sampled data (as I understand it)
>
> Not really. pm3d is a coloring algorithm that can be applied to
> anything constructured from quadrangles. Typically this is a regularly
> gridded surface. There are also options for interpolating an
> irregularly sampled data set to generate a regular grid that can then
> be colored (see "help dgrid3d"). But it sounds like your case doesn't
> need any extra step to regularize the grid.

Without extra options pm3d was re-interpolating my data in some way, so
its results weren't the same as 'with image'. And it's doing something
different internally, and ends up producing bigger (and presumably less
efficient) output.


>> is there overhead for using pm3d in this case? Size? Speed? If there's
>> no overhead, can 'splot with image' simply map to 'splot with pm3d'?
>
> In gnuplot version 5 you can tell pm3d to use a pre-calculated
> RGB color for each grid point.  This RGB value must be in the form of
> a 24-bit packed RGB integer.   I don't know exactly what  the nature of
> the values in your matrix so I cannot provide exact instructions,
> but in essence the command sequence would be
>
>   set pm3d border lc rgb variable
>   set style fill solid 1.0
>   splot $DATA using 1:2:3:(rgbfunc($3)) with pm3d
>
> It is up to you to define rgbfunc() appropriately.
> Here is an example using a regularly sampled grid and a constant
> color value:
>
>   set xrange [-2:2]
>   set yrange [-2:2]
>   set sample 20
>   set isosample 20
>
>   splot '++' using 1:2:(cos(y)*sin(x)):(int(0xdd77aa)) with pm3d
>
> But your special case may not require any of that.
> If the "z" values on a regular grid are to be used also as a  palette
> index then indeed the default pm3d settings may be sufficient.

Thanks for the example. I'll keep this usage in mind for the future, but
it's not applicable to my immediate use case: I have 1-dimensional
values, so the default gnuplot colormapping is what I want.


>> If nothing else, can we barf instead of silently not rendering the
>> contours in the 'with image' case?
>
> The more common complaint is that too many error messages
> are printed for things that are not fatal errors.
> Consider a hypothetical plot that includes some contours, some image data,
> and some other stuff:
>
>     set contour
>     splot A with lines, B with image, C with vectors
>
> The result is a plot with contours for A and no contours for B or C.
> What would be the benefit of spewing errors or failing just because
> B and C are plotted using styles that cannot be contoured?

Yeah, this is an unfortunate consequence of the way we set up our
interface. Ideally, you'd specify the contour options per plot:

  splot A with lines contours, B with image contours, C with vectors

Changing this retroactively is probably more trouble than it's worth.

OK. Let me look at the code and see if I can make 'with image' do what I
think it should do. Don't know when I'll get to it, but that will make
this into a more concrete discussion.

dima

Re: Plotting contours together with 'with image' data

[sfeam <sf...@us...>]

On Saturday, 21 July 2018 12:29:32 Dima Kogan wrote:
> Thanks Karl-Friedrich, Ethan for the suggestions.
> 
> These both sound like they work, but they also sound like workarounds
> instead of solutions. I have already found an ugly workaround that
> works: splot the same data twice: once 'with image' (for the colormap)
> and again 'with lines nosurface' (for the contours).
> 
> Comments inline.
> 
> 
> Ethan A Merritt <sf...@us...> writes:
> 
> > On Wednesday, July 18, 2018 12:37:17 PM PDT Dima Kogan wrote:
> >> Hi. I have a matrix data set, and I'd like to plot it using colors (with
> >> image), AND I'd like to overlay contours on top. This apparently does
> >> not work.
> >>
> >> It looks like 3d 'with image' plots use the data values for colors, but
> >> NOT for z. I.e. you get a colored image at z=0. Since it looks like
> >> contours read the z coordinate to do their thing, this doesn't work.
> >
> > I will try to explain why "with image" is not a plot style that can be
> > contoured.
> >
> > Consider a typical image file: PNG or GIF or JPEG or whatever.
> > It contains red/green/blue and maybe alpha values for each pixel.
> > But no z.  Where would you get a z value from?
> >
> > <snip>
> >
> > Now it is true that gnuplot's "with image" mode can be used for
> > gridded data where the color information is generated by
> > index lookup from gnuplot's own continuous color palette.
> > I guess that's what you are doing, but it is certainly not the general case.
> 
> In all my experience, I've used 'with rgbimage' for image files (.png,
> .jpg, ...) and 'with image' for gridded data. And 'help image' describes
> the gridded data use case first, and at least this suggests that this
> isn't some sort of side-feature.

It's not a side-feature.  It is a method for plotting color-indexed
images, e.g. GIF or non-truecolor PNG.   But indexed color is not the
same thing as "z values".  There are no z values in a GIF file.


> >> pm3d is intended for irregularly-sampled data (as I understand it)
> >
> > Not really. pm3d is a coloring algorithm that can be applied to
> > anything constructured from quadrangles. Typically this is a regularly
> > gridded surface. There are also options for interpolating an
> > irregularly sampled data set to generate a regular grid that can then
> > be colored (see "help dgrid3d"). But it sounds like your case doesn't
> > need any extra step to regularize the grid.
> 
> Without extra options pm3d was re-interpolating my data in some way, so
> its results weren't the same as 'with image'. And it's doing something
> different internally, and ends up producing bigger (and presumably less
> efficient) output.

I think you are mis-interpreting the normal operation of PM3D as
interpolation.  PM3D operates on quadrangles.  Each quadrangle has
four corners.  Each corner has a color.   What color should be used for
the quadrangle?   There are many choices and the command that selects one
of the options is 

    set pm3d corners2color 
		{ mean|geomean|harmean|rms|median|min|max|c1|c2|c3|c4 }

The default option is "mean", and I guess that is what you are calling
interpolation.  If you want each grid point in your original data to
be the sole color source for a pm3d quadrangle ("pixel" if you are
thinking of this as an image) then you want
   set pm3d corners2color c1     (or c2 or c3 or c4)

  
> >> is there overhead for using pm3d in this case? Size? Speed? If there's
> >> no overhead, can 'splot with image' simply map to 'splot with pm3d'?

Many of the output devices can handle image data as a special case.
The corresponding gnuplot driver can define a rectangular area and then
pass only an array of color values to the device.  This is obviously a much
smaller output stream than if the driver must output separate coordinates
and color information for every pixel.  So in general the size of the
output stream increases in the order

   "with image" < "with pm3d" ~= "with image pixels"

> >> If nothing else, can we barf instead of silently not rendering the
> >> contours in the 'with image' case?
> >
> > The more common complaint is that too many error messages
> > are printed for things that are not fatal errors.
> > Consider a hypothetical plot that includes some contours, some image data,
> > and some other stuff:
> >
> >     set contour
> >     splot A with lines, B with image, C with vectors
> >
> > The result is a plot with contours for A and no contours for B or C.
> > What would be the benefit of spewing errors or failing just because
> > B and C are plotted using styles that cannot be contoured?
> 
> Yeah, this is an unfortunate consequence of the way we set up our
> interface. Ideally, you'd specify the contour options per plot:
> 
>   splot A with lines contours, B with image contours, C with vectors
>
> Changing this retroactively is probably more trouble than it's worth.

Right.  The historical design for several of gnuplot's plotting modes
was an all-or-nothing toggle.  "set hidden3d"  "set pm3d" "set contours".
The code evolved to allow you to opt out of the global settings for
individual plot components.  E.g.

   set hidden3d
   splot FOO, BAZ with labels nohidden3d

guarantees that the labels in BAZ will not be obscured by the surface
drawn for FOO.
And for contours:

   set contour surface
   splot  f(x,y),  g(x,y) nocontour, h(x,y) nosurface

produces a plot in which f() gets both surface and contours,
g() gets surface only, and h() gets contours only.

pm3d evolved slightly differently. Instead of requiring a global
setting and allowing plot components to opt out,  we now allow
individual plot components to opt in by using the style "with pm3d".
Yeah that's a bit inconsistent.  You might logically expect there to
be a "nopm3d" option, but there isn't.

   
> OK. Let me look at the code and see if I can make 'with image' do what I
> think it should do. Don't know when I'll get to it, but that will make
> this into a more concrete discussion.

It _cannot_ do what you want, because in general image data does not come
with z values.

	Ethan

> 
> dima

Re: Plotting contours together with 'with image' data

[sfeam <sf...@us...>]

On Saturday, 21 July 2018 16:49:32 sfeam via gnuplot-beta wrote:
> On Saturday, 21 July 2018 12:29:32 Dima Kogan wrote:

> > Yeah, this is an unfortunate consequence of the way we set up our
> > interface. Ideally, you'd specify the contour options per plot:
> > 
> >   splot A with lines contours, B with image contours, C with vectors
> >
> > Changing this retroactively is probably more trouble than it's worth.
> 
> Right.  The historical design for several of gnuplot's plotting modes
> was an all-or-nothing toggle.  "set hidden3d"  "set pm3d" "set contours".
> The code evolved to allow you to opt out of the global settings for
> individual plot components.  E.g.
> 
>    set hidden3d
>    splot FOO, BAZ with labels nohidden3d
> 
> guarantees that the labels in BAZ will not be obscured by the surface
> drawn for FOO.
> And for contours:
> 
>    set contour surface
>    splot  f(x,y),  g(x,y) nocontour, h(x,y) nosurface
> 
> produces a plot in which f() gets both surface and contours,
> g() gets surface only, and h() gets contours only.
> 
> pm3d evolved slightly differently. Instead of requiring a global
> setting and allowing plot components to opt out,  we now allow
> individual plot components to opt in by using the style "with pm3d".
> Yeah that's a bit inconsistent.  You might logically expect there to
> be a "nopm3d" option, but there isn't.

I am thinking that we could use the command

    set pm3d {explicit|implicit}

as a model for opt-in contours.  As it is now, "set contours" causes
all subsequent plots to use contours unless they cannot be contoured
or they opt out with "nocontour".  This is the equivalent of the 
state produced by "set pm3d implicit".   It might work to add a command
"set contours explicit" as we have for pm3d, meaning that subsequent
plots are only contoured if they use an explicit plot style "contours".

I don't think such a change would allow you to create any plot that you
cannot already create, but perhaps making "set pm3d" and "set contours"
more similar would be easier them both easier to understand.
What do you think?    

	Ethan