gnuplot-beta — Developers and beta testers of new gnuplot code

Re: geographic plotting

[Allin C. <cot...@wf...>]

On Fri, 6 Mar 2020, Ethan A Merritt wrote:

[a propos Manfred Schwarb's suggestion of using GeoJSON files
for geographic plotting in gnuplot]

> Good call.  Geojson looks quite promising.
> A real parser should be easy, but even a quick pass through sed
> or a text editor makes it usable.
> I have posted a proof-of-principle example using US state boundary
> data from a sample file on the geojson development site:
>
>  http://skuld.bmsc.washington.edu/people/merritt/gnuplot/

Thanks for the example, Ethan.

I've recently returned to this issue, exploring the shapefiles 
option. I've written some relevant utilities in C, designed to link 
against Frank Warmerdam's lipshape. (I'll be happy to share these 
once they're better tested.) With their help I've managed to create 
quite a nice map of the US states, colorized by state population 
levels. See 
http://ricardo.ecn.wfu.edu/pub/gretl/datamaps/statepop.png .

But the method I'm using for the colorization is clunky and I'm sure 
it can be improved upon. Here's my input file:

<gnuplot>
set term pngcairo size 660,400
set output 'statepop.png'
set linetype 1 lc rgb "white" lw 1

set palette model HSV file 'statepop.hsv'
set cbrange [0:50]
unset colorbox

set xrange [95:210]
set yrange [-175:-135]
set noxtics
set noytics
set border 0
unset key

plot for [i=0:*] 'merc.dat' index i with filledcurves \
  fc palette cb i, 'merc.dat' with lines lt 1
</gnuplot>

In the above, 'merc.dat' contains the state outlines from the .shp
file. The population data get into the picture via the pre-processed
'statepop.hsv'. To produce this (with its shades of blue) I ran a
loop across the states using H = 229/360, V = 93/100, and S[i] =
population of state i divided by the max population value.

Any suggestions on the best way to do the colorization within 
gnuplot? And also, perhaps, how to discretize it (by decile for 
instance)? Is there a way to replace

   fc palette cb i

with something like

   fc palette cb <function of i and the data to be colorized>

?

Allin Cottrell

Re: First release candidate for gnuplot version 5.4

[Achim G. <Str...@ne...>]

Ethan A Merritt writes:
>> The tutorial seems missing from the 5.4 branch and tarball, but still
>> available in master.  Is that intentional?
>
> Yes.  
>
> The latex tutorial was written for the old, old latex drivers "set term latex"
> and "set term eepic" that are no longer included in the gnuplot distribution. 
> The tutorial source is still in master in case it inspires  someone to
> write a new one illustrating use of the newer terminals relevant to latex.

I would suggest to mention that in the release notes and move on, then.

I have made the RC1 available on Cygwin in both 32bit and 64bit variants
for testing.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Factory and User Sound Singles for Waldorf Q+, Q and microQ:
http://Synth.Stromeko.net/Downloads.html#WaldorfSounds

Re: First release candidate for gnuplot version 5.4

[Ethan A M. <me...@uw...>]

On Monday, 13 April 2020 11:45:42 PDT Achim Gratz wrote:
> Ethan A Merritt writes:
> > You can download a source tarball for the first version 5.4 release candidate from 
> >
> >     https://sf.net/projects/gnuplot/files/testing/gnuplot-5.4.rc1.tar.gz
> 
> The tutorial seems missing from the 5.4 branch and tarball, but still
> available in master.  Is that intentional?

Yes.  

The latex tutorial was written for the old, old latex drivers "set term latex"
and "set term eepic" that are no longer included in the gnuplot distribution. 
The tutorial source is still in master in case it inspires  someone to
write a new one illustrating use of the newer terminals relevant to latex.

	Ethan

Re: First release candidate for gnuplot version 5.4

[Achim G. <Str...@ne...>]

Ethan A Merritt writes:
> You can download a source tarball for the first version 5.4 release candidate from 
>
>     https://sf.net/projects/gnuplot/files/testing/gnuplot-5.4.rc1.tar.gz

The tutorial seems missing from the 5.4 branch and tarball, but still
available in master.  Is that intentional?


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptations for KORG EX-800 and Poly-800MkII V0.9:
http://Synth.Stromeko.net/Downloads.html#KorgSDada

Re: Fast image markup?

[Dima K. <gn...@di...>]

Ethan A Merritt <me...@uw...> writes:

> On Thursday, 9 April 2020 14:06:01 PDT Dima Kogan wrote:
>
>> You need to zoom out manually because the pixmap isn't a part of the
>> autoscaling logic. Should it be? It is when using "with rgbimage" I
>> think.
>
> I think you misunderstood the meaning of "first" in that context.
> The default xrange is [-10:10], if you place an image with its lower
> corner at 0,0 then the maximum size that will fit in the plot is
>    set pixmap 1 "/tmp/front2.png" at 0,0 width first 10 behind
> i.e. 10 units in the first axis coordinate system

By default gnuplot scales the axes to fit all the data being plotted.
But the images plotted with "set pixmap" aren't included in this logic,
it looks like. I think they probably should be. So the default xrange of
[-10:10] would be expanded to include the image.


> If you use axis coordinates (e.g. "first 10") then it autoscales.
> If you stay in pixel coordinates then it doesn't scale.

I think this is how it should work, but it doesn't. I do this:

  set pixmap 1 "/tmp/front2.png" at 0,0 width first 300 behind
  plot x

This says "plot the image with a corner at 0,0 scaled such the whole
image is 300 wide along the x axis." The plot I get uses the initial
x scaling of [-10:10], which includes only a tiny corner of the image. I
used axis coordinates and it did not autoscale.

>> It'd also be really nice to use the native size of the image somehow.
>> The default "set pixmap" uses is "native image size in pixel
>> coordinates". But what I'd want usually is "native image size in plot
>> axis coordinates", and this isn't available.
>
> What does that even mean?
>
>> I have to know the size,
>> and to communicate it in the command with something like "first 300".
>
> No, that's wrong.  If you use plot coordinates you *don't* need to know
> the original size in pixels because it will be scaled to match the plot.
> You are specifying a width in plot coordinates.

I'm being loose with terminology; sorry. I'm trying to emulate "with
rgbimage". So if I have an image that's 500 pixels wide I want the x
mapping I get with

  set pixmap 1 "/tmp/front2.png" at 0,0 width first 500

But I don't want to specify "500" in the command. Should we maybe have a
width of "0" mean "original image width", or something? Currently
"set pixmap ... width first 0" is equivalent to "set pixmap ..."


> Pixmaps are not intended as a replacement for "with image".
>
> I think of it more like placing a complicated character glyph within
> a page of text.  You can specify where to put it, and you can make
> it larger or smaller, but in the end it is still the same character.
> Rotation, inversion, or other substantial transformation is left to
> images=pictures rather than to pixmaps=characters.

OK, so maybe pixmaps aren't what I actually want then. It's just a
different use case.


>> I tried big images too. I THINK they work, but it's still really slow
>> compared to any image viewer. Is that expected? Should it be faster than
>> what "with rgbimage" does?
>
> "slow" when doing what?  rotating in 3D?
> Can you do that in a normal image viewer?

Just the simplest usage: plot an image with some annotations. No 3d or
any transformations of anything. Compared to image viewers the speed
difference isn't subtle.


> The difference from rgbimage is that all the "with image" commands treat the
> individual pixels as independent data points.  They are stored and manipulated
> separately, taking up a lot of memory.  The "pixmap" mechanism stores and
> dumps a whole png image, which is much more compact, possibly with rescaling.
> The internal storage is persistant, so it doesn't have to be read in and re-processed
> as new data on each plot/replot etc.
>
> So yeah, it should be faster so far as the gnuplot code is concerned.
> What happens after you send it to an X-server is another matter.
> Once it is in the x11 display list it probably doesn't make any difference
> how it got there.

OK. I'm guessing there's both a gnuplot and an x component in the
performance difference. I'll take a look at some point.

Thanks for explaining

dima

Re: Fast image markup?

[Ethan A M. <me...@uw...>]

On Thursday, 9 April 2020 14:06:01 PDT Dima Kogan wrote:
> Ethan A Merritt <me...@uw...> writes:
> 
> > On Thursday, 9 April 2020 11:58:52 PDT Dima Kogan wrote:
> >> A common use case for me is to mark-up images. So gnuplot would plot an
> >> image, and render stuff on top of it.
> >
> > Already there in 5.4 (please test release candidate!)
> 
> What? Wow! OK.
> 
> First off, I've been running 5.4-rc1 (built from source, new
> x11_waitforinput patch cherry-picked) for a few days now. No surprises
> yet.
> 
> I just tried out 'set pixmap'. For me it only appears to work in the
> most basic cases, and I can't tell if it's any faster. I think there're
> several issues at play.
> 
> I tried small images first. Using http://gnuplot.info/figs/front2.png
> from the gnuplot site.
> 
> This works:
> 
>   set terminal x11
>   set pixmap 1 "/tmp/front2.png" at 0,0
>   plot x
> 
> This doesn't:
> 
>   set terminal x11
>   set pixmap 1 "/tmp/front2.png" at 0,0 width first 300 behind
>   plot x

I think you misunderstood the meaning of "first" in that context.
The default xrange is [-10:10], if you place an image with its lower
corner at 0,0 then the maximum size that will fit in the plot is
   set pixmap 1 "/tmp/front2.png" at 0,0 width first 10 behind
i.e. 10 units in the first axis coordinate system

> The plot pops up, and goes away immediately. Might be x11_waitforinput()
> still needs work, because in the qt terminal it does work. x11 is weird
> in lots of cases here, so I'm using qt for the rest of this email. Will
> revisit x11 later.

I have no idea what that's all about, but possibly it is a result of
trying to draw a bitmap that extends waaaay off screen,
since "width first 300" is 30 times too big to fit on screen.

> You need to zoom out manually because the pixmap isn't a part of the
> autoscaling logic. Should it be? It is when using "with rgbimage" I
> think.

Again a misunderstanding?
If you use axis coordinates (e.g. "first 10") then it autoscales.
If you stay in pixel coordinates then it doesn't scale.

> It'd also be really nice to use the native size of the image somehow.
> The default "set pixmap" uses is "native image size in pixel
> coordinates". But what I'd want usually is "native image size in plot
> axis coordinates", and this isn't available.

What does that even mean?

> I have to know the size,
> and to communicate it in the command with something like "first 300".

No, that's wrong.  If you use plot coordinates you *don't* need to know
the original size in pixels because it will be scaled to match the plot.
You are specifying a width in plot coordinates.

> Usually when talking about images, the y axis is flipped: the (0,0)
> pixel is the TOP-left not the BOTTOM-left. So "set pixmap at" should
> maybe place the top-left pixel of the image? In my usage I generally do
> "set yrange [:] reverse". But even in that usage "set pixmap at" places
> the bottom-left pixel. Among the same lines, it should be possible for
> the image pixel axes to follow the plot axes. In that mode "set yrange
> [:] reverse" would plot the image right side up, and with the default
> axis orientation, it would be upside down.

Pixmaps are not intended as a replacement for "with image".

I think of it more like placing a complicated character glyph within
a page of text.  You can specify where to put it, and you can make
it larger or smaller, but in the end it is still the same character.
Rotation, inversion, or other substantial transformation is left to
images=pictures rather than to pixmaps=characters.

> "set pixel height" and "set pixel size" don't play nicely with "set
> yrange [:] reverse", so for instance this doesn't draw an image at all:
> 
>   set yrange [:] reverse
>   set pixmap 1 "/tmp/front2.png" at 0,0 height 300
>   plot x

Huh. That indeed looks like a bug of some sort.
The requested height is still wrong there (should be <10) but yes setting
yrange reverse messes up the placement.  I will have a look at that.

> "set pixel size" doesn't follow the docs, I think. The docs say it wants
> "set pixel size A B", while it actually wants "set pixel size A,B". Note
> the extra comma.

yes.

> I tried big images too. I THINK they work, but it's still really slow
> compared to any image viewer. Is that expected? Should it be faster than
> what "with rgbimage" does?

"slow" when doing what?  rotating in 3D? 
Can you do that in a normal image viewer?

The difference from rgbimage is that all the "with image" commands treat the
individual pixels as independent data points.  They are stored and manipulated
separately, taking up a lot of memory.  The "pixmap" mechanism stores and
dumps a whole png image, which is much more compact, possibly with rescaling.
The internal storage is persistant, so it doesn't have to be read in and re-processed
as new data on each plot/replot etc.   

So yeah, it should be faster so far as the gnuplot code is concerned.
What happens after you send it to an X-server is another matter.
Once it is in the x11 display list it probably doesn't make any difference
how it got there.

	Ethan

Aw: First release candidate for gnuplot version 5.4

[Bastian M. <bma...@we...>]

You can now download binaries for Windows (64bit), OS/2/eComStation/ArcaOS, and
DOS (32bit) from

     https://sf.net/projects/gnuplot/files/gnuplot/testing/

Cheers, Bastian


> Gesendet: Dienstag, 07. April 2020 um 22:58 Uhr
> Von: "Ethan A Merritt" <me...@uw...>
> An: gnu...@li...
> Betreff: First release candidate for gnuplot version 5.4
>
> You can download a source tarball for the first version 5.4 release candidate from
>
>     https://sf.net/projects/gnuplot/files/testing/gnuplot-5.4.rc1.tar.gz
>
> Release notes here:
>
>     http://gnuplot.sourceforge.net/ReleaseNotes_5_4.html
>
> Please test, but there is no great hurry.
> The final release of 5.4 can wait until it is ready.
>
>
> 	cheers, and happy plotting
>
> 		Ethan
>

Re: Fast image markup?

[Dima K. <gn...@di...>]

Ethan A Merritt <me...@uw...> writes:

> On Thursday, 9 April 2020 11:58:52 PDT Dima Kogan wrote:
>> A common use case for me is to mark-up images. So gnuplot would plot an
>> image, and render stuff on top of it.
>
> Already there in 5.4 (please test release candidate!)

What? Wow! OK.

First off, I've been running 5.4-rc1 (built from source, new
x11_waitforinput patch cherry-picked) for a few days now. No surprises
yet.

I just tried out 'set pixmap'. For me it only appears to work in the
most basic cases, and I can't tell if it's any faster. I think there're
several issues at play.

I tried small images first. Using http://gnuplot.info/figs/front2.png
from the gnuplot site.

This works:

  set terminal x11
  set pixmap 1 "/tmp/front2.png" at 0,0
  plot x

This doesn't:

  set terminal x11
  set pixmap 1 "/tmp/front2.png" at 0,0 width first 300 behind
  plot x

The plot pops up, and goes away immediately. Might be x11_waitforinput()
still needs work, because in the qt terminal it does work. x11 is weird
in lots of cases here, so I'm using qt for the rest of this email. Will
revisit x11 later.

You need to zoom out manually because the pixmap isn't a part of the
autoscaling logic. Should it be? It is when using "with rgbimage" I
think.

It'd also be really nice to use the native size of the image somehow.
The default "set pixmap" uses is "native image size in pixel
coordinates". But what I'd want usually is "native image size in plot
axis coordinates", and this isn't available. I have to know the size,
and to communicate it in the command with something like "first 300".

Usually when talking about images, the y axis is flipped: the (0,0)
pixel is the TOP-left not the BOTTOM-left. So "set pixmap at" should
maybe place the top-left pixel of the image? In my usage I generally do
"set yrange [:] reverse". But even in that usage "set pixmap at" places
the bottom-left pixel. Among the same lines, it should be possible for
the image pixel axes to follow the plot axes. In that mode "set yrange
[:] reverse" would plot the image right side up, and with the default
axis orientation, it would be upside down.

"set pixel height" and "set pixel size" don't play nicely with "set
yrange [:] reverse", so for instance this doesn't draw an image at all:

  set yrange [:] reverse
  set pixmap 1 "/tmp/front2.png" at 0,0 height 300
  plot x

"set pixel size" doesn't follow the docs, I think. The docs say it wants
"set pixel size A B", while it actually wants "set pixel size A,B". Note
the extra comma.

I tried big images too. I THINK they work, but it's still really slow
compared to any image viewer. Is that expected? Should it be faster than
what "with rgbimage" does?

Thanks for the feature!

Re: Fast image markup?

[Ethan A M. <me...@uw...>]

On Thursday, 9 April 2020 11:58:52 PDT Dima Kogan wrote:
> Hi.
> 
> Can somebody chime in on how hard/trivial a new feature would be?
> 
> A common use case for me is to mark-up images. So gnuplot would plot an
> image, and render stuff on top of it. Something like this:
> 
>   plot "image.jpg" binary filetype=auto flipy with rgbimage, ...
> 
> This works, but it is always really slow, and if I'm on a machine that's
> not ridiculously overpowered, it runs out of memory.
> 
> My images are several 1000 pixels per side. So big, but not giant. I
> THINK gnuplot is loading each pixel in the image into a separate point
> to be plotted, which makes sense given how flexible this is. But 99% of
> the time for ME, I'm just plotting the image exactly as it is, with
> maybe some independent scaling in x and y. Would it be massively
> effortful to detect that the user is trying to do the simple thing, and
> to run a different code path that treats the image as an image, and is
> thus fast?
> 
> Implementation suggestions? Would we want another terminal command for
> this?

Already there in 5.4 (please test release candidate!)
See

    http://gnuplot.sourceforge.net/demo_5.4/pixmap.html

But I have not benchmarked it or tried to establish what the limits might be
on a memory-limited machine.



gnuplot> help pixmap

 Syntax:
       set pixmap <index> "filename" at <position>
                  {width <w> | height <h> | size <w> <h>}
                  {front|behind} {center}
       show pixmaps
       unset pixmaps

 The `set pixmap` command is similar to `set object` in that it defines an
 object that will appear on subsequent plots. The rectangular array of
 red/green/blue/alpha values making up the pixmap are read from a png, jpeg,
 or gif file.  The position and extent occupied by the pixmap in the gnuplot
 output may be specified in any coordinate system (see `coordinates`).
 The coordinates given by `at <position>` refer to the lower left
 corner of the pixmap unless keyword `center` is present.

[rest of documentation snipped]

	Ethan

Fast image markup?

[Dima K. <gn...@di...>]

Hi.

Can somebody chime in on how hard/trivial a new feature would be?

A common use case for me is to mark-up images. So gnuplot would plot an
image, and render stuff on top of it. Something like this:

  plot "image.jpg" binary filetype=auto flipy with rgbimage, ...

This works, but it is always really slow, and if I'm on a machine that's
not ridiculously overpowered, it runs out of memory.

My images are several 1000 pixels per side. So big, but not giant. I
THINK gnuplot is loading each pixel in the image into a separate point
to be plotted, which makes sense given how flexible this is. But 99% of
the time for ME, I'm just plotting the image exactly as it is, with
maybe some independent scaling in x and y. Would it be massively
effortful to detect that the user is trying to do the simple thing, and
to run a different code path that treats the image as an image, and is
thus fast?

Implementation suggestions? Would we want another terminal command for
this?

Thanks

Re: 3D boxes in version 5.2

[Ethan A M. <me...@uw...>]

On Wednesday, 8 April 2020 08:57:06 PDT daniloassis wrote:
> Hi 
> 
> I would like to know if it is possible to create 3D graphics but using
> boxes as in the example gallery of version 5.5. Is there a patch for
> version 5.2? I needed these graphics urgently for my paper.

Not in 5.2
But you could download the brand new first release candidate for 5.4

  https://sf.net/projects/gnuplot/files/gnuplot/testing/
	gnuplot-5.4.rc1.tar.gz

Let us know how it goes.

3D boxes in version 5.2

[daniloassis <dan...@ut...>]

Hi 

I would like to know if it is possible to create 3D graphics but using
boxes as in the example gallery of version 5.5. Is there a patch for
version 5.2? I needed these graphics urgently for my paper.

Something like this:http://www.gnuplot.info/demo_5.3/3dboxes.html 

-- 
Danilo Renato de Assis
Departamento de Infra-Estrutura e Serviços - Bloco H
Técnico em Tecnologia da Informação.
UTFPR - Reitoria
(41) 3310-4826

First release candidate for gnuplot version 5.4

[Ethan A M. <me...@uw...>]

You can download a source tarball for the first version 5.4 release candidate from 

    https://sf.net/projects/gnuplot/files/testing/gnuplot-5.4.rc1.tar.gz

Release notes here:

    http://gnuplot.sourceforge.net/ReleaseNotes_5_4.html

Please test, but there is no great hurry.
The final release of 5.4 can wait until it is ready.


	cheers, and happy plotting

		Ethan

Re: segmentation fault when using matrix uniform with index

[Juhász P. <pet...@gm...>]

On Thu, 2020-03-19 at 20:21 -0700, Ethan A Merritt wrote:
> On Thursday, 19 March 2020 15:17:21 PDT Juhász Péter wrote:
> > Hi,
> > 
> > Today I've tried to create a heatmap from a CSV-ish file that
> > contained
> > more than one blocks of data, and I've naively thought that I can
> > combine `plot ... index N` with `plot ... matrix rowheaders
> > columnheaders w image`. Admittedly, it's an edge case, but it's not
> > explicitly mentioned as illegal either.
> > 
> > Unfortunately, it doesn't work:
> > for small files it produces an empty plot with some error messages,
> > and
> > for large files it crashes.
> > 
> > The following one-liner will create a file that can be used to
> > reproduce the crash:
> > 
> > perl -le '$N=100; $,=v9; for (1..2) {print "x",qw(a b)x$N; print
> > "a",(1,2)x$N for 1..$N; print"" for 1..2}' > /tmp/foo
> > 
> > then in gnuplot,
> > 
> > plot '/tmp/foo' i 0 matrix rowheaders columnheaders w image
> > 
> > Decrease the value of $N in the one-liner to generate a file that
> > doesn't crash, for me the threshold was 26.
> > 
> > gnuplot version: 5.5 patchlevel 0 last modified 2019-12-22
> > 
> > The crash happens in graphics.c:process_image.
> > 
> > So it appears that `index` doesn't cooperate with `matrix` - I can
> > accept that this combination is not supported but then there should
> > be
> > a note about the fact in the documentation. And it shouldn't crash
> > in
> > any case.
> 
> I agree that the program should never crash just because the command
> or the data is not exactly as expected.  So there certainly is a bug.
> But I don't think it is a question of 'matrix' not recognizing
> 'index.
> 

You're right! Indeed, it's the `columnheaders` mode, not `matrix` in
general that triggers the crash.

> I think the problem is that it is not well defined what it means to
> have column headers in a file with multiple data blocks.
> 
> The program seems to think that there is a single set of column
> headers
> on the first line and does not expect them to reappear in front of
> each
> data block.  If I take the large test file generated by your perl
> jiffy and
> comment out the second row of headers, then the program seems to
> operate acceptably.

Indeed. If I omit `columnheaders`, I get a warning about missing or
undefined values and the image will contain a black row, but it doesn't
crash. With `columnheaders`, I get a crash with `index 0` but not with
`index 1`. Probably there is an off-by-one error somewhere that gets
exposed by the requirement of column headers.

> 
> I will try to figure out why an unexpect line of headers causes
> a segfault, but beyond that I don't know which is more common:
> 
>     1 line of column headers applying to the entire file
> or
>   a separate line of column headers before each data block.
> 
> Thoughts from anyone who deals with this kind of data file?

The CSV (or something-SV) format itself is not well defined, everyone
uses their own home-grown customary formats. In case of a single file
with multiple data blocks:

- from the software implementation standpoint, it makes more sense to
allow and expect one header line per file, at the top of the file,
because that's simpler to explain and simpler to parse;

- from the user experience standpoint, it's better to allow a header
line for every block: they might not even represent the same kind of
data so each block could have a different number of columns with
different headers.

I think there are examples for both in the wild.

best regards,
Peter Juhasz

> 

Re: segmentation fault when using matrix uniform with index

[Ethan A M. <me...@uw...>]

On Thursday, 19 March 2020 15:17:21 PDT Juhász Péter wrote:
> Hi,
> 
> Today I've tried to create a heatmap from a CSV-ish file that contained
> more than one blocks of data, and I've naively thought that I can
> combine `plot ... index N` with `plot ... matrix rowheaders
> columnheaders w image`. Admittedly, it's an edge case, but it's not
> explicitly mentioned as illegal either.
> 
> Unfortunately, it doesn't work:
> for small files it produces an empty plot with some error messages, and
> for large files it crashes.
> 
> The following one-liner will create a file that can be used to
> reproduce the crash:
> 
> perl -le '$N=100; $,=v9; for (1..2) {print "x",qw(a b)x$N; print
> "a",(1,2)x$N for 1..$N; print"" for 1..2}' > /tmp/foo
> 
> then in gnuplot,
> 
> plot '/tmp/foo' i 0 matrix rowheaders columnheaders w image
> 
> Decrease the value of $N in the one-liner to generate a file that
> doesn't crash, for me the threshold was 26.
> 
> gnuplot version: 5.5 patchlevel 0 last modified 2019-12-22
> 
> The crash happens in graphics.c:process_image.
> 
> So it appears that `index` doesn't cooperate with `matrix` - I can
> accept that this combination is not supported but then there should be
> a note about the fact in the documentation. And it shouldn't crash in
> any case.

I agree that the program should never crash just because the command
or the data is not exactly as expected.  So there certainly is a bug.
But I don't think it is a question of 'matrix' not recognizing 'index.

I think the problem is that it is not well defined what it means to
have column headers in a file with multiple data blocks.

The program seems to think that there is a single set of column headers
on the first line and does not expect them to reappear in front of each
data block.  If I take the large test file generated by your perl jiffy and
comment out the second row of headers, then the program seems to
operate acceptably.

I will try to figure out why an unexpect line of headers causes
a segfault, but beyond that I don't know which is more common:

    1 line of column headers applying to the entire file
or
  a separate line of column headers before each data block.

Thoughts from anyone who deals with this kind of data file?

	Ethan
> 
> best regards,
> Peter Juhasz

segmentation fault when using matrix uniform with index

[Juhász P. <pet...@gm...>]

Hi,

Today I've tried to create a heatmap from a CSV-ish file that contained
more than one blocks of data, and I've naively thought that I can
combine `plot ... index N` with `plot ... matrix rowheaders
columnheaders w image`. Admittedly, it's an edge case, but it's not
explicitly mentioned as illegal either.

Unfortunately, it doesn't work:
for small files it produces an empty plot with some error messages, and
for large files it crashes.

The following one-liner will create a file that can be used to
reproduce the crash:

perl -le '$N=100; $,=v9; for (1..2) {print "x",qw(a b)x$N; print
"a",(1,2)x$N for 1..$N; print"" for 1..2}' > /tmp/foo

then in gnuplot,

plot '/tmp/foo' i 0 matrix rowheaders columnheaders w image

Decrease the value of $N in the one-liner to generate a file that
doesn't crash, for me the threshold was 26.

gnuplot version: 5.5 patchlevel 0 last modified 2019-12-22

The crash happens in graphics.c:process_image.

So it appears that `index` doesn't cooperate with `matrix` - I can
accept that this combination is not supported but then there should be
a note about the fact in the documentation. And it shouldn't crash in
any case.

best regards,
Peter Juhasz

Re: Fixes to X11_waitforinput()

[Dima K. <gn...@di...>]

Ethan A Merritt <me...@uw...> writes:

> Your revised code works well in all the tests that I ran, but I am
> sure that there are people with use-cases I didn't test. If you are
> reading this and find a problem with the new code, let us know.

OK, great to hear. I'm going to use these patches in my day-to-day to
see if I hit any issues. Will post here if I do.

Thanks.

Re: Fixes to X11_waitforinput()

[Ethan A M. <me...@uw...>]

On Saturday, 14 March 2020 23:12:24 PDT Dima Kogan wrote:
> Hi.
> 
> I've been working on fixing broken "pause mouse close" behavior on the
> x11 terminal while x-forwarding. This is
> 
>   https://sourceforge.net/p/gnuplot/bugs/2208/
> 
> In the process I discovered that I already encountered weirdness in this
> area previously:
> 
>   https://sourceforge.net/p/gnuplot/mailman/message/36440354/
> 
> I just fixed all the issues, I think. These are in 2 patches on this
> branch:
> 
>   https://github.com/dkogan/gnuplot/commits/fixed-x11-waitforinput
> 
> The first patch improves diagnostic printing in this area. Instead of
> printing enums as NUMBERS, it prints the enum NAMES. This is infinitely
> more useful.
> 
> The second patch contains the actual fixes. It looks more invasive than
> it is. 99% of the modifications are in X11_waitforinput(). And much of
> it is indentation.
> 
> There were several issues, all of them caused by hoaky input
> multiplexing. I'm attaching two very simple gnuplot scripts that tickled
> the bug. To observe the problems you need to slow down the X connection
> by ssh-ing to any server, and trying to plot through x-forwarding.
> Before any of the fixes this would happen:
> 
>     $ gnuplot < mild.pause.mouse.close.broken.gp
> 
>     gnuplot> ause mouse close
>              ^
>              line 0: invalid command
> 

> [...snip...]

> It works for me so far, but this is an error-prone area. Can somebody
> take a look? The new approach is much better, I think, but more testing
> would be good.

It has been a long time since I looked at the x11 input multiplexing code
but I remember that it was a mess and many observed problems were
dealt with by slapping a special case in rather than rethinking the design.
So yes, a re-write is a good thing.

Your revised code works well in all the tests that I ran, but I am sure that
there are people with use-cases I didn't test.  If you are reading this and
find a problem with the new code, let us know.

Anyhow, I've imported your fixes into the main code repository.
Keep up the good work. 
	
	Ethan

Re: Fixes to X11_waitforinput()

[Dima K. <gn...@di...>]

Ethan Merritt <eam...@gm...> writes:

> Can you send me an idiot's guide to how to extract those patches from
> the github site?   I can view them as html pages, but cannot figure out
> how to download in a form I can feed to `patch`.
>
> Alternatively can you tell me, still a naive git user, how to fetch or
> cherry-pick these commits from your git instance?

Sure

  cd ethans_gnuplot_directory
  git remote add dima gi...@gi...:dkogan/gnuplot.git
  git fetch dima

The patches are now reference-able as

  dima/fixed-x11-waitforinput^

and

  dima/fixed-x11-waitforinput

You can 'git cherry-pick' these, or whatever else you want to do.

Re: Fixes to X11_waitforinput()

[Ethan M. <eam...@gm...>]

On Saturday, 14 March 2020 23:12:24 PDT Dima Kogan wrote:
> Hi.
> 
> I've been working on fixing broken "pause mouse close" behavior on the
> x11 terminal while x-forwarding. This is
> 
>   https://sourceforge.net/p/gnuplot/bugs/2208/
> 
> In the process I discovered that I already encountered weirdness in this
> area previously:
> 
>   https://sourceforge.net/p/gnuplot/mailman/message/36440354/
> 
> I just fixed all the issues, I think. These are in 2 patches on this
> branch:
> 
>   https://github.com/dkogan/gnuplot/commits/fixed-x11-waitforinput

Can you send me an idiot's guide to how to extract those patches from
the github site?   I can view them as html pages, but cannot figure out
how to download in a form I can feed to `patch`. 

Alternatively can you tell me, still a naive git user, how to fetch or
cherry-pick these commits from your git instance?

	thanks,

		Ethan


> The first patch improves diagnostic printing in this area. Instead of
> printing enums as NUMBERS, it prints the enum NAMES. This is infinitely
> more useful.
> 
> The second patch contains the actual fixes. It looks more invasive than
> it is. 99% of the modifications are in X11_waitforinput(). And much of
> it is indentation.
> 
> There were several issues, all of them caused by hoaky input
> multiplexing. I'm attaching two very simple gnuplot scripts that tickled
> the bug. To observe the problems you need to slow down the X connection
> by ssh-ing to any server, and trying to plot through x-forwarding.
> Before any of the fixes this would happen:
> 
>     $ gnuplot < mild.pause.mouse.close.broken.gp
> 
>     gnuplot> ause mouse close
>              ^
>              line 0: invalid command
> 
> This was caused by this block of code:
> 
>     if (IPC_LOCK) {
> #ifdef HAVE_USLEEP
> 	usleep(100);
> #endif
>         if (repeat_count++ < 10000)
> 	    goto AGAIN;
>     }
> 
> The slower link means that 10000 is too low. Bumping it up fixes the
> problem for this data file. But if you then run the other test file, you
> still see issues:
> 
>     $ gnuplot < hard.pause.mouse.close.broken.gp
> 
>     gnuplot> splse mouse close
>              ^
>              line 0: invalid command
> 
> This is an entirely different bug. I chased it down a few days ago, so I
> don't remember the exact details. The gist of the issue was that we
> would read some commands from ipc_back_fd then we would read some stuff
> from stdin, and then we would go back, and read the rest of the stuff
> from ipc_back_fd. This non-atomicity would break stuff. The fix was to
> read ALL the available ipc_back_fd data when possible, BEFORE reading
> anything from stdin.
> 
> The existing code was hoaky and mysterious. There were weird sleeps all
> over the place, and lots of state. The patch here refactors
> X11_waitforinput() to get rid of the sleeps and unneeded loops. The
> IPC_LOCK is gone, and is now another possible option argument to
> X11_waitforinput(). And there're better comments, I think.
> 
> It works for me so far, but this is an error-prone area. Can somebody
> take a look? The new approach is much better, I think, but more testing
> would be good.
> 
> Thanks.
> 
> 

Fixes to X11_waitforinput()

[Dima K. <gn...@di...>]

Hi.

I've been working on fixing broken "pause mouse close" behavior on the
x11 terminal while x-forwarding. This is

  https://sourceforge.net/p/gnuplot/bugs/2208/

In the process I discovered that I already encountered weirdness in this
area previously:

  https://sourceforge.net/p/gnuplot/mailman/message/36440354/

I just fixed all the issues, I think. These are in 2 patches on this
branch:

  https://github.com/dkogan/gnuplot/commits/fixed-x11-waitforinput

The first patch improves diagnostic printing in this area. Instead of
printing enums as NUMBERS, it prints the enum NAMES. This is infinitely
more useful.

The second patch contains the actual fixes. It looks more invasive than
it is. 99% of the modifications are in X11_waitforinput(). And much of
it is indentation.

There were several issues, all of them caused by hoaky input
multiplexing. I'm attaching two very simple gnuplot scripts that tickled
the bug. To observe the problems you need to slow down the X connection
by ssh-ing to any server, and trying to plot through x-forwarding.
Before any of the fixes this would happen:

    $ gnuplot < mild.pause.mouse.close.broken.gp

    gnuplot> ause mouse close
             ^
             line 0: invalid command

This was caused by this block of code:

    if (IPC_LOCK) {
#ifdef HAVE_USLEEP
	usleep(100);
#endif
        if (repeat_count++ < 10000)
	    goto AGAIN;
    }

The slower link means that 10000 is too low. Bumping it up fixes the
problem for this data file. But if you then run the other test file, you
still see issues:

    $ gnuplot < hard.pause.mouse.close.broken.gp

    gnuplot> splse mouse close
             ^
             line 0: invalid command

This is an entirely different bug. I chased it down a few days ago, so I
don't remember the exact details. The gist of the issue was that we
would read some commands from ipc_back_fd then we would read some stuff
from stdin, and then we would go back, and read the rest of the stuff
from ipc_back_fd. This non-atomicity would break stuff. The fix was to
read ALL the available ipc_back_fd data when possible, BEFORE reading
anything from stdin.

The existing code was hoaky and mysterious. There were weird sleeps all
over the place, and lots of state. The patch here refactors
X11_waitforinput() to get rid of the sleeps and unneeded loops. The
IPC_LOCK is gone, and is now another possible option argument to
X11_waitforinput(). And there're better comments, I think.

It works for me so far, but this is an error-prone area. Can somebody
take a look? The new approach is much better, I think, but more testing
would be good.

Thanks.

Re: geographic plotting

[Manfred S. <man...@gm...>]

Am 07.03.20 um 00:58 schrieb Ethan A Merritt:
> On Wednesday, 4 March 2020 12:34:45 PST Allin Cottrell wrote:
>> On Wed, 4 Mar 2020, Manfred Schwarb wrote:
>>
>>> More to the topic, I think the shape format is proprietary,
>>> complicated, spread over multiple files, limited (number of
>>> fields, length of field names, file size,...), so in one word:
>>> "legacy".
>>>
>>> So I don't think it is an appropriate format for interfacing with
>>> gnuplot. I made positive experiences with GeoJSON, it is a simple,
>>> very versatile format, and I think most GIS programs can read and
>>> write this format, so why not use some geojson routine and put it
>>> into gnuplot?
>>
>> I'm all in favor of nicer and non-proprietary formats, but what's
>> the comparison in terms of availability of files representing
>> polygons one might want (US states, counties or commuting zones, EU
>> countries or regions, etc.)? It's my impression that you can easily
>> pick up shapefiles for any/all of these -- nasty as the format may
>> be -- but what about GeoJSON files?
>>
>> Allin Cottrell

Allin, of course you are right, I guess there are almost no public GeoJSON
polygon files available.
GeoJSON is more of a intermediate format to transfer data from one application
to another one.

For me the choice of a pure shapefile solution simply sounds a bit wierd,
either a very slim solution supporting some simple format or a full-fledged
solution using libgdal (which supports ~50 different formats) would make sense.

>
> Good call.  Geojson looks quite promising.
> A real parser should be easy, but even a quick pass through sed
> or a text editor makes it usable.
> I have posted a proof-of-principle example using US state boundary
> data from a sample file on the geojson development site:
>
>   http://skuld.bmsc.washington.edu/people/merritt/gnuplot/
>

Thanks for the example.
Often the issue is, that the provided shapefiles are a complete mess and you have to
extract some polygons from it. With geojson, at least with the variant that gdal writes,
this is very simple: Each polygon is on its own line.
So, assuming the polygons are annotated, you can grep for a polygon and re-add header and
footer afterwards.

> 	Ethan
>
>
>
>
> _______________________________________________
> gnuplot-beta mailing list
> gnu...@li...
> Membership management via: https://lists.sourceforge.net/lists/listinfo/gnuplot-beta
>

Re: geographic plotting

[Ethan A M. <me...@uw...>]

On Wednesday, 4 March 2020 12:34:45 PST Allin Cottrell wrote:
> On Wed, 4 Mar 2020, Manfred Schwarb wrote:
> 
> > More to the topic, I think the shape format is proprietary, 
> > complicated, spread over multiple files, limited (number of 
> > fields, length of field names, file size,...), so in one word: 
> > "legacy".
> >
> > So I don't think it is an appropriate format for interfacing with 
> > gnuplot. I made positive experiences with GeoJSON, it is a simple, 
> > very versatile format, and I think most GIS programs can read and 
> > write this format, so why not use some geojson routine and put it 
> > into gnuplot?
> 
> I'm all in favor of nicer and non-proprietary formats, but what's 
> the comparison in terms of availability of files representing 
> polygons one might want (US states, counties or commuting zones, EU 
> countries or regions, etc.)? It's my impression that you can easily 
> pick up shapefiles for any/all of these -- nasty as the format may 
> be -- but what about GeoJSON files?
> 
> Allin Cottrell

Good call.  Geojson looks quite promising.
A real parser should be easy, but even a quick pass through sed
or a text editor makes it usable.
I have posted a proof-of-principle example using US state boundary
data from a sample file on the geojson development site:

  http://skuld.bmsc.washington.edu/people/merritt/gnuplot/

	Ethan

Re: geographic plotting

[Allin C. <cot...@wf...>]

On Wed, 4 Mar 2020, Manfred Schwarb wrote:

> Am 04.03.20 um 16:07 schrieb Allin Cottrell:
>> On Wed, 4 Mar 2020, Manfred Schwarb wrote:
>>
>>>> This may be putting the cart before the horse, but...
>>>> Do you think it would be helpful to teach gnuplot to read shape files?
>>>> I imagine it would be possible to test for the presence of shapelib.so
>>>> and provide a binary input mode:
>>>>
>>>>    splot <data> binary filetype=shapelib  with polygons
>>>>
>>>> I have never worked with this libary so I don't know if it is organized
>>>> in such a way that this would be possible.
>>>
>>> I do it using the gdal software and some shell magic:
>>>   # ogr2ogr -f "GMT" gaga.gmt gaga.shp
>>>   # grep -v "^#" gaga.gmt | sed 's/>//' > gaga.txt
>>
>> That's also the approach taken by Bob Mesibov at
>> https://www.datafix.com.au/BASHing/2018-10-31.html
>
> Well, you don't need 2 blank lines for separation, a single blank 
> line is enough.

OK, but not a big deal.

> More to the topic, I think the shape format is proprietary, 
> complicated, spread over multiple files, limited (number of 
> fields, length of field names, file size,...), so in one word: 
> "legacy".
>
> So I don't think it is an appropriate format for interfacing with 
> gnuplot. I made positive experiences with GeoJSON, it is a simple, 
> very versatile format, and I think most GIS programs can read and 
> write this format, so why not use some geojson routine and put it 
> into gnuplot?

I'm all in favor of nicer and non-proprietary formats, but what's 
the comparison in terms of availability of files representing 
polygons one might want (US states, counties or commuting zones, EU 
countries or regions, etc.)? It's my impression that you can easily 
pick up shapefiles for any/all of these -- nasty as the format may 
be -- but what about GeoJSON files?

Allin Cottrell

Re: geographic plotting

[Manfred S. <man...@gm...>]

Am 04.03.20 um 16:07 schrieb Allin Cottrell:
> On Wed, 4 Mar 2020, Manfred Schwarb wrote:
>
>>> This may be putting the cart before the horse, but...
>>> Do you think it would be helpful to teach gnuplot to read shape files?
>>> I imagine it would be possible to test for the presence of shapelib.so
>>> and provide a binary input mode:
>>>
>>>    splot <data> binary filetype=shapelib  with polygons
>>>
>>> I have never worked with this libary so I don't know if it is organized
>>> in such a way that this would be possible.
>>
>> I do it using the gdal software and some shell magic:
>>   # ogr2ogr -f "GMT" gaga.gmt gaga.shp
>>   # grep -v "^#" gaga.gmt | sed 's/>//' > gaga.txt
>
> That's also the approach taken by Bob Mesibov at
> https://www.datafix.com.au/BASHing/2018-10-31.html
>

Well, you don't need 2 blank lines for separation, a single blank line is enough.

More to the topic, I think the shape format is proprietary, complicated, spread over
multiple files, limited (number of fields, length of field names, file size,...),
so in one word: "legacy".
So I don't think it is an appropriate format for interfacing with gnuplot.
I made positive experiences with GeoJSON, it is a simple, very versatile format,
and I think most GIS programs can read and write this format, so why not
use some geojson routine and put it into gnuplot?

Cheers,
Manfred


> I'm hopeful that libshape can do the job without requiring the whole
> of gdal. Libshape was developed by Frank Warmerdam, the guy who
> initially developed gdal, and is used by gdal's OGR library.
>
> Allin Cottrell
>