gnuplot-beta — Developers and beta testers of new gnuplot code

Re: Fast image markup?

[Ethan A M. <me...@uw...>]

On Thursday, 9 April 2020 14:06:01 PDT Dima Kogan wrote:
> Ethan A Merritt <me...@uw...> writes:
> 
> > On Thursday, 9 April 2020 11:58:52 PDT Dima Kogan wrote:
> >> A common use case for me is to mark-up images. So gnuplot would plot an
> >> image, and render stuff on top of it.
> >
> > Already there in 5.4 (please test release candidate!)
> 
> What? Wow! OK.
> 
> First off, I've been running 5.4-rc1 (built from source, new
> x11_waitforinput patch cherry-picked) for a few days now. No surprises
> yet.
> 
> I just tried out 'set pixmap'. For me it only appears to work in the
> most basic cases, and I can't tell if it's any faster. I think there're
> several issues at play.
> 
> I tried small images first. Using http://gnuplot.info/figs/front2.png
> from the gnuplot site.
> 
> This works:
> 
>   set terminal x11
>   set pixmap 1 "/tmp/front2.png" at 0,0
>   plot x
> 
> This doesn't:
> 
>   set terminal x11
>   set pixmap 1 "/tmp/front2.png" at 0,0 width first 300 behind
>   plot x

I think you misunderstood the meaning of "first" in that context.
The default xrange is [-10:10], if you place an image with its lower
corner at 0,0 then the maximum size that will fit in the plot is
   set pixmap 1 "/tmp/front2.png" at 0,0 width first 10 behind
i.e. 10 units in the first axis coordinate system

> The plot pops up, and goes away immediately. Might be x11_waitforinput()
> still needs work, because in the qt terminal it does work. x11 is weird
> in lots of cases here, so I'm using qt for the rest of this email. Will
> revisit x11 later.

I have no idea what that's all about, but possibly it is a result of
trying to draw a bitmap that extends waaaay off screen,
since "width first 300" is 30 times too big to fit on screen.

> You need to zoom out manually because the pixmap isn't a part of the
> autoscaling logic. Should it be? It is when using "with rgbimage" I
> think.

Again a misunderstanding?
If you use axis coordinates (e.g. "first 10") then it autoscales.
If you stay in pixel coordinates then it doesn't scale.

> It'd also be really nice to use the native size of the image somehow.
> The default "set pixmap" uses is "native image size in pixel
> coordinates". But what I'd want usually is "native image size in plot
> axis coordinates", and this isn't available.

What does that even mean?

> I have to know the size,
> and to communicate it in the command with something like "first 300".

No, that's wrong.  If you use plot coordinates you *don't* need to know
the original size in pixels because it will be scaled to match the plot.
You are specifying a width in plot coordinates.

> Usually when talking about images, the y axis is flipped: the (0,0)
> pixel is the TOP-left not the BOTTOM-left. So "set pixmap at" should
> maybe place the top-left pixel of the image? In my usage I generally do
> "set yrange [:] reverse". But even in that usage "set pixmap at" places
> the bottom-left pixel. Among the same lines, it should be possible for
> the image pixel axes to follow the plot axes. In that mode "set yrange
> [:] reverse" would plot the image right side up, and with the default
> axis orientation, it would be upside down.

Pixmaps are not intended as a replacement for "with image".

I think of it more like placing a complicated character glyph within
a page of text.  You can specify where to put it, and you can make
it larger or smaller, but in the end it is still the same character.
Rotation, inversion, or other substantial transformation is left to
images=pictures rather than to pixmaps=characters.

> "set pixel height" and "set pixel size" don't play nicely with "set
> yrange [:] reverse", so for instance this doesn't draw an image at all:
> 
>   set yrange [:] reverse
>   set pixmap 1 "/tmp/front2.png" at 0,0 height 300
>   plot x

Huh. That indeed looks like a bug of some sort.
The requested height is still wrong there (should be <10) but yes setting
yrange reverse messes up the placement.  I will have a look at that.

> "set pixel size" doesn't follow the docs, I think. The docs say it wants
> "set pixel size A B", while it actually wants "set pixel size A,B". Note
> the extra comma.

yes.

> I tried big images too. I THINK they work, but it's still really slow
> compared to any image viewer. Is that expected? Should it be faster than
> what "with rgbimage" does?

"slow" when doing what?  rotating in 3D? 
Can you do that in a normal image viewer?

The difference from rgbimage is that all the "with image" commands treat the
individual pixels as independent data points.  They are stored and manipulated
separately, taking up a lot of memory.  The "pixmap" mechanism stores and
dumps a whole png image, which is much more compact, possibly with rescaling.
The internal storage is persistant, so it doesn't have to be read in and re-processed
as new data on each plot/replot etc.   

So yeah, it should be faster so far as the gnuplot code is concerned.
What happens after you send it to an X-server is another matter.
Once it is in the x11 display list it probably doesn't make any difference
how it got there.

	Ethan

Aw: First release candidate for gnuplot version 5.4

[Bastian M. <bma...@we...>]

You can now download binaries for Windows (64bit), OS/2/eComStation/ArcaOS, and
DOS (32bit) from

     https://sf.net/projects/gnuplot/files/gnuplot/testing/

Cheers, Bastian


> Gesendet: Dienstag, 07. April 2020 um 22:58 Uhr
> Von: "Ethan A Merritt" <me...@uw...>
> An: gnu...@li...
> Betreff: First release candidate for gnuplot version 5.4
>
> You can download a source tarball for the first version 5.4 release candidate from
>
>     https://sf.net/projects/gnuplot/files/testing/gnuplot-5.4.rc1.tar.gz
>
> Release notes here:
>
>     http://gnuplot.sourceforge.net/ReleaseNotes_5_4.html
>
> Please test, but there is no great hurry.
> The final release of 5.4 can wait until it is ready.
>
>
> 	cheers, and happy plotting
>
> 		Ethan
>

Re: Fast image markup?

[Dima K. <gn...@di...>]

Ethan A Merritt <me...@uw...> writes:

> On Thursday, 9 April 2020 11:58:52 PDT Dima Kogan wrote:
>> A common use case for me is to mark-up images. So gnuplot would plot an
>> image, and render stuff on top of it.
>
> Already there in 5.4 (please test release candidate!)

What? Wow! OK.

First off, I've been running 5.4-rc1 (built from source, new
x11_waitforinput patch cherry-picked) for a few days now. No surprises
yet.

I just tried out 'set pixmap'. For me it only appears to work in the
most basic cases, and I can't tell if it's any faster. I think there're
several issues at play.

I tried small images first. Using http://gnuplot.info/figs/front2.png
from the gnuplot site.

This works:

  set terminal x11
  set pixmap 1 "/tmp/front2.png" at 0,0
  plot x

This doesn't:

  set terminal x11
  set pixmap 1 "/tmp/front2.png" at 0,0 width first 300 behind
  plot x

The plot pops up, and goes away immediately. Might be x11_waitforinput()
still needs work, because in the qt terminal it does work. x11 is weird
in lots of cases here, so I'm using qt for the rest of this email. Will
revisit x11 later.

You need to zoom out manually because the pixmap isn't a part of the
autoscaling logic. Should it be? It is when using "with rgbimage" I
think.

It'd also be really nice to use the native size of the image somehow.
The default "set pixmap" uses is "native image size in pixel
coordinates". But what I'd want usually is "native image size in plot
axis coordinates", and this isn't available. I have to know the size,
and to communicate it in the command with something like "first 300".

Usually when talking about images, the y axis is flipped: the (0,0)
pixel is the TOP-left not the BOTTOM-left. So "set pixmap at" should
maybe place the top-left pixel of the image? In my usage I generally do
"set yrange [:] reverse". But even in that usage "set pixmap at" places
the bottom-left pixel. Among the same lines, it should be possible for
the image pixel axes to follow the plot axes. In that mode "set yrange
[:] reverse" would plot the image right side up, and with the default
axis orientation, it would be upside down.

"set pixel height" and "set pixel size" don't play nicely with "set
yrange [:] reverse", so for instance this doesn't draw an image at all:

  set yrange [:] reverse
  set pixmap 1 "/tmp/front2.png" at 0,0 height 300
  plot x

"set pixel size" doesn't follow the docs, I think. The docs say it wants
"set pixel size A B", while it actually wants "set pixel size A,B". Note
the extra comma.

I tried big images too. I THINK they work, but it's still really slow
compared to any image viewer. Is that expected? Should it be faster than
what "with rgbimage" does?

Thanks for the feature!

Re: Fast image markup?

[Ethan A M. <me...@uw...>]

On Thursday, 9 April 2020 11:58:52 PDT Dima Kogan wrote:
> Hi.
> 
> Can somebody chime in on how hard/trivial a new feature would be?
> 
> A common use case for me is to mark-up images. So gnuplot would plot an
> image, and render stuff on top of it. Something like this:
> 
>   plot "image.jpg" binary filetype=auto flipy with rgbimage, ...
> 
> This works, but it is always really slow, and if I'm on a machine that's
> not ridiculously overpowered, it runs out of memory.
> 
> My images are several 1000 pixels per side. So big, but not giant. I
> THINK gnuplot is loading each pixel in the image into a separate point
> to be plotted, which makes sense given how flexible this is. But 99% of
> the time for ME, I'm just plotting the image exactly as it is, with
> maybe some independent scaling in x and y. Would it be massively
> effortful to detect that the user is trying to do the simple thing, and
> to run a different code path that treats the image as an image, and is
> thus fast?
> 
> Implementation suggestions? Would we want another terminal command for
> this?

Already there in 5.4 (please test release candidate!)
See

    http://gnuplot.sourceforge.net/demo_5.4/pixmap.html

But I have not benchmarked it or tried to establish what the limits might be
on a memory-limited machine.



gnuplot> help pixmap

 Syntax:
       set pixmap <index> "filename" at <position>
                  {width <w> | height <h> | size <w> <h>}
                  {front|behind} {center}
       show pixmaps
       unset pixmaps

 The `set pixmap` command is similar to `set object` in that it defines an
 object that will appear on subsequent plots. The rectangular array of
 red/green/blue/alpha values making up the pixmap are read from a png, jpeg,
 or gif file.  The position and extent occupied by the pixmap in the gnuplot
 output may be specified in any coordinate system (see `coordinates`).
 The coordinates given by `at <position>` refer to the lower left
 corner of the pixmap unless keyword `center` is present.

[rest of documentation snipped]

	Ethan

Fast image markup?

[Dima K. <gn...@di...>]

Hi.

Can somebody chime in on how hard/trivial a new feature would be?

A common use case for me is to mark-up images. So gnuplot would plot an
image, and render stuff on top of it. Something like this:

  plot "image.jpg" binary filetype=auto flipy with rgbimage, ...

This works, but it is always really slow, and if I'm on a machine that's
not ridiculously overpowered, it runs out of memory.

My images are several 1000 pixels per side. So big, but not giant. I
THINK gnuplot is loading each pixel in the image into a separate point
to be plotted, which makes sense given how flexible this is. But 99% of
the time for ME, I'm just plotting the image exactly as it is, with
maybe some independent scaling in x and y. Would it be massively
effortful to detect that the user is trying to do the simple thing, and
to run a different code path that treats the image as an image, and is
thus fast?

Implementation suggestions? Would we want another terminal command for
this?

Thanks

Re: 3D boxes in version 5.2

[Ethan A M. <me...@uw...>]

On Wednesday, 8 April 2020 08:57:06 PDT daniloassis wrote:
> Hi 
> 
> I would like to know if it is possible to create 3D graphics but using
> boxes as in the example gallery of version 5.5. Is there a patch for
> version 5.2? I needed these graphics urgently for my paper.

Not in 5.2
But you could download the brand new first release candidate for 5.4

  https://sf.net/projects/gnuplot/files/gnuplot/testing/
	gnuplot-5.4.rc1.tar.gz

Let us know how it goes.

3D boxes in version 5.2

[daniloassis <dan...@ut...>]

Hi 

I would like to know if it is possible to create 3D graphics but using
boxes as in the example gallery of version 5.5. Is there a patch for
version 5.2? I needed these graphics urgently for my paper.

Something like this:http://www.gnuplot.info/demo_5.3/3dboxes.html 

-- 
Danilo Renato de Assis
Departamento de Infra-Estrutura e Serviços - Bloco H
Técnico em Tecnologia da Informação.
UTFPR - Reitoria
(41) 3310-4826

First release candidate for gnuplot version 5.4

[Ethan A M. <me...@uw...>]

You can download a source tarball for the first version 5.4 release candidate from 

    https://sf.net/projects/gnuplot/files/testing/gnuplot-5.4.rc1.tar.gz

Release notes here:

    http://gnuplot.sourceforge.net/ReleaseNotes_5_4.html

Please test, but there is no great hurry.
The final release of 5.4 can wait until it is ready.


	cheers, and happy plotting

		Ethan

Re: segmentation fault when using matrix uniform with index

[Juhász P. <pet...@gm...>]

On Thu, 2020-03-19 at 20:21 -0700, Ethan A Merritt wrote:
> On Thursday, 19 March 2020 15:17:21 PDT Juhász Péter wrote:
> > Hi,
> > 
> > Today I've tried to create a heatmap from a CSV-ish file that
> > contained
> > more than one blocks of data, and I've naively thought that I can
> > combine `plot ... index N` with `plot ... matrix rowheaders
> > columnheaders w image`. Admittedly, it's an edge case, but it's not
> > explicitly mentioned as illegal either.
> > 
> > Unfortunately, it doesn't work:
> > for small files it produces an empty plot with some error messages,
> > and
> > for large files it crashes.
> > 
> > The following one-liner will create a file that can be used to
> > reproduce the crash:
> > 
> > perl -le '$N=100; $,=v9; for (1..2) {print "x",qw(a b)x$N; print
> > "a",(1,2)x$N for 1..$N; print"" for 1..2}' > /tmp/foo
> > 
> > then in gnuplot,
> > 
> > plot '/tmp/foo' i 0 matrix rowheaders columnheaders w image
> > 
> > Decrease the value of $N in the one-liner to generate a file that
> > doesn't crash, for me the threshold was 26.
> > 
> > gnuplot version: 5.5 patchlevel 0 last modified 2019-12-22
> > 
> > The crash happens in graphics.c:process_image.
> > 
> > So it appears that `index` doesn't cooperate with `matrix` - I can
> > accept that this combination is not supported but then there should
> > be
> > a note about the fact in the documentation. And it shouldn't crash
> > in
> > any case.
> 
> I agree that the program should never crash just because the command
> or the data is not exactly as expected.  So there certainly is a bug.
> But I don't think it is a question of 'matrix' not recognizing
> 'index.
> 

You're right! Indeed, it's the `columnheaders` mode, not `matrix` in
general that triggers the crash.

> I think the problem is that it is not well defined what it means to
> have column headers in a file with multiple data blocks.
> 
> The program seems to think that there is a single set of column
> headers
> on the first line and does not expect them to reappear in front of
> each
> data block.  If I take the large test file generated by your perl
> jiffy and
> comment out the second row of headers, then the program seems to
> operate acceptably.

Indeed. If I omit `columnheaders`, I get a warning about missing or
undefined values and the image will contain a black row, but it doesn't
crash. With `columnheaders`, I get a crash with `index 0` but not with
`index 1`. Probably there is an off-by-one error somewhere that gets
exposed by the requirement of column headers.

> 
> I will try to figure out why an unexpect line of headers causes
> a segfault, but beyond that I don't know which is more common:
> 
>     1 line of column headers applying to the entire file
> or
>   a separate line of column headers before each data block.
> 
> Thoughts from anyone who deals with this kind of data file?

The CSV (or something-SV) format itself is not well defined, everyone
uses their own home-grown customary formats. In case of a single file
with multiple data blocks:

- from the software implementation standpoint, it makes more sense to
allow and expect one header line per file, at the top of the file,
because that's simpler to explain and simpler to parse;

- from the user experience standpoint, it's better to allow a header
line for every block: they might not even represent the same kind of
data so each block could have a different number of columns with
different headers.

I think there are examples for both in the wild.

best regards,
Peter Juhasz

> 

Re: segmentation fault when using matrix uniform with index

[Ethan A M. <me...@uw...>]

On Thursday, 19 March 2020 15:17:21 PDT Juhász Péter wrote:
> Hi,
> 
> Today I've tried to create a heatmap from a CSV-ish file that contained
> more than one blocks of data, and I've naively thought that I can
> combine `plot ... index N` with `plot ... matrix rowheaders
> columnheaders w image`. Admittedly, it's an edge case, but it's not
> explicitly mentioned as illegal either.
> 
> Unfortunately, it doesn't work:
> for small files it produces an empty plot with some error messages, and
> for large files it crashes.
> 
> The following one-liner will create a file that can be used to
> reproduce the crash:
> 
> perl -le '$N=100; $,=v9; for (1..2) {print "x",qw(a b)x$N; print
> "a",(1,2)x$N for 1..$N; print"" for 1..2}' > /tmp/foo
> 
> then in gnuplot,
> 
> plot '/tmp/foo' i 0 matrix rowheaders columnheaders w image
> 
> Decrease the value of $N in the one-liner to generate a file that
> doesn't crash, for me the threshold was 26.
> 
> gnuplot version: 5.5 patchlevel 0 last modified 2019-12-22
> 
> The crash happens in graphics.c:process_image.
> 
> So it appears that `index` doesn't cooperate with `matrix` - I can
> accept that this combination is not supported but then there should be
> a note about the fact in the documentation. And it shouldn't crash in
> any case.

I agree that the program should never crash just because the command
or the data is not exactly as expected.  So there certainly is a bug.
But I don't think it is a question of 'matrix' not recognizing 'index.

I think the problem is that it is not well defined what it means to
have column headers in a file with multiple data blocks.

The program seems to think that there is a single set of column headers
on the first line and does not expect them to reappear in front of each
data block.  If I take the large test file generated by your perl jiffy and
comment out the second row of headers, then the program seems to
operate acceptably.

I will try to figure out why an unexpect line of headers causes
a segfault, but beyond that I don't know which is more common:

    1 line of column headers applying to the entire file
or
  a separate line of column headers before each data block.

Thoughts from anyone who deals with this kind of data file?

	Ethan
> 
> best regards,
> Peter Juhasz

segmentation fault when using matrix uniform with index

[Juhász P. <pet...@gm...>]

Hi,

Today I've tried to create a heatmap from a CSV-ish file that contained
more than one blocks of data, and I've naively thought that I can
combine `plot ... index N` with `plot ... matrix rowheaders
columnheaders w image`. Admittedly, it's an edge case, but it's not
explicitly mentioned as illegal either.

Unfortunately, it doesn't work:
for small files it produces an empty plot with some error messages, and
for large files it crashes.

The following one-liner will create a file that can be used to
reproduce the crash:

perl -le '$N=100; $,=v9; for (1..2) {print "x",qw(a b)x$N; print
"a",(1,2)x$N for 1..$N; print"" for 1..2}' > /tmp/foo

then in gnuplot,

plot '/tmp/foo' i 0 matrix rowheaders columnheaders w image

Decrease the value of $N in the one-liner to generate a file that
doesn't crash, for me the threshold was 26.

gnuplot version: 5.5 patchlevel 0 last modified 2019-12-22

The crash happens in graphics.c:process_image.

So it appears that `index` doesn't cooperate with `matrix` - I can
accept that this combination is not supported but then there should be
a note about the fact in the documentation. And it shouldn't crash in
any case.

best regards,
Peter Juhasz

Re: Fixes to X11_waitforinput()

[Dima K. <gn...@di...>]

Ethan A Merritt <me...@uw...> writes:

> Your revised code works well in all the tests that I ran, but I am
> sure that there are people with use-cases I didn't test. If you are
> reading this and find a problem with the new code, let us know.

OK, great to hear. I'm going to use these patches in my day-to-day to
see if I hit any issues. Will post here if I do.

Thanks.

Re: Fixes to X11_waitforinput()

[Ethan A M. <me...@uw...>]

On Saturday, 14 March 2020 23:12:24 PDT Dima Kogan wrote:
> Hi.
> 
> I've been working on fixing broken "pause mouse close" behavior on the
> x11 terminal while x-forwarding. This is
> 
>   https://sourceforge.net/p/gnuplot/bugs/2208/
> 
> In the process I discovered that I already encountered weirdness in this
> area previously:
> 
>   https://sourceforge.net/p/gnuplot/mailman/message/36440354/
> 
> I just fixed all the issues, I think. These are in 2 patches on this
> branch:
> 
>   https://github.com/dkogan/gnuplot/commits/fixed-x11-waitforinput
> 
> The first patch improves diagnostic printing in this area. Instead of
> printing enums as NUMBERS, it prints the enum NAMES. This is infinitely
> more useful.
> 
> The second patch contains the actual fixes. It looks more invasive than
> it is. 99% of the modifications are in X11_waitforinput(). And much of
> it is indentation.
> 
> There were several issues, all of them caused by hoaky input
> multiplexing. I'm attaching two very simple gnuplot scripts that tickled
> the bug. To observe the problems you need to slow down the X connection
> by ssh-ing to any server, and trying to plot through x-forwarding.
> Before any of the fixes this would happen:
> 
>     $ gnuplot < mild.pause.mouse.close.broken.gp
> 
>     gnuplot> ause mouse close
>              ^
>              line 0: invalid command
> 

> [...snip...]

> It works for me so far, but this is an error-prone area. Can somebody
> take a look? The new approach is much better, I think, but more testing
> would be good.

It has been a long time since I looked at the x11 input multiplexing code
but I remember that it was a mess and many observed problems were
dealt with by slapping a special case in rather than rethinking the design.
So yes, a re-write is a good thing.

Your revised code works well in all the tests that I ran, but I am sure that
there are people with use-cases I didn't test.  If you are reading this and
find a problem with the new code, let us know.

Anyhow, I've imported your fixes into the main code repository.
Keep up the good work. 
	
	Ethan

Re: Fixes to X11_waitforinput()

[Dima K. <gn...@di...>]

Ethan Merritt <eam...@gm...> writes:

> Can you send me an idiot's guide to how to extract those patches from
> the github site?   I can view them as html pages, but cannot figure out
> how to download in a form I can feed to `patch`.
>
> Alternatively can you tell me, still a naive git user, how to fetch or
> cherry-pick these commits from your git instance?

Sure

  cd ethans_gnuplot_directory
  git remote add dima gi...@gi...:dkogan/gnuplot.git
  git fetch dima

The patches are now reference-able as

  dima/fixed-x11-waitforinput^

and

  dima/fixed-x11-waitforinput

You can 'git cherry-pick' these, or whatever else you want to do.

Re: Fixes to X11_waitforinput()

[Ethan M. <eam...@gm...>]

On Saturday, 14 March 2020 23:12:24 PDT Dima Kogan wrote:
> Hi.
> 
> I've been working on fixing broken "pause mouse close" behavior on the
> x11 terminal while x-forwarding. This is
> 
>   https://sourceforge.net/p/gnuplot/bugs/2208/
> 
> In the process I discovered that I already encountered weirdness in this
> area previously:
> 
>   https://sourceforge.net/p/gnuplot/mailman/message/36440354/
> 
> I just fixed all the issues, I think. These are in 2 patches on this
> branch:
> 
>   https://github.com/dkogan/gnuplot/commits/fixed-x11-waitforinput

Can you send me an idiot's guide to how to extract those patches from
the github site?   I can view them as html pages, but cannot figure out
how to download in a form I can feed to `patch`. 

Alternatively can you tell me, still a naive git user, how to fetch or
cherry-pick these commits from your git instance?

	thanks,

		Ethan


> The first patch improves diagnostic printing in this area. Instead of
> printing enums as NUMBERS, it prints the enum NAMES. This is infinitely
> more useful.
> 
> The second patch contains the actual fixes. It looks more invasive than
> it is. 99% of the modifications are in X11_waitforinput(). And much of
> it is indentation.
> 
> There were several issues, all of them caused by hoaky input
> multiplexing. I'm attaching two very simple gnuplot scripts that tickled
> the bug. To observe the problems you need to slow down the X connection
> by ssh-ing to any server, and trying to plot through x-forwarding.
> Before any of the fixes this would happen:
> 
>     $ gnuplot < mild.pause.mouse.close.broken.gp
> 
>     gnuplot> ause mouse close
>              ^
>              line 0: invalid command
> 
> This was caused by this block of code:
> 
>     if (IPC_LOCK) {
> #ifdef HAVE_USLEEP
> 	usleep(100);
> #endif
>         if (repeat_count++ < 10000)
> 	    goto AGAIN;
>     }
> 
> The slower link means that 10000 is too low. Bumping it up fixes the
> problem for this data file. But if you then run the other test file, you
> still see issues:
> 
>     $ gnuplot < hard.pause.mouse.close.broken.gp
> 
>     gnuplot> splse mouse close
>              ^
>              line 0: invalid command
> 
> This is an entirely different bug. I chased it down a few days ago, so I
> don't remember the exact details. The gist of the issue was that we
> would read some commands from ipc_back_fd then we would read some stuff
> from stdin, and then we would go back, and read the rest of the stuff
> from ipc_back_fd. This non-atomicity would break stuff. The fix was to
> read ALL the available ipc_back_fd data when possible, BEFORE reading
> anything from stdin.
> 
> The existing code was hoaky and mysterious. There were weird sleeps all
> over the place, and lots of state. The patch here refactors
> X11_waitforinput() to get rid of the sleeps and unneeded loops. The
> IPC_LOCK is gone, and is now another possible option argument to
> X11_waitforinput(). And there're better comments, I think.
> 
> It works for me so far, but this is an error-prone area. Can somebody
> take a look? The new approach is much better, I think, but more testing
> would be good.
> 
> Thanks.
> 
> 

Fixes to X11_waitforinput()

[Dima K. <gn...@di...>]

Hi.

I've been working on fixing broken "pause mouse close" behavior on the
x11 terminal while x-forwarding. This is

  https://sourceforge.net/p/gnuplot/bugs/2208/

In the process I discovered that I already encountered weirdness in this
area previously:

  https://sourceforge.net/p/gnuplot/mailman/message/36440354/

I just fixed all the issues, I think. These are in 2 patches on this
branch:

  https://github.com/dkogan/gnuplot/commits/fixed-x11-waitforinput

The first patch improves diagnostic printing in this area. Instead of
printing enums as NUMBERS, it prints the enum NAMES. This is infinitely
more useful.

The second patch contains the actual fixes. It looks more invasive than
it is. 99% of the modifications are in X11_waitforinput(). And much of
it is indentation.

There were several issues, all of them caused by hoaky input
multiplexing. I'm attaching two very simple gnuplot scripts that tickled
the bug. To observe the problems you need to slow down the X connection
by ssh-ing to any server, and trying to plot through x-forwarding.
Before any of the fixes this would happen:

    $ gnuplot < mild.pause.mouse.close.broken.gp

    gnuplot> ause mouse close
             ^
             line 0: invalid command

This was caused by this block of code:

    if (IPC_LOCK) {
#ifdef HAVE_USLEEP
	usleep(100);
#endif
        if (repeat_count++ < 10000)
	    goto AGAIN;
    }

The slower link means that 10000 is too low. Bumping it up fixes the
problem for this data file. But if you then run the other test file, you
still see issues:

    $ gnuplot < hard.pause.mouse.close.broken.gp

    gnuplot> splse mouse close
             ^
             line 0: invalid command

This is an entirely different bug. I chased it down a few days ago, so I
don't remember the exact details. The gist of the issue was that we
would read some commands from ipc_back_fd then we would read some stuff
from stdin, and then we would go back, and read the rest of the stuff
from ipc_back_fd. This non-atomicity would break stuff. The fix was to
read ALL the available ipc_back_fd data when possible, BEFORE reading
anything from stdin.

The existing code was hoaky and mysterious. There were weird sleeps all
over the place, and lots of state. The patch here refactors
X11_waitforinput() to get rid of the sleeps and unneeded loops. The
IPC_LOCK is gone, and is now another possible option argument to
X11_waitforinput(). And there're better comments, I think.

It works for me so far, but this is an error-prone area. Can somebody
take a look? The new approach is much better, I think, but more testing
would be good.

Thanks.

Re: geographic plotting

[Manfred S. <man...@gm...>]

Am 07.03.20 um 00:58 schrieb Ethan A Merritt:
> On Wednesday, 4 March 2020 12:34:45 PST Allin Cottrell wrote:
>> On Wed, 4 Mar 2020, Manfred Schwarb wrote:
>>
>>> More to the topic, I think the shape format is proprietary,
>>> complicated, spread over multiple files, limited (number of
>>> fields, length of field names, file size,...), so in one word:
>>> "legacy".
>>>
>>> So I don't think it is an appropriate format for interfacing with
>>> gnuplot. I made positive experiences with GeoJSON, it is a simple,
>>> very versatile format, and I think most GIS programs can read and
>>> write this format, so why not use some geojson routine and put it
>>> into gnuplot?
>>
>> I'm all in favor of nicer and non-proprietary formats, but what's
>> the comparison in terms of availability of files representing
>> polygons one might want (US states, counties or commuting zones, EU
>> countries or regions, etc.)? It's my impression that you can easily
>> pick up shapefiles for any/all of these -- nasty as the format may
>> be -- but what about GeoJSON files?
>>
>> Allin Cottrell

Allin, of course you are right, I guess there are almost no public GeoJSON
polygon files available.
GeoJSON is more of a intermediate format to transfer data from one application
to another one.

For me the choice of a pure shapefile solution simply sounds a bit wierd,
either a very slim solution supporting some simple format or a full-fledged
solution using libgdal (which supports ~50 different formats) would make sense.

>
> Good call.  Geojson looks quite promising.
> A real parser should be easy, but even a quick pass through sed
> or a text editor makes it usable.
> I have posted a proof-of-principle example using US state boundary
> data from a sample file on the geojson development site:
>
>   http://skuld.bmsc.washington.edu/people/merritt/gnuplot/
>

Thanks for the example.
Often the issue is, that the provided shapefiles are a complete mess and you have to
extract some polygons from it. With geojson, at least with the variant that gdal writes,
this is very simple: Each polygon is on its own line.
So, assuming the polygons are annotated, you can grep for a polygon and re-add header and
footer afterwards.

> 	Ethan
>
>
>
>
> _______________________________________________
> gnuplot-beta mailing list
> gnu...@li...
> Membership management via: https://lists.sourceforge.net/lists/listinfo/gnuplot-beta
>

Re: geographic plotting

[Ethan A M. <me...@uw...>]

On Wednesday, 4 March 2020 12:34:45 PST Allin Cottrell wrote:
> On Wed, 4 Mar 2020, Manfred Schwarb wrote:
> 
> > More to the topic, I think the shape format is proprietary, 
> > complicated, spread over multiple files, limited (number of 
> > fields, length of field names, file size,...), so in one word: 
> > "legacy".
> >
> > So I don't think it is an appropriate format for interfacing with 
> > gnuplot. I made positive experiences with GeoJSON, it is a simple, 
> > very versatile format, and I think most GIS programs can read and 
> > write this format, so why not use some geojson routine and put it 
> > into gnuplot?
> 
> I'm all in favor of nicer and non-proprietary formats, but what's 
> the comparison in terms of availability of files representing 
> polygons one might want (US states, counties or commuting zones, EU 
> countries or regions, etc.)? It's my impression that you can easily 
> pick up shapefiles for any/all of these -- nasty as the format may 
> be -- but what about GeoJSON files?
> 
> Allin Cottrell

Good call.  Geojson looks quite promising.
A real parser should be easy, but even a quick pass through sed
or a text editor makes it usable.
I have posted a proof-of-principle example using US state boundary
data from a sample file on the geojson development site:

  http://skuld.bmsc.washington.edu/people/merritt/gnuplot/

	Ethan

Re: geographic plotting

[Allin C. <cot...@wf...>]

On Wed, 4 Mar 2020, Manfred Schwarb wrote:

> Am 04.03.20 um 16:07 schrieb Allin Cottrell:
>> On Wed, 4 Mar 2020, Manfred Schwarb wrote:
>>
>>>> This may be putting the cart before the horse, but...
>>>> Do you think it would be helpful to teach gnuplot to read shape files?
>>>> I imagine it would be possible to test for the presence of shapelib.so
>>>> and provide a binary input mode:
>>>>
>>>>    splot <data> binary filetype=shapelib  with polygons
>>>>
>>>> I have never worked with this libary so I don't know if it is organized
>>>> in such a way that this would be possible.
>>>
>>> I do it using the gdal software and some shell magic:
>>>   # ogr2ogr -f "GMT" gaga.gmt gaga.shp
>>>   # grep -v "^#" gaga.gmt | sed 's/>//' > gaga.txt
>>
>> That's also the approach taken by Bob Mesibov at
>> https://www.datafix.com.au/BASHing/2018-10-31.html
>
> Well, you don't need 2 blank lines for separation, a single blank 
> line is enough.

OK, but not a big deal.

> More to the topic, I think the shape format is proprietary, 
> complicated, spread over multiple files, limited (number of 
> fields, length of field names, file size,...), so in one word: 
> "legacy".
>
> So I don't think it is an appropriate format for interfacing with 
> gnuplot. I made positive experiences with GeoJSON, it is a simple, 
> very versatile format, and I think most GIS programs can read and 
> write this format, so why not use some geojson routine and put it 
> into gnuplot?

I'm all in favor of nicer and non-proprietary formats, but what's 
the comparison in terms of availability of files representing 
polygons one might want (US states, counties or commuting zones, EU 
countries or regions, etc.)? It's my impression that you can easily 
pick up shapefiles for any/all of these -- nasty as the format may 
be -- but what about GeoJSON files?

Allin Cottrell

Re: geographic plotting

[Manfred S. <man...@gm...>]

Am 04.03.20 um 16:07 schrieb Allin Cottrell:
> On Wed, 4 Mar 2020, Manfred Schwarb wrote:
>
>>> This may be putting the cart before the horse, but...
>>> Do you think it would be helpful to teach gnuplot to read shape files?
>>> I imagine it would be possible to test for the presence of shapelib.so
>>> and provide a binary input mode:
>>>
>>>    splot <data> binary filetype=shapelib  with polygons
>>>
>>> I have never worked with this libary so I don't know if it is organized
>>> in such a way that this would be possible.
>>
>> I do it using the gdal software and some shell magic:
>>   # ogr2ogr -f "GMT" gaga.gmt gaga.shp
>>   # grep -v "^#" gaga.gmt | sed 's/>//' > gaga.txt
>
> That's also the approach taken by Bob Mesibov at
> https://www.datafix.com.au/BASHing/2018-10-31.html
>

Well, you don't need 2 blank lines for separation, a single blank line is enough.

More to the topic, I think the shape format is proprietary, complicated, spread over
multiple files, limited (number of fields, length of field names, file size,...),
so in one word: "legacy".
So I don't think it is an appropriate format for interfacing with gnuplot.
I made positive experiences with GeoJSON, it is a simple, very versatile format,
and I think most GIS programs can read and write this format, so why not
use some geojson routine and put it into gnuplot?

Cheers,
Manfred


> I'm hopeful that libshape can do the job without requiring the whole
> of gdal. Libshape was developed by Frank Warmerdam, the guy who
> initially developed gdal, and is used by gdal's OGR library.
>
> Allin Cottrell
>

Re: geographic plotting

[Allin C. <cot...@wf...>]

On Wed, 4 Mar 2020, Manfred Schwarb wrote:

> > This may be putting the cart before the horse, but...
> > Do you think it would be helpful to teach gnuplot to read shape files?
> > I imagine it would be possible to test for the presence of shapelib.so
> > and provide a binary input mode:
> >
> >    splot <data> binary filetype=shapelib  with polygons
> >
> > I have never worked with this libary so I don't know if it is organized
> > in such a way that this would be possible.
>
> I do it using the gdal software and some shell magic:
>   # ogr2ogr -f "GMT" gaga.gmt gaga.shp
>   # grep -v "^#" gaga.gmt | sed 's/>//' > gaga.txt

That's also the approach taken by Bob Mesibov at
https://www.datafix.com.au/BASHing/2018-10-31.html

I'm hopeful that libshape can do the job without requiring the whole
of gdal. Libshape was developed by Frank Warmerdam, the guy who
initially developed gdal, and is used by gdal's OGR library.

Allin Cottrell

Re: geographic plotting

[Manfred S. <man...@gm...>]

>
> This may be putting the cart before the horse, but...
> Do you think it would be helpful to teach gnuplot to read shape files?
> I imagine it would be possible to test for the presence of shapelib.so
> and provide a binary input mode:
>
>    splot <data> binary filetype=shapelib  with polygons
>
> I have never worked with this libary so I don't know if it is organized
> in such a way that this would be possible.


I do it using the gdal software and some shell magic:
  # ogr2ogr -f "GMT" gaga.gmt gaga.shp
  # grep -v "^#" gaga.gmt | sed 's/>//' > gaga.txt

HTH,
Manfred


>
> A quick look at the spec for the shapefile format makes it look horrible
> (mixed little-ending big-endian in the same file?! variable record types
> that are required not to vary?!),  but if it is hidden by a usable library ....
>
> 	Ethan
>
>
>
>
>
>
>
> _______________________________________________
> gnuplot-beta mailing list
> gnu...@li...
> Membership management via: https://lists.sourceforge.net/lists/listinfo/gnuplot-beta
>

Re: geographic plotting

[Allin C. <cot...@wf...>]

On Tue, 3 Mar 2020, Ethan A Merritt wrote:

> On Tuesday, 3 March 2020 17:05:40 PST Allin Cottrell wrote:
>> On Tue, 3 Mar 2020, Ethan A Merritt wrote:
>>
>>> On Tuesday, 3 March 2020 14:30:40 PST Allin Cottrell wrote:
>>>> This question has been posed before (a few years back) but I'm
>>>> hoping someone may be able to point me to the state-of-the-art,
>>>> if there is such.
>>>>
>>>> I'd like to be able to produce via gnuplot a map of (for example)
>>>> the USA showing the states colored by median income, or PM2.5
>>>> particulate emissions, or whatever.
>>>>
>>>> I understand that the starting point would be a suitable US states
>>>> shapefile (I know where I can get that), which would have to be
>>>> translated to a format (resembling world.dat, I suppose) that can be
>>>> understood by gnuplot. I have some notion of how that might be done.
>>>>
>>>> The next issue would be to get gnuplot to colorize specified areas
>>>> of the map according to the values of some additional data.
>>>> Unfortunately I have little notion of how to do that.
>>>
>>> The development version of gnuplot supports
>>>
>>> 	splot <data> with polygons
>>
>> Thanks Ethan, and also Dima in this thread. This sounds promising!
>> Right now I'm not sure when I'll find time to pursue this in depth,
>> but I'm definitely interested and will push it when I can.
>>
>> Allin Cottrell
>
> This may be putting the cart before the horse, but...
> Do you think it would be helpful to teach gnuplot to read shape files?

I'm sure it would be helpful...

> I imagine it would be possible to test for the presence of shapelib.so
> and provide a binary input mode:
>
>   splot <data> binary filetype=shapelib  with polygons
>
> I have never worked with this libary so I don't know if it is organized
> in such a way that this would be possible.

but I'm not yet clear on that point myself. I've taken a quick look 
at shapelib-1.5.0 (which configures and compiles without fuss on my 
machine) and seen that it can convert both .shp and .dbf files to 
what look like comprehensible text formats that "ought to be" 
usable (shp -> polygons that could go into CSV quite easily, and dbf 
-> descriptions of the elements of the shp file in their order of
appearance).

But when I said I "had some notion" of how shapelib could help, that 
wasn't an understatement! Perhaps better to say I "had an inkling" 
of how it could help.

I aim to follow up, but I'll be offline next week so I won't get 
back to this till the week after.

Allin

Re: geographic plotting

[Ethan A M. <me...@uw...>]

On Tuesday, 3 March 2020 17:05:40 PST Allin Cottrell wrote:
> On Tue, 3 Mar 2020, Ethan A Merritt wrote:
> 
> > On Tuesday, 3 March 2020 14:30:40 PST Allin Cottrell wrote:
> >> This question has been posed before (a few years back) but I'm
> >> hoping someone may be able to point me to the state-of-the-art,
> >> if there is such.
> >>
> >> I'd like to be able to produce via gnuplot a map of (for example)
> >> the USA showing the states colored by median income, or PM2.5
> >> particulate emissions, or whatever.
> >>
> >> I understand that the starting point would be a suitable US states
> >> shapefile (I know where I can get that), which would have to be
> >> translated to a format (resembling world.dat, I suppose) that can be
> >> understood by gnuplot. I have some notion of how that might be done.
> >>
> >> The next issue would be to get gnuplot to colorize specified areas
> >> of the map according to the values of some additional data.
> >> Unfortunately I have little notion of how to do that.
> >
> > The development version of gnuplot supports
> >
> > 	splot <data> with polygons
> 
> Thanks Ethan, and also Dima in this thread. This sounds promising!
> Right now I'm not sure when I'll find time to pursue this in depth, 
> but I'm definitely interested and will push it when I can.
> 
> Allin Cottrell

This may be putting the cart before the horse, but...
Do you think it would be helpful to teach gnuplot to read shape files?
I imagine it would be possible to test for the presence of shapelib.so
and provide a binary input mode:

   splot <data> binary filetype=shapelib  with polygons

I have never worked with this libary so I don't know if it is organized
in such a way that this would be possible.

A quick look at the spec for the shapefile format makes it look horrible
(mixed little-ending big-endian in the same file?! variable record types
that are required not to vary?!),  but if it is hidden by a usable library ....

	Ethan
	

Re: geographic plotting

[Allin C. <cot...@wf...>]

On Tue, 3 Mar 2020, Ethan A Merritt wrote:

> On Tuesday, 3 March 2020 14:30:40 PST Allin Cottrell wrote:
>> This question has been posed before (a few years back) but I'm
>> hoping someone may be able to point me to the state-of-the-art,
>> if there is such.
>>
>> I'd like to be able to produce via gnuplot a map of (for example)
>> the USA showing the states colored by median income, or PM2.5
>> particulate emissions, or whatever.
>>
>> I understand that the starting point would be a suitable US states
>> shapefile (I know where I can get that), which would have to be
>> translated to a format (resembling world.dat, I suppose) that can be
>> understood by gnuplot. I have some notion of how that might be done.
>>
>> The next issue would be to get gnuplot to colorize specified areas
>> of the map according to the values of some additional data.
>> Unfortunately I have little notion of how to do that.
>
> The development version of gnuplot supports
>
> 	splot <data> with polygons

Thanks Ethan, and also Dima in this thread. This sounds promising!
Right now I'm not sure when I'll find time to pursue this in depth, 
but I'm definitely interested and will push it when I can.

Allin Cottrell