gnuplot-beta

"pause mouse close" almost works, but not completely

[Dima K. <gn...@di...>]

Hi.

A debugging question. I have a trivial gnuplot script; the specific
script doesn't matter. For instance this in tst.gp:

  plot x
  pause mouse close

What I'd like to happen is to run

  load "tst.gp"

then an interactive plot pops up (x11 or qt or wxt terminals in my
case). When I close the interactive plot (by pressing 'q' for instance)
I'd like the 'pause mouse close' command to exit, and I'd like to get
back to the "gnuplot>" prompt. In my case this doesn't happen. With 'q'
the interactive plot window does go away, but gnuplot doesn't go back to
looking for input. I can press "enter", and I get the prompt back then.

Some minor debugging tells me that after 'q' I still have a gnuplot_x11
helper process. Both gnuplot_x11 and gnuplot are sitting in select()
waiting for input, and I need to kick the selects with another "enter"
to get a prompt back. My gnuplot comes vanilla from Debian. Is this what
people are observing? Any obvious causes? I'd like to ask before diving
into a debugging session.

dima

Re: "pause mouse close" almost works, but not completely

[sfeam <sf...@us...>]

On Saturday, 13 October 2018 17:01:03 Dima Kogan wrote:
> Hi.
> 
> A debugging question. I have a trivial gnuplot script; the specific
> script doesn't matter. For instance this in tst.gp:
> 
>   plot x
>   pause mouse close
> 
> What I'd like to happen is to run
> 
>   load "tst.gp"
> 
> then an interactive plot pops up (x11 or qt or wxt terminals in my
> case). When I close the interactive plot (by pressing 'q' for instance)
> I'd like the 'pause mouse close' command to exit, and I'd like to get
> back to the "gnuplot>" prompt. In my case this doesn't happen. With 'q'
> the interactive plot window does go away, but gnuplot doesn't go back to
> looking for input. I can press "enter", and I get the prompt back then.

I have been chasing a similar problem with the wxt terminal built in
"monothreaded" mode.  The code sequence after receiving a  "window closed"
event is confusing and may not be the same for different terminal types.  

wxt:  I think I found and fixed this yesterday for 5.3.
      If it works for everyone this should be back-ported to 5.2

x11:  In theory a terminal driver can see the window close event either
      as a GE_keypress event with the 1st parameter set to GP_Cancel
      or as a GE_reset event.  x11 handled the first case but failed to
      catch the second case.  Patch below.

qt:   May need a similar addition fix


	Ethan

%%%%%
diff --git a/term/x11.trm b/term/x11.trm
index 366cc0c..11556f1 100644
--- a/term/x11.trm
+++ b/term/x11.trm
@@ -954,6 +954,11 @@ AGAIN:
 			    return '\0';
 			}
 		    }
+		    if (ge.type == GE_reset) {
+			/* Error or window close */
+			paused_for_mouse = 0;
+			return '\0';
+		    }
 		}
 	    } else if (options == TERM_ONLY_CHECK_MOUSING) {
 		return '\0';
%%%%%


> 
> Some minor debugging tells me that after 'q' I still have a gnuplot_x11
> helper process. Both gnuplot_x11 and gnuplot are sitting in select()
> waiting for input, and I need to kick the selects with another "enter"
> to get a prompt back. My gnuplot comes vanilla from Debian. Is this what
> people are observing? Any obvious causes? I'd like to ask before diving
> into a debugging session.
>
> dima

Re: "pause mouse close" almost works, but not completely

[Dima K. <gn...@di...>]

sfeam <sf...@us...> writes:

> x11:  In theory a terminal driver can see the window close event either
>       as a GE_keypress event with the 1st parameter set to GP_Cancel
>       or as a GE_reset event.  x11 handled the first case but failed to
>       catch the second case.  Patch below.

Thank you very much. This patch fixes the bulk of the problem. There's
still a small vestige of the previous incorrect behavior that maybe is
unimportant, but I'll mention it anyway.

1. Run gnuplot
2. plot x
3. pause mouse close
4. 'q' in the popped-up terminal window. This gets us back to a
   "gnuplot>" prompt
5. If you start typing into THIS new prompt, you get a SECOND gnuplot>
   prompt to type into. Presumably the select() in the gnuplot process
   is still waiting for some data from the gnuplot_x11 helper that
   doesn't exist anymore

Thanks again.

Re: "pause mouse close" almost works, but not completely

[sfeam <sf...@us...>]

On Sunday, 14 October 2018 14:46:40 Dima Kogan wrote:
> sfeam <sf...@us...> writes:
> 
> > x11:  In theory a terminal driver can see the window close event either
> >       as a GE_keypress event with the 1st parameter set to GP_Cancel
> >       or as a GE_reset event.  x11 handled the first case but failed to
> >       catch the second case.  Patch below.
> 
> Thank you very much. This patch fixes the bulk of the problem. There's
> still a small vestige of the previous incorrect behavior that maybe is
> unimportant, but I'll mention it anyway.
> 
> 1. Run gnuplot
> 2. plot x
> 3. pause mouse close
> 4. 'q' in the popped-up terminal window. This gets us back to a
>    "gnuplot>" prompt
> 5. If you start typing into THIS new prompt, you get a SECOND gnuplot>
>    prompt to type into. Presumably the select() in the gnuplot process
>    is still waiting for some data from the gnuplot_x11 helper that
>    doesn't exist anymore

That is a consequence of a fix/hack/work-around that was added to mouse.c
a long time ago.  See lines starting with 2323 in mouse.c.

The original problem was that if you used the "pause mouse key" command
from a script, i.e. gnuplot was taking input from a pipe, then the first
character of the subsequent command was lost.  This generated an error
because usually losing the first character of a valid command leaves
an invalid command.  One fix is to add an extra blank line or at least
and extra blank character in the script file.  The cleaner fix seemed
to be to add back a harmless character '\n' in the mouse handling, which
could then be lost or not lost with no harm done.

It is possible that this recent set of fixes may have had the side effect
of also fixing the original lose-the-next-character bug.  So what you are
seeing is the added '\n' that was the work-around for a bug that may no
longer be present.

Commenting out the code block as mouse.c:2323 will get rid of the extra
prompt you are seeing.  That may or may not restore the problem of losing
character with piped input - it needs to be tested.

	Ethan

Re: "pause mouse close" almost works, but not completely

[Dima K. <gn...@di...>]

I just hit another bug in the same area; a more serious one this time.

When I run gnuplot normally with 'pause mouse close', it blocks
somewhere, and doesn't waste cycles while we're waiting. In my usage,
however, I rarely run gnuplot directly. I almost always use either
feedgnuplot (a shell frontend) or gnuplotlib (a plotting interface for
numpy in python). At least in the latter case, the 'pause mouse close'
works, but gnuplot spins instead of blocking, which wastes CPU
resources.

I'm attaching a tiny program in python that shows the issue. You should
change the GNUPLOT_SRC_DIR definition in that program to point to your
source tree. This test program

- spawns gnuplot (as a child of the python)
- asks it to plot something
- "pause mouse close"
- "print xxx"
- reads gnuplot output until it sees "xxx". This is the most reliable
  way I've found to let programs talk to gnuplot. If you know of a
  better synchronization method, please tell me

When I run this python program I see gnuplot repeatedly call usleep(10)
in X11_waitforinput() in x11.trm. I haven't looked enough at this code
path to understand what it's trying to accomplish. It feels like we
should never be doing this: all waiting should happen in the select() or
something like it. I also don't understand why I'm hitting this code
path in python, but not if I run gnuplot interactively ("gnuplot" and
then type in the commands) and not if I run it as a script (put the
"plot" and "pause mouse close" into tst.gp, and "gnuplot tst.gp").

Ideas?

Thanks

Re: "pause mouse close" almost works, but not completely

[sfeam <sf...@us...>]

On Sunday, 14 October 2018 18:25:31 Dima Kogan wrote:
> I just hit another bug in the same area; a more serious one this time.
> 
> When I run gnuplot normally with 'pause mouse close', it blocks
> somewhere, and doesn't waste cycles while we're waiting. In my usage,
> however, I rarely run gnuplot directly. I almost always use either
> feedgnuplot (a shell frontend) or gnuplotlib (a plotting interface for
> numpy in python). At least in the latter case, the 'pause mouse close'
> works, but gnuplot spins instead of blocking, which wastes CPU
> resources.
> 
> I'm attaching a tiny program in python that shows the issue. You should
> change the GNUPLOT_SRC_DIR definition in that program to point to your
> source tree. This test program
> 
> - spawns gnuplot (as a child of the python)
> - asks it to plot something
> - "pause mouse close"
> - "print xxx"
> - reads gnuplot output until it sees "xxx". This is the most reliable
>   way I've found to let programs talk to gnuplot. If you know of a
>   better synchronization method, please tell me
> 
> When I run this python program I see gnuplot repeatedly call usleep(10)
> in X11_waitforinput() in x11.trm. I haven't looked enough at this code
> path to understand what it's trying to accomplish. It feels like we
> should never be doing this: all waiting should happen in the select() or
> something like it. 

It is on purpose, wisely chosen or not.

> I also don't understand why I'm hitting this code
> path in python, but not if I run gnuplot interactively ("gnuplot" and
> then type in the commands) and not if I run it as a script (put the
> "plot" and "pause mouse close" into tst.gp, and "gnuplot tst.gp").

This is explained in the comments at x11.trm:894
        /* When taking input from the console, we are willing to wait here  */
        /* until the next character is typed. But if input is from a script */
        /* we just want to check for hotkeys or mouse input and then leave  */
        /*  again without waiting on stdin.                                 */

When the input is from the console we can select on stdin and not spin.
But if input is from a pipe this will always return immediately unless the
other end of the pipe does something clever to interlock operations.
That would make "pause mouse"  useless.  So if the input is from a pipe then
it spins at line 991  as you found.  If you find the cost of spinning too
severe you could bump the usleep to a higher value:

%%%%%
diff --git a/term/x11.trm b/term/x11.trm
index 11556f1..364d4b4 100644
--- a/term/x11.trm
+++ b/term/x11.trm
@@ -988,7 +988,7 @@ AGAIN:
     /* Same sort of thing if we are specifically waiting for mouse input. */
     if (paused_for_mouse) {
 #ifdef HAVE_USLEEP
-   usleep(10);
+ usleep(10000);
 #endif
        goto AGAIN;
     }
%%%%%

As shown by "top" on my desktop machine that reduces the cost of spinning
from about 10% cpu to <1% cpu.
The downside is that this adds a 10msec delay in response to keystrokes.
That may well be a good trade-off.  Or we could make the test more complicated
and use the shorter sleep for actual keystrokes (pause mouse key) and the
longer sleep when waiting for a close event (pause mouse close).

Let me know if that works for you.

Or maybe the whole loop can be written more cleverly so that the select
isn't useless when a pipe is involved.

	Ethan

Re: "pause mouse close" almost works, but not completely

[Dima K. <gn...@di...>]

sfeam <sf...@us...> writes:

> On Sunday, 14 October 2018 18:25:31 Dima Kogan wrote:
>> I just hit another bug in the same area; a more serious one this time.
>>
>> When I run gnuplot normally with 'pause mouse close', it blocks
>> somewhere, and doesn't waste cycles while we're waiting. In my usage,
>> however, I rarely run gnuplot directly. I almost always use either
>> feedgnuplot (a shell frontend) or gnuplotlib (a plotting interface for
>> numpy in python). At least in the latter case, the 'pause mouse close'
>> works, but gnuplot spins instead of blocking, which wastes CPU
>> resources.
>>
>> I'm attaching a tiny program in python that shows the issue. You should
>> change the GNUPLOT_SRC_DIR definition in that program to point to your
>> source tree. This test program
>>
>> - spawns gnuplot (as a child of the python)
>> - asks it to plot something
>> - "pause mouse close"
>> - "print xxx"
>> - reads gnuplot output until it sees "xxx". This is the most reliable
>>   way I've found to let programs talk to gnuplot. If you know of a
>>   better synchronization method, please tell me
>>
>> When I run this python program I see gnuplot repeatedly call usleep(10)
>> in X11_waitforinput() in x11.trm. I haven't looked enough at this code
>> path to understand what it's trying to accomplish. It feels like we
>> should never be doing this: all waiting should happen in the select() or
>> something like it.
>
> It is on purpose, wisely chosen or not.
>
>> I also don't understand why I'm hitting this code
>> path in python, but not if I run gnuplot interactively ("gnuplot" and
>> then type in the commands) and not if I run it as a script (put the
>> "plot" and "pause mouse close" into tst.gp, and "gnuplot tst.gp").
>
> This is explained in the comments at x11.trm:894
>         /* When taking input from the console, we are willing to wait here  */
>         /* until the next character is typed. But if input is from a script */
>         /* we just want to check for hotkeys or mouse input and then leave  */
>         /*  again without waiting on stdin.                                 */
>
> When the input is from the console we can select on stdin and not spin.
> But if input is from a pipe this will always return immediately unless the
> other end of the pipe does something clever to interlock operations.

OK, I don't fully grok this, but it feels wrong. If I strace the gnuplot
process when running inside python, the spin looks like this:

   2482  select(5, [0 4], NULL, NULL, NULL) = 1 (in [0])
   2482  nanosleep({tv_sec=0, tv_nsec=10000}, NULL) = 0
   2482  select(5, [0 4], NULL, NULL, NULL) = 1 (in [0])
   2482  nanosleep({tv_sec=0, tv_nsec=10000}, NULL) = 0
   2482  select(5, [0 4], NULL, NULL, NULL) = 1 (in [0])
   2482  nanosleep({tv_sec=0, tv_nsec=10000}, NULL) = 0
   ....

I.e.

- we select(stdin, ipc_back_fd)
- select() returns immediately because there's data available on stdin
- we DON'T read this data
- we sleep a bit
- we select() again
- Since we never read the data select() said was available the first
  time, select() returns immediately again
- And we spin

Surely this can't be the intended behavior?

Re: "pause mouse close" almost works, but not completely

[Dima K. <gn...@di...>]

sfeam <sf...@us...> writes:

> If you can draw up another scheme using only select I'd be happy to
> revisit the code. Maybe the code in the qt terminal could be used as a
> guide.

OK, that would require actually understanding what waitforinput() is
supposed to do. Is there a description somewhere? qt terminal? Should I
just study the code?

Re: "pause mouse close" almost works, but not completely

[sfeam <sf...@us...>]

On Sunday, 14 October 2018 22:41:04 Dima Kogan wrote:
> sfeam <sf...@us...> writes:
> 
> > On Sunday, 14 October 2018 18:25:31 Dima Kogan wrote:
> >> I just hit another bug in the same area; a more serious one this time.
> >>
> >> When I run gnuplot normally with 'pause mouse close', it blocks
> >> somewhere, and doesn't waste cycles while we're waiting. In my usage,
> >> however, I rarely run gnuplot directly. I almost always use either
> >> feedgnuplot (a shell frontend) or gnuplotlib (a plotting interface for
> >> numpy in python). At least in the latter case, the 'pause mouse close'
> >> works, but gnuplot spins instead of blocking, which wastes CPU
> >> resources.
> >>
> >> I'm attaching a tiny program in python that shows the issue. You should
> >> change the GNUPLOT_SRC_DIR definition in that program to point to your
> >> source tree. This test program
> >>
> >> - spawns gnuplot (as a child of the python)
> >> - asks it to plot something
> >> - "pause mouse close"
> >> - "print xxx"
> >> - reads gnuplot output until it sees "xxx". This is the most reliable
> >>   way I've found to let programs talk to gnuplot. If you know of a
> >>   better synchronization method, please tell me
> >>
> >> When I run this python program I see gnuplot repeatedly call usleep(10)
> >> in X11_waitforinput() in x11.trm. I haven't looked enough at this code
> >> path to understand what it's trying to accomplish. It feels like we
> >> should never be doing this: all waiting should happen in the select() or
> >> something like it.
> >
> > It is on purpose, wisely chosen or not.
> >
> >> I also don't understand why I'm hitting this code
> >> path in python, but not if I run gnuplot interactively ("gnuplot" and
> >> then type in the commands) and not if I run it as a script (put the
> >> "plot" and "pause mouse close" into tst.gp, and "gnuplot tst.gp").
> >
> > This is explained in the comments at x11.trm:894
> >         /* When taking input from the console, we are willing to wait here  */
> >         /* until the next character is typed. But if input is from a script */
> >         /* we just want to check for hotkeys or mouse input and then leave  */
> >         /*  again without waiting on stdin.                                 */
> >
> > When the input is from the console we can select on stdin and not spin.
> > But if input is from a pipe this will always return immediately unless the
> > other end of the pipe does something clever to interlock operations.
> 
> OK, I don't fully grok this, but it feels wrong. If I strace the gnuplot
> process when running inside python, the spin looks like this:
> 
>    2482  select(5, [0 4], NULL, NULL, NULL) = 1 (in [0])
>    2482  nanosleep({tv_sec=0, tv_nsec=10000}, NULL) = 0
>    2482  select(5, [0 4], NULL, NULL, NULL) = 1 (in [0])
>    2482  nanosleep({tv_sec=0, tv_nsec=10000}, NULL) = 0
>    2482  select(5, [0 4], NULL, NULL, NULL) = 1 (in [0])
>    2482  nanosleep({tv_sec=0, tv_nsec=10000}, NULL) = 0
>    ....
> 
> I.e.
> 
> - we select(stdin, ipc_back_fd)
> - select() returns immediately because there's data available on stdin
> - we DON'T read this data
> - we sleep a bit
> - we select() again
> - Since we never read the data select() said was available the first
>   time, select() returns immediately again
> - And we spin
> 
> Surely this can't be the intended behavior?

It is the intended behaviour.
It may not be ideal but it works.
If you can draw up another scheme using only select I'd be
happy to revisit the code.  Maybe the code in the qt
terminal could be used as a guide.

	Ethan

Re: "pause mouse close" almost works, but not completely

[sfeam <sf...@us...>]

> > I.e.
> > 
> > - we select(stdin, ipc_back_fd)
> > - select() returns immediately because there's data available on stdin
> > - we DON'T read this data
> > - we sleep a bit
> > - we select() again
> > - Since we never read the data select() said was available the first
> >   time, select() returns immediately again
> > - And we spin
> > 
> > Surely this can't be the intended behavior?
> 
> It is the intended behaviour.
> It may not be ideal but it works.
> If you can draw up another scheme using only select I'd be
> happy to revisit the code.  Maybe the code in the qt
> terminal could be used as a guide.
> 
> 	Ethan

Or maybe it is as simple as 
            FD_ZERO(&fds);
			if (!paused_for_mouse)
            	FD_SET(fd, &fds);
            FD_SET(ipc_back_fd, &fds);
			select()

I'll look at it some more tomorrow

	Ethan