#2279 Incorrect reading of characters from pipe

[MBaz]

I am the author of Gaston.jl (https://github.com/mbaz/Gaston.jl), a gnuplot front-end for the Julia programming language. Gaston uses the standard streams stdin, stdout and stderr to communicate with gnuplot.

An issue was recently reported by Windows users (https://github.com/mbaz/Gaston.jl/issues/136). When plotting, gnuplot sometimes produces errors such as:

gnuplot> eset session
         ^
         line 0: invalid command

which is interesting, since Gaston.jl sends the command reset session. These errors are intermittent. The "invalid command" always looks like one of the commands that Gaston.jl sends to gnuplot (via stdin), but with a few missing or corrupted characters.

This issue seems to occur only in Windows 10 (at least a couple of different versions, which are listed in the issue referenced above). The issue never happens in Linux.

In order to eliminate the possibility of a bug in the Julia language, I reproduced the problem using Python. The steps to reproduce are as follows.

First, create a file with name, say, data.dat and contents:

1 1
2 2
3 3

Second, run these commands in Python 3:

from subprocess import Popen, PIPE
gnuplot = 'gnuplot'
p = Popen([gnuplot, '-persist'], stdin=PIPE)
cmd = b"""reset session
          set term qt 1
          plot 'data.dat'
          reset session
          set output
          set print '-'
          print 'Done'
       """
p.stdin.write(cmd); p.stdin.flush()

You may need to run the last line a few times. The problem seems to happen most often when the plot window is closed using q, before sending the commands to gnuplot again.

I can reproduce this issue with the Windows binaries provided at

http://tmacchant3.starfree.jp/gnuplot/Eng/winbin/

which don't include the latest rc versions. Unfortunately, I'm not set up to build gnuplot on Windows. I'll try to provide any information and run any tests you need.

Finally, I hope this report is not too late to be fixed before 5.4.0 is released!

[Ethan Merritt]

"seems to happen most often when the plot window is closed using q". This makes me wonder if issue 2098 "pause mouse key in qt terminal returns two events on "q" is involved. https://sourceforge.net/p/gnuplot/bugs/2098/

If the problem is specific to Windows, I cannot be of much help. But I may be able to fix the double-event problem in issue #2098, and if it happens to also fix this problem on Windows that would be a bonus.

[MBaz]

Ethan,

Issue 2098 is quite interesting and might be related. I think that there is a link between keyboard events and reading from stdin, where those two processes should be completely independent.

In what follows, keep in mind that working with streams in Windows is extremely slow. In the command sequence I describe above, the plot usually takes less than a couple of seconds to appear, but the full sequence may take as long as six or seven seconds to complete (where, in Linux, it's essentially instantaneous).

If I set the terminal with set term qt ctrlq and run my sequence of commands, I see the plot appear. Then, I hit q. The plot doesn't close, but I immediately see a warning from gnuplot complaining about a syntax error or other problem. For example, I can get:

gnuplot> Done''
                ^
                line 0: invalid command

but I've also gotten stuff like:

line 0: warning: Cannot find or open file "C:\qt 1 ctrlq"

which seems to show that gnuplot gets completely confused about the data in stdin.

This is not 100% reproducible, and different keys seems to have different effects. Since each try takes as much as ten seconds, it's not easy to find patterns. But I think that there is definitely something wrong in the way that stdin is handled.

BTW, eveything in this post was tested with gnuplot 5.2 patchlevel 7, on Windows 10.

[Ethan Merritt]

Reading from stdin cannot be "completely independent" from keyboard events because operations such as pause mouse must listen to both simultaneously. Each terminal that supports mousing must provide a routine term->waitforinput that does this multiplexing. For the qt terminal this is qt_term.cpp routine qt_waitforinput().

Under linux the multiplexing is implemented by select() acting jointly on stdin and on the fd used for input from the plot window.

Under Windows, however, a completely different mechanism is used (qt_term lines 1111-1262. I can see in the code that it is complicated but I am not at all familiar with the Windows system calls or operations. Equivalent Windows-specific multiplexing code is used also in the wxt terminal routine wxt_gui.cpp: wxt_waitforinput().

Hint for anyone working on this problem:

As I recall, a similar problem used to affect the linux wxt terminal. This was ameliorated by a makeshift fix in wxt_exec_event() that used ungetc to stuff a character back into the input stream buffer. This makeshift is commented out in the current linux code, I think because the select() code mentioned above was reworked to make it no longer needed. Perhaps the same makeshift fix would ameliorate the current Windows code? Here is the relevant code snipper from version 5.2.8

#if defined(_WIN32)
        wxt_process_one_event(&event);
        return true;
#elif defined(WXT_MONOTHREADED)
        if (wxt_process_one_event(&event))
            /* FIXME: No longer be needed? */
            /* ungetc('\n',stdin) */
            ;
        return true;
#else

[MBaz]

I see. I just sent q to gnuplot from Julia (via stdin) and saw the plot close. This surprises me, because typing q in gnuplot's console doesn't close the window -- I just get a q on the command line. It's almost like the q gets sent to the plot window!

Speaking with vey little knowledge of gnuplot's design decisions, I guess I would have expected the plot window to have its own stdin, and a separate communications channel to gnuplot.

I think the problem is exacerbated on Windows because it is so slow that one may use the keyboard before the plot is complete, while gnuplot is still receiving commands from stdin.

(I'm not convinced this is the only issue, since sometimes the problem occurs without any keyboard interaction. Maybe there is some plot window to/from gnuplot communication happening and interfering with stdin?)

Would it be a good idea to provide an independent IPC mechanism between gnuplot and other programs? The stdin/stdout method is so fragile, and in many ways inadequate. For example, it is difficult to tell from outside when a plot command has finished. A TCP-based mechanism with a simple protocol would be so nice to have.

[Ethan Merritt]

I do not fully understand your description (maybe because you are describing operation on Windows?). What is "gnuplot's console" that is distinct from stdin? [edit: see separate reply]

As to general design - every graphics toolkit (qt wxWidgets x11 aqua ...) seems to have its own idea about how to deal wtih communication, and the linux, Windows, and Mac libraries for the same toolkit are often different. So it's hard to generalize. Add to the mix a separation between the user's keyboard input and whatever channel it is that is feeding a command stream to gnuplot and it gets even more complicated. So let's confine the discussion to Qt since that's what you are reporting.

For a simple gnuplot session on linux the focus policy is crucial. A keypress event when the focus is in a qt plot window is normally seen by Qt, handled through a Qt event stack, and passed to gnuplot_qt as a Qt event; then gnuplot_qt decides whether or not to send it via a separate ipc channel to the main gnuplot process. Such keypress events are not [supposed to be] seen by the command parser. But if the focus is in the gnuplot console window a keypress is seen on stdin by the main gnuplot process via libreadline .

Your question implies some third pathway. Help me better understand how julia is driving gnuplot. Does it open a pseudo-tty pair (stdin/stdout) and talk to gnuplot that way? (I was under the impression that Windows in general does not work that way but I could be wrong). If so, who is listening to the user keyboard and how does gnuplot know that a key has been pressed? Does it go keypress→julia→gnuplot?

Independent IPC: I use gnuplot extensively via piped stdin, either from scripts or programs run at the command line or as a back-end to web-driven computation. In my experience this is extremely robust. I agree with you that the reverse direction, querying gnuplot state by reading its stdout, is less usable. It is possible to open a separate output channel using gnuplot's set print command to send checkpoints or other information explicitly in the course of a scripted run. I have done this occasionally. You suggest TCP. I know this acronym only in the context of network protocols; can you clarify or expand on this idea?

[Ethan Merritt]

I do not fully understand your description (maybe because you are describing operation on Windows?). What is "gnuplot's console" that is distinct from stdin?

Wait a minute, maybe I do understand. Is it that julia starts up a full gnuplot session, with separate controlling terminal etc, rather than piping to a captive gnuplot process that is not otherwise visible to the user? This is where my unfamiliarity with running under Windows limits my insight. I don't know whether this is necessary or not.

[MBaz]

Ethan, first of all, thanks for the detailed explanation and for pointing out where I am being unclear. I will try to clarify. Keep in mind I'm also not a Windows expert or even much of a user; I mainly use linux. I'm just trying to debug an issue that Windows users found in my code.

For context, I develop Gaston.jl, a plotting package for the Julia language, that uses gnuplot to plot. Gaston's documentation is here, and the source code is here.

Also important: Julia uses libuv for I/O. This library abstracts the differences between windows, mac and unix, so that they all look like unix. So, exactly the same code and the same techniques work in all three OSs.

I start gnuplot by running it from Julia, and connecting to its stdin, stdout and stderr streams (see here). After that, I can send commands to gnuplot by writing to its stdin, and I can read from stdout and stderr. [By the way, I implemented a "protocol" just as you suggested: I have gnuplot print to its stdout when it is done plotting (see here for an example).]

The Julia language has a REPL, and I implement plotting functions that get "translated" and sent to gnuplot via its stdin. For example, I can type this in the Julia REPL:

julia> t = -5:0.01:5
julia> plot(t, cos)

and get a plot produced by gnuplot. So, there is no keyboard involved between Julia and gnuplot.

I understand your explanation about how things work with window focus and the different processes involved. This makes sense when I start gnuplot from a terminal and then open some plot windows.

When it gets very confusing to me is the case where I have a process sending commands to gnuplot via stdin. In that case, gnuplot never has focus (it has no window!). I had assumed things worked like this:

my process ---> gnuplot <----> plot window <--- keyboard

where, crucially, the plot window has its own stdin and its own communication channel to gnuplot. But, from what I have seen, things seem to work like this, whenever a plot exists:

my process ---> plot window <----> gnuplot
keyboard --------^

The problem with this approach, and the source of the bug my users reported, is that the commands I'm sending get intermingled with the keyboard and with any other data that the plot window is also sending to gnuplot. (This is especially noticeable in Windows, where the communication between my process and gnuplot is so slow).

(Evidence that things work this way is that, when my process sends a q, the plot window closes; gnuplot itself would exit if it ever saw that q).

Please let me know if my explanation makes sense. I'd also be interested to know if you think this is a bug, or at least something that should be improved in gnuplot.

Regarding TCP: the idea is that both my process and gnuplot would open a socket on a common port number, and exchange data by reading and writing to the socket. It may be easier to implement a simple communication protocol over TCP than over stdin/stdout. Also, you would avoid the problem of stdin/stdout being extremely slow in windows.

[MBaz]

More tests: I can reproduce the problem with the wxt terminal, but NOT with the windows terminal.

[Ethan Merritt]

I am trying to get up to speed with julia and Gaston. So far

• I installed julia on linux Magiea 7.1 from tarball julia-1.4.2-linux-x86_64.tar.gz
• I cloned the git source for Gaston, but I think this is not actually used because following the instructions in README.md I told julia ] add Gaston and it downloaded a compiled copy from somewhere.
• I told julia test Gaston.
  • It immediately printed a Manifest of about 25 modules ending with " [4ec0a83e] Unicode"
  • It then pinned CPU usage silently for about 2-3 minutes.
  • Finally it printed out

Test Summary:           | Pass  Total
Figure and set commands |   15     15
Test Summary: | Pass  Broken  Total
2-D plots     |   43       3     46
Test Summary: | Pass  Total
Histograms    |    2      2
Test Summary: | Pass  Total
Images        |    3      3
Test Summary: | Pass  Total
3-D plots     |   12     12
Test Summary: | Pass  Total
Multiplot     |    1      1
Test Summary: | Pass  Total
Saving plots  |    6      6
Test Summary:          | Pass  Total
Tests that should fail |   14     14
    Testing Gaston tests passed 

• Should I worry about 3 Broken tests? Is 2-3 minutes elapsed time normal?
• I ran the test command again. This run was a bit faster (30 seconds); exactly the same output.
• I tried several of the 2D demos from 2dplots.md; they seemed to work
• I tried to replicate the 3D x^2 * besselj0(y) surface plot from the top of the Gaston.jl docs page, and compared it side-by-side to plotting splot besj0(y)*x*x directly in gnuplot. I could match the surface itself, but FWIW the lighting model does not match.

Now I'm stuck. You mention sending a 'q' to gnuplot from the julia command line. How can I do that? Is there a syntax for sending arbitratry gnuplot commands from the julia command line? How can I tell julia to use a local copy of Gaston that I may have edited rather than the pre-compiled copy in its inventory?

Clarifications on communication paths:
I can think of no scenario where anything you type into the julia session should go to gnuplot_qt, the separate process that manages the plot window, rather than to the main gnuplot process. gnuplot_qt does have its own stdin/stdout/stderr, but that is not how it sees keyboard or mouse events. Those come in via the Qt runtime event handling stack. So your second diagram makes no sense to me. What exactly did you observe that led to this model?

Finally - is this tracker an OK medium for continuing this discussion or should I contact you by Email?

[MBaz]

Ethan, thank you so much for the time and energy spent on this. I hope it turns out to be worthwhile in the end.

You seem to have installed Gaston correctly. I'm assuming we're on Linux. Try these two commands:

julia> plot(1:10)
julia> Gaston.gnuplot_send("q")

(The gnuplot_send command is not exposed directly to users; that is why you have to include Gaston.. Also, this function appends a newline to the string sent to gnuplot.)

What I see is that the plot window disappears.

Now, let's try on gnuplot. If I run

gnuplot> plot sin(x)

and then return the focus to the gnuplot terminal, typing q <Enter> causes gnuplot to exit.

This tells me (but I could be wrong) that the "q" sent by Gaston is going to the plot window, not to gnuplot. I don't understand how that can happen, because I never opened a communications channel to the plot window, unless gnuplot itself is redirecting its stdin to the window.

Since the discussion is turning out to be pretty long, I'd prefer to continue by email, if that is OK with you. My email username is miguelbaz, and my email server is protonmail.ch.

[Ethan Merritt]

Can I ask you to try one more thing? You said that you tested using Windows binaries from http://tmacchant3.starfree.jp/gnuplot/Eng/winbin/ which seem to have disappeared and anyway were out of date. Can you test again using the Windows binary for 5.4.rc1?
https://sourceforge.net/projects/gnuplot/files/gnuplot/testing/gp54rc1-win64-mingw.7z/download

I suspect it will act the same, but who knows, maybe some crucial thing has changed.

[Bastian Märkisch]

Unfortunately, I can still reproduce this with 5.4.2.