#2134 Gnuplot eats my line

[Martin Saturka]

• connect to gnuplot via pipe (under Linux)
• plot a figure (by a command sent via the pipe)
• close the figure: by "q" on it, or by corner-"x" pressing
• send some (whatever) stuff to gnuplot via the pipe

The first line from the last stuff gets ignored/lost by gnuplot;
It was working well on older gnuplot versions: tested OK with 5.0, 52.0, 5.2.2;
It is wrong (with that eaten line) on: 5.2.6, current git 5.3;

[Ethan Merritt]

Not enough information.
What terminal is the plot directed to? Is it in "persist" mode?
What exactly is the sequence of commands?
From your description, gnuplot not exit before the plot window is closed. Why? Was there a "pause mouse close" command? If not, was there some other "pause" command?

[Martin Saturka]

It is for "qt" and "x11" terminals, "wx" terminal works well.
I've just looked, and it works well up to gnuplot 5.2.5, including it (that is for all the three terminals above).
It is since 5.2.6 where it is "hungry" at "qt" and "x11" terminals (though "wx" remains well).

I do not use "persist" mode, since I know it is evil. And no "pause" was set here either.
Gnuplot is alive, since the pipe is not closed on window closing (the pipe even does not know that anything was done to the plot window).

The commands can be anything, e.g.:
plot(sin(x))
or:
a = 1
plot(sin(a*x))

The first line (after window closing) gets always ignored. If it is a plot/splot, it is not plotted. If it is an assignment, the variable is left unknown (or with previous value if it was defined previously).

[Ethan Merritt]

I do not understand your description. If there is no "pause" statement then gnuplot does not wait for the plot to be drawn, let alone wait for it to be drawn, inspected, and closed. So it will have already executed the following commands before you even have a chance to close the plot. What is the "next" command in that situation? The normal sequence for this sort of application is to follot the plot command with "pause mouse" or "pause mouse close" so that the two end of the pipe stay in sync.

[Martin Saturka]

I see the confusion: I mean situation when I do not close the pipe, while you've supposed situations where a user writes to a pipe and closes it.
I'll show you a simple way how you can test it, using Tcl/Tk, preferably under Linux.
Start tcl, tclsh and type in, after the % cursors (the first line is longer, and only broken in this view):
set gh [open "|/usr/local/bin/gnuplot >> /tmp/gp001.txt 2>> /tmp/gp002.txt" "w"]
puts $gh "plot sin(x)"
flush $gh
close the plot window here
puts $gh "plot x**2"
flush $gh
puts $gh "plot x**3"
flush $gh
...
close $gh
Notice that the close command comes only after maaany plottings (here depicted by the ellipsis), once the overall session ends. And I mean an issues long before the close. Here it is that the plot x**2 command gets ignored for gnuplot 5.2.6 and 5.3.

This nonclosing way is popular in programmatic interfaces to gnuplot, since you do not need to wait for gnuplot startup on each and every plotting.

[Ethan Merritt]

I think we are still not on the same page. I am not talking about closing the pipe. I am talking about synchronizing operations performed by the user-facing program (the "write" end of the pipe) and operations performed by the gnuplot display (the "read" end of the pipe). In the sequence you list above there is no synchronization, so losing intermediate plot commands is expected. So far as gnuplot is concerned the sequence you show is equivalent to:

plot sin(x); plot x**2; plot x**3;

That may make sense if you are in multiplot mode or if the output terminal is one that creates a new page for each plot (e.g. postscript, pdf). Otherwise for an interactive output terminal (x11 wxt qt ...) the first two plots are not visible because they last only a fraction of a second.

For your scripted sequence to work as intended there need to be "pause" commands everywhere you have a "flush".

I also think you really do want "persist" in the gnuplot invocation. Otherwise your final plot will be lost.

[Martin Saturka]

I see this page of confusion. You assume that it is all post to gnuplot at once, while I meant (though my fault, had not written) that it is meant to be done by interactive programs that only post a plot command to gnuplot when user asks for that.

It works in a way that the interactive software sends some lines (and flushes them) when user wants a plot, then the pipe is kept open and without any input for some time (while user inspects the plot), and only if user asks for another plot the software sends another bunch of lines to gnuplot.

Please try to consider that it works, since it works: an amount of interactive software and users of that software use it. And it just works. Without any "pause", without any "-persist" in invocations, sigh. Thus another lines of code and comments:

After having the pipe open, user considers to plot sin(x), thus the interactive software sends to gnuplot:
puts $gh "plot sin(x)"
flush $gh
Now user inspects the plot, there is no pause nor whatever. This may be for e.g. 5 mins, or 5 years. After some time, the user decides to do something else, and (among others) user closes the plot window (it is not a necessary part of user workflow, but it comes to an issue with current gnuplot, thus doing it here).
Then after, say 15 mins, user decides to plot x^2, thus the software sends to gnuplot next lines:
puts $gh "plot x**2"
flush $gh
Now in gnuplot versions prior to 5.2.6, this lead to another plot (of x^2) being shown. While for 5.2.6 (and qt/x11 terminals) the first line received after closing window (even if long time after the window closing event) gets ignored, thus nothing gets plot here for 5.2.6 (with qt/x11).
If it were sent as:
puts $gh "\nplot x**2"
flush $gh
it would worked even here, but that added \n is silly.
If user decides to plot another graph (and without any manual closing of any window), it works again, like say sending:
puts $gh "plot x**3"
flush $gh
Here the wise user can do any inspect of the plot without any pause or whatever. And if sending (after e.g. 15 mins) next lines without any manual window closing, so that we do not encounter the 5.2.6 bug:
puts $gh "plot x**4"
flush $gh
the user gets another plot for playing (for any long time the user wants).

It is really simple. No magic here. And gnuplot 5.2.6 acquired a bug (it happens, that's life), and that bug should be repaired.

[Ethan Merritt]

I see.
Now I begin to guess what might be happening. Thank you for providing a complete description of the work flow.

I am wondering whether it really loses an entire line, or whether it loses only a single character. In the latter case I would expect an error message "invalid command ^" referring to the string lot x**2 that remains after the first character is dropped. But you may not see the error message if you are note catching stderr. Sending an extra '\n' rescues the subsequent command because swallowing the character '\n' does nothing and hurts nothing.

I ask because this is a failure mode we have seen before. To avoid it, the code itself does does
ungetc('\n',stdin)
to stuff an extra '\n' back into the input stream when the plot window is closed. But (here is the important point) it does this in the code that handles "pause mouse". In the past that has been sufficient because code similar to yours really does issue a "pause mouse" command. Your code apparently does not, and that could well explain the failure you see. At the moment I do not see why this would be any different in version 5.2.6 than in earlier versions but there are only a small number of changes so I will consider each as a possible source. Will you be able to test a patch if I find a possible fix?

[Martin Saturka]

OK, sometimes our previous experiences lead to differing expectations; that's life. The main thing is to find out where we diverged, to help us to converge again :-)
Regarding this case, I am reading stderr, and there is no such error in there. I.e. on the first sight it looks like a whole line lost (though I cannot be sure on that).

I can test a patch if you put it here, though I have to warn about what had happened to me in a different project: After I reported a bug there (I tend to report bugs to help projects), a developer started to do apparently random changes in code, repeatedly claiming that it just got fixed. Though it was still wrong, and after a bunch of wasted testings I left them.

BTW The Tcl code above can be used even if you are not familiar with Tcl/Tk. It sends both stdout and stderr into files, and you can write any command (not just the plot ones) into the puts lines. One of my intentions was actually to provide a simple testing/debugging tool via that Tcl use.

[Ethan Merritt]

Here you go. The attached patch extends the existing set of conditions under which a reset event (e.g. from closure of a plot window) is handled by issuing ungetc('\n',stdin). I think this will cover your tcl + pipe case. I am less sure that it won't introduce extraneous newlines in the output from other scripting implementations.
Patch 0001-... applies against current development tip.
Patch 0002-.... [should] apply against the stable 5.2 source.

[Martin Saturka]

Thanks, I'll try it probably at weekend.

[Martin Saturka]

BTW What is the commit that causes this bug, please? I mean that since it is a regression between 5.2.5 and 5.2.6, it should be possible to identify the cause, and possibly heal it where it starts instead of dealing with symptoms.

[Ethan Merritt]

As they [sort of] say: "One man's bugfix is another man's regression".

The difference between 5.2.5 and 5.2.6 relevant to your case is a change/fix to address the issue that programs or scripts that cared about the plot window being closed saw different event notifications depending on how the window was closed (e.g. ctrl-q vs alt-F4 vs icon on titlebar vs Xkill etc) and what window manager was in use. This caused variable behavior of "pause mouse close", "bind close", reinitialization of the terminal on demand, and maybe some other things I am forgetting.

The commits below tried to consolidate all of these into delivery of a GE_reset event back to the core program.
4acb0010 x11
e2b25bec wxt
ea9f4e28 qt

None of this explains exactly why arrival of a mousing event sometimes causes a character to be removed from the head of the input queue, and why it does not happen for terminal input. If you can figure that out, great! It may be due to something buried in the input library (libreadline or libedit).

[Martin Saturka]

I understand that fighting readline is difficult. Therefore I feel unease when dealing just with symptoms here.

When I disabled readline in compiling, gnuplot does not chew my lines. By that I got a thought: Is not readline meant for interactive (meant via terminal) use? While this issue is at pipe-wise use.
Should not detection of pipe-wise environment disable readline, instead of putting newlines in? It may be more work now, though it may prevent issues in the future.

I got another issue, see bug #2139, though with compile-wise disabled readline (by now at wx), thus it needs some another work just by that. A funny thing is that missing lines (i.e. this bug) are not an issue for wxt, while that readline-disabled bug (#2139) only is for wxt.

[Martin Saturka]

Have tried the patch and it works for me on both 5.2 and 5.3 branches.
Would prefer to see the real casue, though dealing with readline is a bigger task by itself.
Thanks for the patch.