#2116 Gnuplot 5.2.6 issue with pipes on MS-Windows

[Eli Zaretskii]

I see strange problems with using gnuplot 5.2.6 on MS-Windows when gnuplot's stdin and stdout are connected to pipes. The initial screen does not appear on the reading end of the pipe connected to stdout, perhaps because of buffering? (Is there a way to tell gnuplot to use line buffering in this case?)

Previous versions of gnuplot came with pgnuplot.exe, which could be used with pipes. Now we are supposed to use gnuplot.exe, but it seems it doesn't behave the same as the old pgnuplot.

This was observed while using gnuplot as the plotting back-end of Calc in Emacs. The only way I could get a graph plotted was to use "set terminal qt", otherwise the gnuplot command to load a file with definitions of a graph does not cause gnuplot to show the graphical window with the graph.

Let me know what additional details I can provide for this issue to be looked into.

[Hans-Bernhard Broeker]

If anything, it's the old behaviour that was buggy. gnuplot running non-interactively has never shown a prompt or the start-up blurb. Only the Windows GUI version did that, because Windows GUI has no concept of stdin, stdout, redirection or anything of the sort. pgnuplot was only ever a stop-gap measure to enable some level of pipe-ability.

Nor is any of this particularly new. pgnuplot has been removed from binary the packages since version 5.0 already.

The difference in behaviour regarding the need to choose a terminal probably happened during installation of the new binary package. 5.2.6 supports a choice of interactive terminal drivers via the environment variable GNUTERM. You let setup create that, or set it later.

[Eli Zaretskii]

The problem with the behavior you consider correct is that it effectively makes gnuplot on Unix behave differently when invoked as a subprocess, because on Unix it communicates with its parent process via ptys, which I guess you consider "interactive" invocation.

The end result of this is the hoops through which Emacs needs to jump to produce graphs from Calc on MS-Windows. Please take a look at calc-graph.el. It also means that Emacs cannot show the user the gnuplot responses to the commands, as it does on Unix.

Would it be possible to add a command-line switch to gnuplot that would generate the start-up blurb and echo the commands as it does interactively, even when it communicates via a pipe? Then Emacs could use that switch when invoking gnuplot.

I don't think I understand what you wrote about the terminal. I was trying to describe an incorrect behavior, whereby, unless the terminal is set to "qt", there's no graphic window with the graph visible after sending the "load FILE" command to gnuplot, where FILE includes all the commands to plot a graph. IOW, this is not a problem with setting the default terminal, this is a problem with producing a graph from non-interactive invocation. Strangely, if I invoke gnuplot via a pipe from cmd.exe, the problem doesn't happen What could be the reason for that?

[Eli Zaretskii]

I have additional details regarding the abnormal behavior of gnuplot when invoked with terminal set to "windows" or "wxt". This seems to be related to the fact that Emacs on MS-Windows by default invokes console programs with the wShowWindow member of the STARTUPINFO structure set to SW_HIDE, to avoid popping up the console window of the sub-program. If I disable this feature of Emacs, then gnuplot runs as expected with any terminal type, but it also pops up the console window (in addition to the GUI window where it shows the graph). If terminal is set to "qt", the console window closes right away, but with other terminal types it stays until gnuplot exits.

wgnuplot_pipes, being a console program, behaves the same as gnuplot.

So I'm guessing that somehow gnuplot needs its console window to be visible in order for it to show the GUI window. The "qt" terminal behaves differently, because (as I recall reading somewhere) Qt has an emulation of 'fork', and gnuplot uses it.

Perhaps gnuplot could be changed to still show its GUI window even if the console window is hidden?

[Tatsuro MATSUOKA]

Mechanism graph draw procufures of qt terminal is different from those of windows and wxt terminals. On qt terminal, graph is drawn by driver program gnuplot_qt.exe and thus graph appears in your first case. On windows and wxt terminals, graph is drawn by gnuplot itself. Thus the different behaviors between qt and other terminals (wxt and windows) that you met are expected behaivors. The issue you met is not gnuplot specific but all CLI programs that oupuput GUI staff. The issue is well known and wierd on windows.

[Eli Zaretskii]

The issue is well known and wierd on windows.

By "well known" and "expected" I guess you mean known to gnuplot developers and expected by them? Because I find this information to be very hard to come by, it's neither in the gnuplot manual nor can be found by searching the Internet for relevant phrases. The Internet search, for example, comes full of outdated information regarding pgnuplot, that is no longer part of the distribution. So perhaps one way of improving the situation is just to have it described in the manual? In particular, the description of the terminal types, which otherwise provides a wealth of information about each terminal, is completely silent on the details you mention above, and so is the README-Windows.txt file. (And while at that, maybe describe wgnuplot_pipes.exe in the Windows-specific README.)

As for the adverse effects of running gnuplot with its window hidden by SW_HIDE, this is also not immediately clear. For one counter-example, Emacs on Windows is also created as a console program which shows a GUI window, and yet running Emacs with SW_HIDE does not cause its GUI window not to appear. Also, the pgnuplot program didn't have this problem. So if gnuplot cannot be changed to behave like pgnuplot did in this case, at the very least I suggest to document this restriction. Running gnuplot as a sub-process that displays graphs is an important use case, so programmers working on MS-Windows should understand these aspects to be able to invoke gnuplot correctly.

[Bastian Märkisch]

And while at that, maybe describe wgnuplot_pipes.exe in the Windows-specific README.

In version 5.3 we included the following text:

* wgnuplot_pipes.exe:  Variant with full pipe functionality at the expense of
  an additional console window.

That should have made it to the 5.2 README, too. Thanks for pointing this out.

[Bastian Märkisch]

On second thought: we likely will remove wgnuplot_pipes from 5.4. Its functionality is (mostly) available with wgnuplot nowadays already: it opens a console if you use a pipe command. Remaining cases might be to see the output of console commands like !dir.

[Tatsuro MATSUOKA]

I will consider documentation improvement. But my Emglish is very poor so that I need helps from other contributers. I think that README-Windows.txt is proper place to describe the pipe issue that you have met. I will post discussion on develomment mailing list
(gnuplot-beta@lists.sourceforge.net)

The program "pgnuplot.exe" runs without any dll files and search the name wgnuplot.exe. You can copy it from gnuplot-4.6.7 windows binary package to 5.2.6 bin folder.
Please try and report result.

[Eli Zaretskii]

Thanks in advance for improving the docs regarding these tricky issues.

I copied pgnuplot.exe from Gnuplot 4.6.0, and it worked just fine with Gnuplot 5.2.6. Thanks.

[Bastian Märkisch]

Being likely responsible for any Windows specific bugs in recent versions, I have to admit that I am not aware that this behaviour is "well-known". Tatsuro, it would be great if you could provide references / links .

To me it seems to boils down to incorrect/insufficient parameters to a ShowWindow() call. This should hopefully be easy enough to fix. I remenber having changed these for both wxt and windows terminals in the past. I'll investigate.

[Bastian Märkisch]

Looks like this is related to [#1986]. Might be the very same issue really.

[Eli Zaretskii]

I think you are right, it sounds like the same issue.

[Bastian Märkisch]

Got it for the windows terminal. We overlooked the case with the freshly created window in a "hidden" state. We now check for that and do a second ShowWindow: [f60324]

The case of the wxt terminal is more complicated: -persist seems to be broken there.

[Bastian Märkisch]

Also fixed for wxt, but --persist is still broken.

[Bastian Märkisch]

Turned out to be a red hering. Finally fixed in branch master. Will go to 5-2-stable after some testing.

[Tatsuro MATSUOKA]

Well known is my misunderstand.
I met the similar issue when I tried to connect gnuplot from SciTE editor many years ago.
In the SciTE editor documetation,
https://www.scintilla.org/SciTEDoc.html
See Command subsystem => console
Description for windows is differ from that of GTK+(linux) and macOS.
gnuplot does not show graph on console subsystem for windows but shows linux.

I googled the origin and found the decription. I googled again now I cannot find any pages.

[Tatsuro MATSUOKA]

Eli Zaretskii
I have uploaded gnuplot binaries of development version with Bastians's corrections mentioned the above.
http://tmacchant3.starfree.jp/gnuplot/Eng/winbin/
Please try gnuplot.exe in the above and report results.

[Eli Zaretskii]

Thank you.

I installed the 32-bit binaries from there on a Windows 7 machine (I understand XP is no longer supported). It seems to work much better with Emacs, but there is one strange glitch: if I set the terminal to "windows", the GUI window opens, but is blank. I need to click on Options and select the GDI+ backend, then the graph appears. So it looks like some problem with the Direct2D backend, which is the default.

[Bastian Märkisch]

Thanks for the informative bug report and verifying that the (original) issue is solved. Do I understand correctly that both win/GDI+ and wxt now work for you? About the D2D initialisation issue I have to investigate - I do not have that problem here.

Note that as a work-around you can change the default backend to GDI+ by setting GraphGDI+=1 and GraphD2D=0 in wgnuplot.ini in you %APPDATA% directory. The Update wgnuplot.ini menu item should do that for you.

Concerning XP: it should still be easy enough to compile for XP. The only code ingnuplot which requires Win7 directly is the D2D backend, which can be left out. But the MSYS2/MinGW64 + Qt + ... compiler system and ecosystem we use requires Win7 or later.

[Eli Zaretskii]

Yes, "windows" with GDI+ and "wxt" work in the snapshot I tried. ("qt" also works, as it did before.) I hope you will be able to find the issue with Direct2D; let me know if I can provide some additional information to help you. Or maybe GDI+ should become the default?

I understand the issue with XP support. It means gnuplot 5.2.6 (or 5.2.7 if there will be a bugfix release from the 5.2 series) is the last version that supports XP, is that right?

[Bastian Märkisch]

Or maybe GDI+ should become the default?

The windows terminal underwent a transition from "GDI only" to "GDI mixed with GDI+" to "GDI+" and "D2D". With version 5.2 we removed the transitional "GDI/GDI+ mixed version", introduced "D2D" and deprectated the "GDI" code. Version 5.3 removes "GDI" from the default build. The idea is to also fade out GDI+ eventually in favor of the superior (i.e. faster, better font support) D2D variant. We still need to keep it to produce EMF(+) files though.

I understand the issue with XP support. It means gnuplot 5.2.6 (or 5.2.7 if there will be a bugfix release from the 5.2 series) is the last version that supports XP, is that right?

My understanding is that the compiler environment is really the limiting factor (MSYS2 is rolling-release....). I remember having checked which binary version actually was the last to support Windows XP, see my post here: [support-requests:#227]. The last binary version on SF which runs on XP is 5.0.4.

[Bastian Märkisch]

I have opened up a follow-up bug report for this issue, so it does not get lost [#2123].
Unfortunately, I cannot reproduce this behaviour using a Windows 10 machine at the moment. So this might be a bit tricky.