#2532 windows: writing UTF-16 to a pipe is incorrect (was: pgf is unable to check gnuplot version

[Anonymous]

It seems like gnuplot v5.4 patchlevel 3 introduced a bug in the way the -V flag is processed.

In gnuplot v5.4 patchlevel 2 and older, the command gnuplot -V > file.txt generates a text file with the version number in it as expected.

On the same computer (Windows 11) when using gnuplot v5.4 patchlevel 3, the same version command generates an empty text file. This causes pgf to fail to recognize the gnuplot version. It then defaults to 4.4, and makes all my log-scale plots go haywire.

The test build of, gnuplot v5.5, the text file contains the version number, but it shows up like g n u p l o t 5 . 5 p a t c h l e v e l 0 with some non-ASCII character between all the letters. This leads pgf to throw an error that says

A funny symbol that I can't read has just been input.

If I fall back to patchlevel 2, my documents compile as expected.

[Tatsuro MATSUOKA]

Perhaps you mean gnuplot for native Windows.
For gnuplot-5.4.3 on the Cygwin and the Ubuntu (WSL),

gnuplot -V > file.txt

works as expected.
However, on 5.4.3 on windows (mingw), file.txt is empty as reportred.

I execute on 5.5 (mingw) built by myself it works as expected.

Please try my build 5.5 binary on http://tmacchant33.starfree.jp/gnuplot_bin.html .

[Rushi]

On version 5.5 downloaded from the link in your comment, I get

g n u p l o t 5 . 5 p a t c h l e v e l 0

Note the spaces between all the characters. Those show up as some non-printable characters in notepad++.

Those characters do cause problems with pgf version detection code. For example, compiling the code

\documentclass{standalone}
\usepackage{pgfplots}
\pgfplotsset{compat=1.18}
\begin{document}
\begin{tikzpicture}
    \begin{semilogxaxis}[xmin=1,xmax=100]
        \addplot[domain=1:100] gnuplot{log10(x)};
    \end{semilogxaxis}
\end{tikzpicture}
\end{document}

gives me the error

runsystem(gnuplot -V >test.pgf-plot.vrs)...executed.

! Text line contains an invalid character.
<read 2> g^^@
             n^^@u^^@p^^@l^^@o^^@t^^@ ^^@5^^@.^^@5^^@ ^^@p^^@a^^@t^^@c^^@h^^...
l.7         \addplot[domain=1:100] gnuplot{log10(x)};

A funny symbol that I can't read has just been input.
Continue, and I'll forget that it ever happened.

That same LaTeX code compiles correctly with version 5.4 patchlevel 2. The code compiles without errors with patchlevel 3, but because pgf cannot check gnuplot version, it assumes that gnuplot is providing output in logscale, which causes the plot to be curved instead of straight.

[Tatsuro MATSUOKA]

I cannot reproduce on my build.
For me, I got on notepad++ on windows 10 21H2

gnuplot -V > file.txt
gives

gnuplot 5.5 patchlevel 0

Does someone else try this on windows?

[Rushi]

I think I know what is going on. On my Windows 11 machine, if I use Windows Command Prompt, the version is printed out correctly. If I use Windows PowerShell, I get the problematic NUL characters.

Version 5.4, Patchlevel 2 works on both, though, so I assume this is fixable?

I can't figure out why LaTeX is using PowerShell, though, but on Windows 11, it seems like PowerShell is the default.

If anyone has access to Windows 11, please try this:
1) Open C:\WINDOWS\System32\cmd.exe
2) Run gnuplot -V > file.txt. This should result in a text file with the version number in it as expected, with no NUL characters.
3) In the same Command Prompt, try to compile this LaTeX code with pdflatex and shell-escape enabled:

\documentclass{standalone}
\usepackage{pgfplots}
\pgfplotsset{compat=1.18}
\begin{document}
\begin{tikzpicture}
    \begin{semilogxaxis}[xmin=1,xmax=100]
        \addplot[domain=1:100] gnuplot{log10(x)};
    \end{semilogxaxis}
\end{tikzpicture}
\end{document}

This should work in theory, but it does not. Instead, I get

! Text line contains an invalid character.
<read 2> g^^@
             n^^@u^^@p^^@l^^@o^^@t^^@ ^^@5^^@.^^@5^^@ ^^@p^^@a^^@t^^@c^^@h^^...
l.7             \addplot[domain=1:100] gnuplot{log10(x)};

?
! Emergency stop.

It seems like the shell-escape flag in LaTeX is invoking PowerShell and getting the NUL characters in the gnuplot version string even when LaTeX is run in Command Prompt.

[Rushi]

Looks like this issue has to do with character encoding, based on this question. Were there some changes in character encoding for the version string between versions 5.4 patchlevel 2 and 5.5?

[Hans-Bernhard Broeker]

Those funny characters are not, in fact, charactes. They're simply null bytes, inserted into the output because something convinced the runtime library of the used executable that it's supposed to be outputting its messages in UTF-16 encoding. Unlike the sane rest of the world, Microsoft decided to use that instead of UTF-8 to represent text internally, all the back when they started on Windows NT.

If you tell notepad++ about the text being in that format, it'll display it just fine.

[Rushi]

Thank you, I understand the issue now; however, the conflict with LaTeX and pgf still remains. I cannot compile LaTeX documents with log scale plots on Windows 11 using anything newer than version 5.4 patchlevel 2. All versions older than that work just fine.

Not sure if this needs to be fixed in gnuplot or in the Windows LaTeX distribution, but since older gnuplot versions work, I was hoping it would be an easy fix for gnuplot developers. If you all do not think so, I will create MikTeX and TexLive support tickets.

[Rushi]

If you tell notepad++ about the text being in that format, it'll display it just fine.

@broeker and @tmacchant3, I did some more digging and unfortunately, it does look like the text file that the gnuplot binary is outputting is corrupted, not just encoded in a different encoding.

On Windows 11, using PowerShell, if I execute

write-output "test" > ver.txt

I get a file that uses UTF-16 BOM LE encoding, which notepad recognizes without any additional actions on my part and displays correctly. However, with

gnuplot -V > ver.txt

I get a file that also uses UTF-16 BOM LE encoding, which notepad recognizes correctly, but still displays with the NUL characters.

[Tatsuro MATSUOKA]

Rushi

I have made test build gnuplot-5.4.4beta (Release candidate of 5.4.4) on windows(mingw64).
With 5.4.4,
gnuplot -V > file.txt
seems to work correctrly even on the PowerShell.

If you want try a test build 5.4.4beta binary package, please let me know (tmacchant3@yahoo.co.jp).
I will update it on my private web site and tell you the URL.

[Rushi]

Yes, I will be happy to give it a go. Thanks!

[Tatsuro MATSUOKA]

Rushi

http://tmacchant33.starfree.jp/Files/gnuplot-5.4.4beta/

[Rushi]

This works flawlessly, thanks!

[Tatsuro MATSUOKA]

Please use this until the official release of 5.4.4.

[Tatsuro MATSUOKA]

Ethan
For the development branch, gnuplot -V > ver.txt does not wok as expected (null bytes are inserted). Perhaps Bastian will solves the issue.

[Ethan Merritt]

I may have misunderstood. I thought that Rushi reported that the 5.4.4 executable you built was working without a problem.

Did you make a change for 5.4.4 that is not in the development version?
What was the change?
Is it a change to the source or a change to the build environment, or something else?

[Tatsuro MATSUOKA]

I did not do anything on 5.4 branch. Perhaps Bastian already made changes in stable branch before this report. I tried to see what were changed in the stable branch but I was not able to figure out what were changed for the fix. I will try to see the 5.4.3 and 5.4.4 sources again.

[Bastian Märkisch]

Wrinting to a pipe in UTF-16 is incorrect indeed. Re-opening.

[Bastian Märkisch]

Interestingly, the behaviour is unchanged for 6.0.2. A local build using MSYS2/CLANG64 does not show this behaviour. So the culprit might be the different C library, not gnuplot itself. My build uses the more modern UCRT C library instead of the MSVCRT. I will test this theory on an old machine which still has MINGW32/MINGW64 installed.
@tmacchant3 what is your setup to build the official releases?