#1756 pipe out line ending problem on windows

[Anonymous]

Using version 5.0.3 on windows, suppose that we have the following python3 program (test.py):

from sys import stdin
data = stdin.buffer.read(None)
open("test.png","wb").write(data)

This program simply reads from the standard input binary stream and writes what it sees into test.png. For example type sample.png | test.py effectively copies sample.png into test.png.

Now suppose that we run the following commands in gnuplot

set term png
set output "| test.py"
plot sin(x)
set output

this will create a plot and pipe it through that python program. However, the resulting png is unreadable. After some investigation, I discovered the problem is that the file that is written out using the normal output methods (set output sample.png) differs from the piped version in that every occurance of "\n" in the good version is replaced by "\r\n" in the bad version. In other words, every newline in the good version is replaced by the windows newline in the bad version.

I don't know if the fault lies with Windows or with gnuplot here. I suspect that gnuplot is translating the newlines itself. This is exactly what should be happening for text output, but is not desireable for binary output. It is possible to replace the bad newlines with good newlines with another process (set output "| fixnewlines.py | test.py" where fixnewlines simply replaces "\r\n" with "\n"), so there is a workaround, but it is a bit of a hack.

Obviously, this example isn't particularly useful. There is no reason to pipe through a program that simply writes the results to disc, but similar scenarios exist where we may be piping through a conversion program, a server, or any number of things. This example is useful for illustrating the problem, however.

There should be a way to turn off the newline behavior in this case - possibly a command like set lineendings unix where the normal windows behavior could be restored with set lineendings windows. There may be such an option but if there is, I am unaware of it.

[Ethan Merritt]

Is this the libgd png terminal or the cairo png terminal?
Either way the output is done via a single call to a graphics library routine after the image is complete, not line by line from gnuplot.

[Matthew Halverson]

I believe that it is the libgd png terminal, but I just checked with both and it looks like it occurs with both. Looking at the beginning of the file with python with

data = open("test.png","rb").read()
data[:10] # b'\x89PNG\r\r\n\x1a\r\n'

we see that in the file "\n" occurs in "\r\n" combinations. Replacing these with just "\n" and writing out

data = data.replace("\r\n","\n")
open("test2.png","wb").write(data)

we get a png that is viewable.

The same problem occurs with the gif terminal (set term gif).

[Ethan Merritt]

This sounds like an issue with pipes on Windows. It's not clear from
your tests whether the problem is on the libgd output end or the
python input end or both. I would guess that in principle one or both
ends could be flagged as binary data, but I don't know where that
would be done. Maybe ask on stackoverflow or a Windows forum?

On Thu, Mar 17, 2016 at 11:50 AM, Matthew Halverson
mhalver@users.sf.net wrote:

I believe that it is the libgd png terminal, but I just checked with both and it looks like it occurs with both. Looking at the beginning of the file with python with

~~~~
data = open("test.png","rb").read()
data[:10] # b'\x89PNG\r\r\n\x1a\r\n'
~~~~

we see that in the file "\n" occurs in "\r\n" combinations. Replacing these with just "\n" and writing out

~~~~
data = data.replace("\r\n","\n")
open("test2.png","wb").write(data)
~~~~

we get a png that is viewable.

The same problem occurs with the gif terminal (set term gif).

---

** [bugs:#1756] pipe out line ending problem on windows**

Status: open
Priority:
Labels: windows pipe output lineendings line-endings
Created: Thu Mar 17, 2016 08:05 AM UTC by Anonymous
Last Updated: Thu Mar 17, 2016 06:14 PM UTC
Owner: nobody

Using version 5.0.3 on windows, suppose that we have the following python3 program (test.py):

~~~~
from sys import stdin
data = stdin.buffer.read(None)
open("test.png","wb").write(data)
~~~~

This program simply reads from the standard input binary stream and writes what it sees into test.png. For example type sample.png | test.py effectively copies sample.png into test.png.

Now suppose that we run the following commands in gnuplot

~~~~
set term png
set output "| test.py"
plot sin(x)
set output
~~~~

this will create a plot and pipe it through that python program. However, the resulting png is unreadable. After some investigation, I discovered the problem is that the file that is written out using the normal output methods (set output sample.png) differs from the piped version in that every occurance of "\n" in the good version is replaced by "\r\n" in the bad version. In other words, every newline in the good version is replaced by the windows newline in the bad version.

I don't know if the fault lies with Windows or with gnuplot here. I suspect that gnuplot is translating the newlines itself. This is exactly what should be happening for text output, but is not desireable for binary output. It is possible to replace the bad newlines with good newlines with another process (set output "| fixnewlines.py | test.py" where fixnewlines simply replaces "\r\n" with "\n"), so there is a workaround, but it is a bit of a hack.

Obviously, this example isn't particularly useful. There is no reason to pipe through a program that simply writes the results to disc, but similar scenarios exist where we may be piping through a conversion program, a server, or any number of things. This example is useful for illustrating the problem, however.

There should be a way to turn off the newline behavior in this case - possibly a command like set lineendings unix where the normal windows behavior could be restored with set lineendings windows. There may be such an option but if there is, I am unaware of it.

---

Sent from sourceforge.net because you indicated interest in https://sourceforge.net/p/gnuplot/bugs/1756/

To unsubscribe from further messages, please visit https://sourceforge.net/auth/subscriptions/

[Matthew Halverson]

I have tried with both the png (libgd terminal) and the pngcairo terminal with the same results. Additionally, I've been able to pump data through the python program from other sources, so the problem is only occuring when the data is coming from gnuplot.

By reading the data in with the python program by stdin.buffer.read instead of stdin.read, it is being treated as binary data.

[Ethan Merritt]

At line 346 of term.c I see

#ifdef OS2
# define POPEN_MODE ("wb")
#else
# define POPEN_MODE ("w")
#endif

Possibly if you change that to "wb" for the Windows case also you would see different behavior, but I'm just guessing blindly. Hopefully one of the developers working with Windows can offer help.

[Matthew Halverson]

I've been staring at the source code trying to find something that could cause it, but as my C knowledge is only slightly above "Hello, World!" level, and I'm not familiar with the structure of the code, I wasn't having much luck with it.

I am certain that is where the problem is, as that is used further down at line 380 to open the pipe. Doing some reseach, the first paragraph at https://cygwin.com/cygwin-ug-net/using-textbinary.html describes exactly what I am seeing, and some further research suggests that mode "w" and mode "wb" are treated the same under the POSIX standard (so either of those can be used on Linux), but they are different under other systems (see the quote at http://stackoverflow.com/questions/5302196/popen-to-pass-binary-data-between-processes). In particular, Windows will mangle "\n" exactly as I am seeing.

It is unused other than at that one line, so it can be safely changed without altering any other behavior.

I suspect that this was written that way because of a similar problem on OS2, but nobody realized that the same thing happens on Windows as well. Originally, it was probably just equal to w for everything. The else condition is probably meant to cover all POSIX compliant systems where the distinction doesn't matter. In fact, it may even be ok to just set it equal to wb in all cases - it won't hurt POSIX compliant systems, but will fix problems with all non-compliant systems.

I am unable to compile the source under windows, so hopefully one of the windows developers can take a look at it. This has to be the cause of the bug.

[Tatsuro MATSUOKA]

#ifdef OS2
# define POPEN_MODE ("wb")
#else
# define POPEN_MODE ("w")
#endif

Ah! You are absolutely right. On windows one has to use "wb" for binary file.
I do not why no one has been aware for that.

A patch is attached.

Binary with patch applied is available from the following:
http://www.tatsuromatsuoka.com/gnuplot/Eng/winbin/gp510-20160318-win32-mingw.patched.zip
(The above will be deleted after the test.)

[Ethan Merritt]

My concern is that many terminal types really do put out line-terminated text files (postscript, svg, latex, ...). If this line is changed would it fix pipes containing png output but break pipes containing anything else?

[Tatsuro MATSUOKA]

You are right. The patch gives incorrect results.

I have tested in png and svg terminal.
The patched binary gives empty png and avg files.

One has to consider other way.

[Matthew Halverson]

Those two terminals are working for me with the patched binary.

[Tatsuro MATSUOKA]

BTW, gnuplot binary works on wine. You can test by yourself using the test binary.

[Matthew Halverson]

Most windows programs are perfectly happy with "\n" for a line terminator. The days of complaining about not having "\r\n" seem to be past (Notepad.exe is the only program I know of that still doesn't like "\n"), so it probably won't break anything else.

I tried the provided binary and the png and svg terminals work fine

set term png
set output "test.png"
plot sin(x)
set output

and

set term svg
set output "test.svg"
plot sin(x)
set output

but now when doing set output "| test.py" (with the inital program test.py that I gave) I am getting an error: ** Could not execute pipe ' test.py'. Writing to binary pipes is not supported.** I suspect that there was an option that wasn't enabled when that binary was compiled (or is is a problem somewhere in one of the libraries). Otherwise, this probably will fix the bug.

[Tatsuro MATSUOKA]

After reboot PC, the patched binary gives not empty png and svg files. Thanks!

[Matthew Halverson]

What about the writing to binary pipes is not supported error message? Can that be fixed? As of now, using that provided binary, this seems to disable output pipes altogether on Windows.

[Matthew Halverson]

The entire text of test.py occurs in my original post. It is simply the three lines

from sys import stdin
data = stdin.buffer.read(None)
open("test.png","wb").write(data)

The original problems is that when running

set term png
set output "| test.py"
plot sin(x)
set output

in order to pipe the output through the program test.py, the resulting test.png was unreadable because the pipe replaced all ocurrences of "\n" with "\r\n" which is fine for text data, but obviously unacceptable for binary data.

Adding the line data = data.replace("\r\n","\n") between the second and third line in test.py makes the above create a working png. So there is a work-around, but the issue is that the output is getting mangled because of using mode w instead of mode wb.

With the patched binary, unfortunately this set output line (set output "| test.py") now does not even work. Instead it gives the error message writing to binary pipes is not supported

It looks like this error is raised by line 880 in src/win/winmain.c

[Tatsuro MATSUOKA]

The lack of python knowldegement, I cannot try for fix this issue at the moment.
I have python 2.7 but not 3 and test.py should be placed at the appropriate folder.
Please give me advise for python.

[Matthew Halverson]

Python is not actually the issue. Any process that reads the standard input and writes it to a file can be used to demonstrate the same problem.

If testprogram is any program that reads standard input and writes it to a file test.png, then set output "| testprogram" should demonstrate the same problem.

[Matthew Halverson]

It does look like fixing this may break code elsewhere. Most of the code is well beyond my knowledge of C, and I don't have a clue what these "fake pipes" starting at line 828 of winmain.c are, but they seem to rely on the the mode being w instead of wb. Without changing these, the code will not support the other change to mode wb. If these could not be modified to work with the wb mode, that could fix this problem.

Unless the changes can be made to winmain.c as well, this may need to be left alone. It may just be mode w has to be left and the implementation isn't perfect on windows. That may be why the wrong mode is used on windows.

There is a workaround. If patchnewlines is a program that translates "\r\n" to "\n", then

set output "| patchnewlines | process" works to pipe data to process under windows when the data is binary. The fact that this mangling is consistant means that it can be reversed by another process.

This extra step is annoying (set output "| process" would work directly on Linux), but it can be done.

[Tatsuro MATSUOKA]

I have written a small code. (aaa.c)

#include <stdio.h>

int main(void)
{
    int a;
    while(1){
        a=getchar();
        if(a==EOF) return(0);
        putchar(a);
    }
}

and test on patched gnuplot.exe

gnuplot> set term png
Terminal type set to 'png'
Options are 'nocrop enhanced size 640,480 font "arial,12" '
gnuplot> set out "|aaa"
gnuplot> plot sin(x)
ëPNG
gnuplot> set out
gnuplot>

and I did not see "writing to binary pipes is not supported"

[Tatsuro MATSUOKA]

> what these "fake pipes" starting at line 828 of winmain.c are

"fake pipes" is implemented by Bastian to emulated pipe on wgnuplot.exe and is not irrevant to gnuplot.exe.

[Matthew Halverson]

Yes, I am now seeing that the error message (writing to binary pipes is not supported) is being generated on wgnuplot.exe but not gnuplot.exe, so this patch breaks wgnuplot.exe.

[Matthew Halverson]

I am still seeing the same problem occur with the patched version (using gnuplot.exe). Namely that the data piped out using set output "| someprocess" is translating "\n" to "\r\n", so it seems that this is not the cause of the problem, but does break wgnuplot.exe.

[Tatsuro MATSUOKA]

OK. "line 346 of term.c" is not the place to attatch the patch. But I do have enough time to see further at the moment.

[Matthew Halverson]

When you can get back to it, and beings you were unsure of my python program, I've been trying the past couple of hours to create a similar tool in C (which I am very inexperienced with, so I don't promise that this a well designed tool, and likely has many issues, and I think has an unecessary header, but it can be used to illustrate the same problem). Here is testoutput.c

#include <stdio.h>
#include <fcntl.h>
#include <io.h>

int main() {
    _setmode(_fileno(stdin),_O_BINARY);
    FILE *output;
    output = fopen("test.png","wb");
    int c;
    while ((c=fgetc(stdin))!=EOF) {
        fputc(c,output);
    }
    fclose(output);
}

That does the same thing as the python program (reads standard input as binary and writes to a file test.png). Compile that as testoutput.exe and in gnuplot run

set term png
set output "| testoutput"
plot sin(x)
set output

this will pipe the png through testoutput which writes it to test.png. You will find that test.png is unreadable, but if you replace every occurence of "\r\n" with "\n" in it, it becomes workable. Thus the problem is that the newline characters are getting mangled on the way out of the pipe (which is fine with text but bad for binary).

I really hope that that helps illustrate the problem further, and is clearer to you than my python example, or at least more usable.