I'm having trouble using named pipes (FIFOs) to zip the output of a gtm process.
I'm running GT.M V5.4-001 Linux x86_64 on Debian and I'm using the following code on the GT.M side:
pipe ; Test output to named pipe
s pipe="output.pipe"
o pipe:(nowrap:fifo)
u pipe
f i=1:1:33 w $tr($j("",1000)," ","a")
w "z"
c pipe
q
At the other end of the pipe I have:
rm output.pipe; mkfifo output.pipe ; gzip <output.pipe >output.gz
If the output is longer than 32k then it gets truncated. If I remove the nowrap argument then I get all the output in the gz file, but it has an unwanted linefeed at the 32k boundary.
Is this a known problem? Is there any workaround?
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Yes… Wrapping and "record length" logic seems a bit convoluted at best. Someone should probably put together a "How-To" based list for common use cases and it would be nice if the defaults were more "Unix-like" - as in "I just want simple byte-streams, don't try to 'help' me by inserting 'stuff' unless I ask for it…"
Anyway, general workarounds for this type of thing include:
1) You can add an explicit recordsize, which moves the problem out and probably sucks up memory. This works up to about 2^20 bytes:
o pipe:(nowrap:fifo:recordsize=2**20)
2) You can reset $x between writes - this "tricks" GT.m into not thinking it needs to add the LF:
f i=1:1:33 w $tr($j("",1000)," ","a") s $x=0
Of course, this only works if you are outputting less than the number of bytes in the "buffer" at a time.
3) You still get the "extra" LF at the end of the file unless you do this before the CLOSE also:
s $x=0 c pipe
4) In this case (with fifo) the following seems to work best:
o pipe:(nowrap:fifo:stream)
I believe that STREAM is supposed to get us back to a more "Unix-like" byte-stream and it seems to be pretty reliable these days but YMMV for Sequential Files, Pipes, Fifos, Sockets, etc.
Hope that helps,
-bob
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Bob
Thanks using (nowrap:fifo:stream) seems to work.
It does lead on to another problem that I've noticed. If the GT.M process is run from a terminal session and the terminal session gets disconnected then the process appears to lock up. There is no further output to the pipe.
George
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Interesting. Whenever I try "killing" the GT.m process (even with kill -9 or mupip stop) the process on the other end of the pipe seems to act like it was closed properly, or the signal is just ignored. Probably the "terminal session" is still alive and blocking on terminal I/O. I suspect it will timeout and clean things up eventually. Do you see the process in 'ps -e'?
-bob
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Yes, I can still see it in the process list. It doesn't go away quickly and I'm not sure how long it persists,
The process isn't doing *any* terminal I/O. Normally a process will just continue until such time as it completes or it tries to do some I/O to the terminal. When it finds that the terminal isn't there then it dies.
In this case there's no terminal I/O and so I'd expect it continue to completion. Seems like there's something about having a FIFO device open that requires it to have an established terminal session.
Anyway, for the moment, I've set it up to run from a cron job which means that I'm avoiding this problem.
George
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I'm having trouble using named pipes (FIFOs) to zip the output of a gtm process.
I'm running GT.M V5.4-001 Linux x86_64 on Debian and I'm using the following code on the GT.M side:
pipe ; Test output to named pipe
s pipe="output.pipe"
o pipe:(nowrap:fifo)
u pipe
f i=1:1:33 w $tr($j("",1000)," ","a")
w "z"
c pipe
q
At the other end of the pipe I have:
rm output.pipe; mkfifo output.pipe ; gzip <output.pipe >output.gz
If the output is longer than 32k then it gets truncated. If I remove the nowrap argument then I get all the output in the gz file, but it has an unwanted linefeed at the 32k boundary.
Is this a known problem? Is there any workaround?
Yes… Wrapping and "record length" logic seems a bit convoluted at best. Someone should probably put together a "How-To" based list for common use cases and it would be nice if the defaults were more "Unix-like" - as in "I just want simple byte-streams, don't try to 'help' me by inserting 'stuff' unless I ask for it…"
Anyway, general workarounds for this type of thing include:
1) You can add an explicit recordsize, which moves the problem out and probably sucks up memory. This works up to about 2^20 bytes:
2) You can reset $x between writes - this "tricks" GT.m into not thinking it needs to add the LF:
f i=1:1:33 w $tr($j("",1000)," ","a") s $x=0Of course, this only works if you are outputting less than the number of bytes in the "buffer" at a time.
3) You still get the "extra" LF at the end of the file unless you do this before the CLOSE also:
4) In this case (with fifo) the following seems to work best:
I believe that STREAM is supposed to get us back to a more "Unix-like" byte-stream and it seems to be pretty reliable these days but YMMV for Sequential Files, Pipes, Fifos, Sockets, etc.
Hope that helps,
-bob
Bob
Thanks using (nowrap:fifo:stream) seems to work.
It does lead on to another problem that I've noticed. If the GT.M process is run from a terminal session and the terminal session gets disconnected then the process appears to lock up. There is no further output to the pipe.
George
Interesting. Whenever I try "killing" the GT.m process (even with kill -9 or mupip stop) the process on the other end of the pipe seems to act like it was closed properly, or the signal is just ignored. Probably the "terminal session" is still alive and blocking on terminal I/O. I suspect it will timeout and clean things up eventually. Do you see the process in 'ps -e'?
-bob
Yes, I can still see it in the process list. It doesn't go away quickly and I'm not sure how long it persists,
The process isn't doing *any* terminal I/O. Normally a process will just continue until such time as it completes or it tries to do some I/O to the terminal. When it finds that the terminal isn't there then it dies.
In this case there's no terminal I/O and so I'd expect it continue to completion. Seems like there's something about having a FIFO device open that requires it to have an established terminal session.
Anyway, for the moment, I've set it up to run from a cron job which means that I'm avoiding this problem.
George