Problem with using nowrap on named pipe

  • George James

    George James - 2011-09-03

    I'm having trouble using named pipes (FIFOs) to zip the output of a gtm process.

    I'm running GT.M V5.4-001 Linux x86_64 on Debian and I'm using the following code on the GT.M side:

    pipe ; Test output to named pipe
    s pipe="output.pipe"
    o pipe:(nowrap:fifo)
    u pipe
    f i=1:1:33 w $tr($j("",1000)," ","a")
    w "z"
    c pipe

    At the other end of the pipe I have:
    rm output.pipe; mkfifo output.pipe ; gzip <output.pipe >output.gz

    If the output is longer than 32k then it gets truncated.  If I remove the nowrap argument then I get all the output in the gz file, but it has an unwanted linefeed at the 32k boundary.

    Is this a known problem?  Is there any workaround?

  • Bob Isch

    Bob Isch - 2011-09-03

    Yes…  Wrapping and "record length" logic seems a bit convoluted at best.  Someone should probably put together a "How-To" based list for common use cases and it would be nice if the defaults were more "Unix-like" - as in "I just want simple byte-streams, don't try to 'help' me by inserting 'stuff' unless I ask for it…"

    Anyway, general workarounds for this type of thing include:

    1) You can add an explicit recordsize, which moves the problem out and probably sucks up memory.  This works up to about 2^20 bytes:

       o pipe:(nowrap:fifo:recordsize=2**20)

    2) You can reset $x between writes - this "tricks" GT.m into not thinking it needs to add the LF:

       f i=1:1:33 w $tr($j("",1000)," ","a") s $x=0

    Of course, this only works if you are outputting less than the number of bytes in the "buffer" at a time.

    3) You still get the "extra" LF at the end of the file unless you do this before the CLOSE also:

       s $x=0 c pipe

    4) In this case (with fifo) the following seems to work best:

       o pipe:(nowrap:fifo:stream)

    I believe that STREAM is supposed to get us back to a more "Unix-like" byte-stream and it seems to be pretty reliable these days but YMMV for Sequential Files, Pipes, Fifos, Sockets, etc.

    Hope that helps,

  • George James

    George James - 2011-09-03

    Thanks using (nowrap:fifo:stream) seems to work.

    It does lead on to another problem that I've noticed.  If the GT.M process is run from a terminal session and the terminal session gets disconnected then the process appears to lock up.  There is no further output to the pipe.


  • Bob Isch

    Bob Isch - 2011-09-03

    Interesting.  Whenever I try "killing" the GT.m process (even with kill -9 or mupip stop) the process on the other end of the pipe seems to act like it was closed properly, or the signal is just ignored.  Probably the "terminal session" is still alive and blocking on terminal I/O.  I suspect it will timeout and clean things up eventually.  Do you see the process in 'ps -e'?

  • George James

    George James - 2011-09-03

    Yes, I can still see it in the process list.  It doesn't go away quickly and I'm not sure how long it persists,

    The process isn't doing *any* terminal I/O.  Normally a process will just continue until such time as it completes or it tries to do some I/O to the terminal.  When it finds that the terminal isn't there then it dies.

    In this case there's no terminal I/O and so I'd expect it continue to completion.  Seems like there's something about having a FIFO device open that requires it to have an established terminal session. 

    Anyway, for the moment, I've set it up to run from a cron job which means that I'm avoiding this problem.



Log in to post a comment.

Get latest updates about Open Source Projects, Conferences and News.

Sign up for the SourceForge newsletter:

JavaScript is required for this form.

No, thanks