#972 Lines() / chars() return 1 for stdin when no data is present

4.1.2
closed
Mark Miesfeld
none
5
2012-09-07
2010-12-28
U. Zinngrebe
No

These problems were supposedly fixed in 4.1.0 but apparently aren't.
Also in trunk


OK as first stage in the pipeline:

uli@ulmo:~/bin> rexx test.rex <temp.dat | more
HELLO
aaa
bbb
ccc
GOODBYE


As second stage in the pipeline, STDIN does not receive input:

uli@ulmo:~/bin> more <temp.dat | rexx test.rex
HELLO
GOODBYE


test.rex:

say hello

do while lines()
line = linein()
call lineout , line
-- say line
end

say goodbye

temp.dat:

aaa
bbb
ccc


uli@ulmo:~/bin> rpm -qf /usr/bin/rexx
ooRexx-4.1.0-6435.opensuse1130.x86_64

uli@ulmo:~/bin> rexx -v
Open Object Rexx Version 4.1.0
Build date: Dec 3 2010
Addressing Mode: 64


After re-installing from trunk (ver. 6525) the problem appears solved,
except that in the second pipeline stage lines() is true one line beyond the end of file:

uli@ulmo:~/bin> more <temp.dat | rexx test.rex
HELLO
aaa
bbb
ccc

GOODBYE

uli@ulmo:~/bin> rexx test.rex <temp.dat | more
HELLO
aaa
bbb
ccc
GOODBYE

uli@ulmo:~/bin> rpm -qf /usr/bin/rexx
ooRexx-4.2.0-6525.opensuse1140.x86_64

uli@ulmo:~/bin> uname -a
Linux ulmo 2.6.34.7-0.5-desktop #1 SMP PREEMPT 2010-10-25 08:40:12 +0200 x86_64 x86_64 x86_64 GNU/Linux

Discussion

  • U. Zinngrebe
    U. Zinngrebe
    2010-12-28

     
    Attachments
  • Mark Miesfeld
    Mark Miesfeld
    2012-08-04

    Some notes on this. In Windows it looks okay to me in both 4.1.0 and trunk:

    C:\work.ooRexx\bug.work\stdinPipeLineReappeared>rexx -v
    Open Object Rexx Version 4.1.0
    Build date: Dec 5 2010
    Addressing Mode: 64

    C:\work.ooRexx\bug.work\stdinPipeLineReappeared>more <temp.dat | rexx test.rex
    HELLO
    aaa
    bbb
    ccc
    GOODBYE

    C:\work.ooRexx\bug.work\stdinPipeLineReappeared>rexx -v
    Open Object Rexx Version 4.2.0 - Internal Test Version
    Build date: Jul 30 2012
    Addressing Mode: 64

    C:\work.ooRexx\bug.work\stdinPipeLineReappeared>more <temp.dat | rexx test.rex
    HELLO
    aaa
    bbb
    ccc
    GOODBYE

    Note that the empty line depends on the format of temp.dat

    If temp.dat is this, in hex:

    616161OD 0A626262 0D0A6363 630D0A

    The output is as above. But, if temp.dat is this, in hex:

    616161OD 0A626262 0D0A6363 630D0A0D
    0A

    The output is this:

    C:\work.ooRexx\bug.work\stdinPipeLineReappeared>more <temp.dat | rexx test.rex
    HELLO
    aaa
    bbb
    ccc

    GOODBYE

    C:\work.ooRexx\bug.work\stdinPipeLineReappeared>

    which also seems correct to me.

    I'll test tomorrow on a Linux system.

     
  • U. Zinngrebe
    U. Zinngrebe
    2012-08-04

    Hi Mark,

    thank you for looking into that old bug report. Two years later I find it difficult to understand myself.

    I just checked (trunk, rev. 7988) and the empty line is still there. The file temp.dat doesn't have two trailing end of line sequences:

    uli@ulmo:~> echo 'aaa' >temp.dat
    uli@ulmo:~> echo 'bbb' >>temp.dat
    uli@ulmo:~> echo 'ccc' >>temp.dat
    uli@ulmo:~> cat temp.dat
    aaa
    bbb
    ccc
    uli@ulmo:~> cat bin/test

    Then I modified the test program to see what is coming through the pipeline, and it is the nul string:

    uli@ulmo:~> cat bin/test.rex
    say hello
    do i=1 while lines()
    line = linein()
    call lineout, i c2x(line) '.' c2d(line)
    end
    say goodbye
    uli@ulmo:~> more <temp.dat |="" rexx="" test.rex="" HELLO="" 1="" 616161="" .="" 6381921="" 2="" 626262="" .="" 6447714="" 3="" 636363="" .="" 6513507="" 4="" .="" 0="" GOODBYE="" uli@ulmo:~="">

    That makes me wonder whether the more command would append an end-or-file sequence (like CTRL-Z),
    which is recognised & discarded by linein(), but taken for data by lines().

    Cheers, Uli

    btw: isn't it odd that c2d() returns zero for the nul string ?

     
  • Mark Miesfeld
    Mark Miesfeld
    2012-08-04

    Hi Uli,

    I'm going to change the title of this bug to reflect the odd blank line we see in Linux.

    The actual piping works in Linux under both 4.1.1 and trunk. I didn't go back and test 4.1.0, the bug could exist there, but as long as it is not in the base for 4.1.2 or in trunk I think we are good.

    In the below, temp.dat does not have a new line at the end of the third line. It is just:

    616161OA 626262 0A6363 63

    [miesfeld@Hawk flushAtClose]$ rexx -v
    Open Object Rexx Version 4.1.1
    Build date: Aug 4 2012
    Addressing Mode: 64

    [miesfeld@Hawk flushAtClose]$ more <temp.dat | rexx test.rex
    HELLO
    aaa
    bbb
    ccc
    GOODBYE
    [miesfeld@Hawk flushAtClose]$ rexx -v
    Open Object Rexx Version 4.2.0
    Build date: Aug 4 2012
    Addressing Mode: 64

    [miesfeld@Hawk flushAtClose]$ more <temp.dat | rexx test.rex
    HELLO
    aaa
    bbb
    ccc
    GOODBYE

    If I add a new line at the end of the third line, i.e.:

    616161OA 626262 0A6363 630A

    then I get that empty line:

    [miesfeld@Hawk flushAtClose]$ more <temp.dat | rexx test.rex
    HELLO
    aaa
    bbb
    ccc

    GOODBYE
    [miesfeld@Hawk flushAtClose]$

     
  • Mark Miesfeld
    Mark Miesfeld
    2012-08-05

    Committed revision 8134. trunk.

    The problem here was really with the blank line. Uli's first assetion that the bug(s) had reappeared was not really correct.

    Normally, stdin is marked as a TTY and this code in hasData() gives the correct results of stdin:

    // tty devices require special handling
    if (isTTY)
    {
        int bytesWaiting;
        ioctl(fileHandle, FIONREAD, &bytesWaiting);
        if (bytesWaiting)
        {
            return true;
        }
        else {
            return false;
        }
    }
    
    // we might have something buffered, but also check the
    // actual stream.
    return !atEof();
    

    But when stdin comes through a pipe, it is not marked as a TTY and, on unix, atEof() does not return true until after an attempt is made to read stdin that fails.

    On Windows atEof() works correctly for stdin() because it uses a function not available on unix.

     
  • Mark Miesfeld
    Mark Miesfeld
    2012-08-05

    Hi Uli,

    Just to answer some of your last questions. If you change your test program to this, I think you will understan what was happening. (Run it before you get the lastest from trunk.)

    line = ""
    say 'Blank line c2d:' c2d(line)
    say
    say hello

    i = 0
    do while lines()
    i += 1
    say 'Loop' i
    line = linein(); say stream( stdin, 'STATE')
    call lineout , line
    end

    say goodbye

    You will get this:

    Blank line c2d: 0

    HELLO
    Loop 1
    READY
    aaa
    Loop 2
    READY
    bbb
    Loop 3
    READY
    ccc
    Loop 4
    NOTREADY

    GOODBYE

    So, c2d() on a blank line gives you the 0.

    If you look at the above, you see that it goes through the loop 4 times rather than 3. That is because lines() was returning 1 when there actually was not any characters to be read. When you do the "line = linein()" in the 4th loop, it actually fails, but your code does a lineout() any way. line is a blank line, but the blank line was not really read from stdin(); linein() actually failed.

    If in your code you checked for not ready after each linein() and did not do a lineout() if the linein() failed, the results would be as expected.

     
  • Mark Miesfeld
    Mark Miesfeld
    2012-08-06

    Committed revision 8138.

     
  • Mark Miesfeld
    Mark Miesfeld
    2012-08-06

    Committed revision 8138. this was to the 4.1 fixes branch.

     
  • Mark Miesfeld
    Mark Miesfeld
    2012-09-07

    • status: pending --> closed
    • pending_work_items: --> none
     


Anonymous


Cancel   Add attachments