#1460 fileevent readable translation mode auto

obsolete: 8.3.2

When fileevent (in translation mode "auto") sees a CR,
it apparently
waits to see if the next character is a LF before it
proclaims EOL. It
is looking one character ahead. In this way it avoids
proclaiming two
EOL's for one CRLF sequence.

Would it not be better for fileevent to look one
character behind
instead? In this mode, when it sees a CR, it could
immediately proclaim
EOL. Also, when it sees a LF, immediately proclaim EOL,
*unless* the
previous character was CR, in which case the LF could
just be


  • tcl and sh scripts

  • Logged In: YES

    IIRC from the discussion this request arose from, we should use a little (two-state) state
    machine to spot whether to gobble CR/LF (the state is a boolean saying whether the last
    character seen was a linefeed.) None of this is needed when in any other line-translation mode
    than auto.

  • Logged In: YES

    Note, that we have such an state-machine, search for SAW_CR.
    It rather seems that there is some bad interaction of this
    machine with the event system.

    • priority: 5 --> 6
    • assigned_to: nobody --> andreas_kupries
  • Logged In: YES

    All new files mentioned below are part of the uploaded
    archive "fet-notes.tar.gz".

    When trying out the example I noted that for me
    "fileeventthing-sh" did _not_ generate lines having \r (CR)
    as line separator. See file "shout" (SHell OUT) for an
    example, looked not with an editor, but 'od -c' to get a
    unimpaired look into the file.

    Because of this I wrote "numbergen", a tcl script doing the
    same. Its output is "shut.ng" and does contain the correct
    line separators, i.e. CR (side note: My emacs went into Mac
    submode! when displaying the file).

    The script "fet" is "fileeventthing-tclsh" modified to use
    "numbergen" for the spawned command pipe. It was also
    modified to display the times the fileevent was fired before
    printing the received line.

    Now testing the interaction all lines came in 1-second
    intervals, including the line for number 10. After that line
    was a 21-second wait for EOF, as expected.

    See file "fet.out" for a log.

    So, I was unable to reproduce the error on my system:

    % parray tcl_platform
    tcl_platform(byteOrder) = littleEndian
    tcl_platform(machine) = i586
    tcl_platform(os) = Linux
    tcl_platform(osVersion) = 2.2.17
    tcl_platform(platform) = unix
    tcl_platform(user) = tcl
    % info tclversion
    % info patchlevel

    Griff, please use the new scripts to check this out on your
    system too.

    Note, "numbergen" does [fconfigure stdout -buffering line
    -translation cr] to get the right line-separators and to
    avoid explicit flushing. This could be a difference with the
    original application too. IOW the original application /
    echo script may buffer the last line longer than is
    intended. The C library often acts differently, especially
    with regard to buffering output, when it detects that it
    talks to a pipe and not a pty (= terminal like xterm). This
    could be another reason for the problem you experience.

  • Archive with files to my notes above.

  • Logged In: NO


    Thanks for checking into this.

    I can see from the "shout" file that your echo
    command doesn't know about printf-style character
    sequences, like \r, \t, etc. or is in a mode
    where it is pretending not to know about them. I
    guess maybe I should have used a C program
    instead of a Bourne-shell script to generate the
    test output. Sorry about that. The echo thing is
    a common portability problem with scripts.

    Anyway, I have written a little C program to do
    the same thing, and I will upload it (if I can
    figure out how!). Obviously, you can compile it
    with something like "gcc -o fileeventthing
    fileeventthing.c" . Then edit
    fileeventthing-tclsh to call fileeventthing
    instead of the Bourne shell script.

    Note that I've put in an fflush after each number
    in the C program.

    Your numbergen script does not exactly duplicate
    what fileeventthing.c and fileeventthing-sh do,
    since you are outputting a CR at the end of each
    line, not the beginning. I did try it, though,
    and it worked the same for me as for you.

    So, let's start all over, but please use the
    fileeventthing.c program instead. Thanks!

  • Logged In: NO

    Hmm, I didn't see how to upload another file, so I will
    email fileeventthing.c to Andreas.

  • Logged In: YES

    Uploading "fileeventthing.c".

    And the fact that the \r is send at the _beginning_ of each
    line is also important, I missed that.

  • New generator application.

  • Logged In: YES

    Ok, tried it with the new generator ...
    I can now reproduce the reported problem, but it is slightly

    1. There is a delay in displaying the last line with number
    2. When the line is finally displayed the timecode in front
    of it (I used "fet") shows that the fileevent fired one
    second after the last one, IOW, immediately after the line

    This means that the [gets] blocked. And it is also clear why
    it blocked. There is neither CR nor LF after the last
    character (the '0' of the 10) of the last line. IOW the line
    is not complete, so Tcl IO system waits for more characters
    to complete the line. The wait stops only after the EOF
    arrives 20 seconds later.

    The fact \r is sent at the beginning of each line is
    crucical here, because it is now missing at the end of the
    last line. This line can be pronounced complete only after
    we got EOF.

  • Logged In: NO

    Duh, I see it all now.

    So fileevent readable *does* know about CR-only
    lines after all. We've been tackling the wrong
    issue, and it's my fault. Sorry!

    I think I got us off track when I made the
    assumption that the application that I am working with
    appends \r to the lines in question. It actually
    prepends them.

    Then we got more off track when you said (in

    > No, it just doesn't know wether the CR is alone
    > or will be followed by a LF and so its waits for
    > the next character to arrive to decide wether the
    > line-ending was the CR it saw or is CRLF.

    I took issue with *that* (though I think we know
    now that that isn't actually the case) and it
    snowballed from there. By the time I noted that
    you have to prepend the \r to duplicate the
    problem, I was so deep into your assertion that
    fileevent (in auto mode) *has* to see an LF after
    CR before proclaiming EOL that I didn't realize
    that by prepending \r gets would be what is
    blocking! Argh!

    Sorry! Let's just forget the whole thing.

    • status: open --> closed-invalid
  • Logged In: YES

    Closing this now that we know that it is no bug in Tcl at