Work at SourceForge, help us to make it a better place! We have an immediate need for a Support Technician in our San Francisco or Denver office.


#76 full_buffer keyword truncates whole buffer

Mutsuhito Iikura


I'm using Expect on Linux.
I found these two codes have different behavior.

# without full_buffer
expect {
Prompt {}

# with full_buffer
expect {
full_buffer {
exp_continue -continue_timer

The second case (expect with full_buffer) occasionally misses PROMPT.
But the first one does not.
This comes from the difference between buffer treating codes in 'expect.c'.
In the first case, buffer is truncated in function exp_buffer_shuffle.

expect.c: exp_buffer_shuffle
1967: skiplen = numchars/3;
1994: newlen = numchars - skiplen;
1995: memmove(str, p, newlen * sizeof(Tcl_UniChar));

In the second case, buffer truncation is prepared in function eval_case_string.
As you can see, full_buffer matches whole buffer. Not 1/3.

expect.c: eval_case_string
977: if (((expSizeGet(esPtr)*3) >= (esPtr->input.max*2)) && (numchars > 0)) {
978: o->e = e;
979: o->matchlen = numchars;
980: o->matchbuf = str;
981: o->esPtr = esPtr;

Therefore it misses PROMPT on the max size boundary.
I hope these are same. So I tried to fix as follows.
It's works fine for me.

979:- o->matchlen = numchars;
979:+ o->matchlen = numchars/3;

Is this bug or feature?


  • Does Expect 5.43 behave in the same manner ?

  • Yes, it's same as 5.43.

    Both versions drop whole buffer when full_buffer keyword matched.
    And, there is a little difference which I do not care about this time.
    Because max buffer size is different between 5.43 and 5.44,
    (match_max*2+1 in 5.43, and match_max*3+1 in 5.44)
    prompt-missing is occur in different offset.

    My problem is my logging routine below has potential error.

    expect {
    full_buffer {
    # My logging code here. It manipulates files.
    exp_continue -continue_timer
    PROMPT {
    # also here

    It's avoided by setting huge number to match_max in advance,
    or provide intermediate match pattern (like \n) to clear buffer.
    But then I got bad performance or very specific code.
    I don't have any more ideas.

  • Tom Dexter
    Tom Dexter

    I discovered this old bug after having a very difficult time debugging unpredictable timeout issues in some expect code of my own, which I traced to this very same issue. In my case I was running commands via ssh on a system that does not support sending commands directly, requiring an interactive approach that has to recognize the command prompt.

    Exactly as described here, my timeouts being were caused by cases where by full_buffer happened to occur with the end of the buffer in the middle of the command prompt. In this case, the command prompt pattern did not match, as it had not yet seen the full prompt, and the full_buffer truncated the entire buffer, preventing the command prompt from ever being seen by its pattern.

    Just as described, the only reliable fix I could find was to add an arbitrary pattern that would periodically flush the buffer preventing the full buffers from occurring altogether.

    I'm running expect- under Gentoo and can also verify that a patch like the one described:

    --- expect-  2013-11-21 09:10:53.000000000 -0500
    +++ expect-   2013-11-21 09:11:09.000000000 -0500
    @@ -976,7 +976,7 @@
        /* condition is (size >= max*2/3) <=> (size*3 >= max*2) */
        if (((expSizeGet(esPtr)*3) >= (esPtr->input.max*2)) && (numchars > 0)) {
        o->e = e;
    -       o->matchlen = numchars;
    +       o->matchlen = numchars/3;
            o->matchbuf = str;
        o->esPtr = esPtr;

    ...does in fact prevent this.


  • Jeffrey Hobbs
    Jeffrey Hobbs

    I suspect that TCL_UTF_MAX may be the right value here, but that should be

  • Nothing to do with TCL_UTF_MAX.
    To handle matching across buffer boundaries the buffer is treated like a sliding window. For this purpose it is split into thirds, which only incidentally matches the utf-max.

    At one point the window seems to be sliding too far, dropping data it should not, thus preventing a pattern from matching where it should have.

    The fix looks ok, shortening how much the window slides forward.

    • assigned_to: Andreas Kupries
    • Group: -->
    • status: open --> closed-fixed
  • Fix accepted, and committed.
    Version bumped to 5.45.4.

  • Version 5.45.3, not .4. Apologies.