#57 Connect/read race-condition

API
open
nobody
General (24)
5
2015-07-10
2011-02-13
Anonymous
No

It's visible only on local connections, on services that uses welcome banners (like SMTP).

Connecting bufferevent sometimes raise readcb first and then eventcb.

Tested on: libevent v2.0.10

Discussion

  • Nick Mathewson

    Nick Mathewson - 2011-02-13

    Using what OS? What backend? Creating the bufferevent with what flags? If windows, using IOCP?

    Do you have a test program to reproduce this with?

     
  • Nobody/Anonymous

    Linux 2.6.x on x86_64. I don't know how to check which backend was used (I'm new to libevent), but event_get_supported_methods gives three: epoll, poll and select (in this order).

    # gcc race.c -o race -Wall -levent_core -levent_extra
    # ./race
    Connected.
    220 localhost ESMTP Exim
    # ./race
    220 localhost ESMTP Exim
    Connected.
    # ./race
    220 localhost ESMTP Exim
    Connected.

    test program:

    #include <event2/dns.h>
    #include <event2/bufferevent.h>
    #include <event2/buffer.h>
    #include <event2/util.h>
    #include <event2/event.h>
    #include <stdio.h>

    void readcb(struct bufferevent *bev, void *ptr)
    {
    char buf[1024];
    int n;
    struct evbuffer *input = bufferevent_get_input(bev);
    while ((n = evbuffer_remove(input, buf, sizeof(buf))) > 0) {
    fwrite(buf, 1, n, stdout);
    }
    }

    void eventcb(struct bufferevent *bev, short events, void *ptr)
    {
    if (events & BEV_EVENT_CONNECTED) {
    printf("Connected.\n");
    }
    }

    int main(int argc, char **argv)
    {
    struct event_base *base;
    struct event_dns_base *dns_base;

    base = event_base_new();
    dns_base = evdns_base_new(base, 1);

    struct bufferevent *bev;

    bev = bufferevent_socket_new(base, -1, BEV_OPT_CLOSE_ON_FREE);
    bufferevent_setcb(bev, readcb, NULL, eventcb, base);
    bufferevent_enable(bev, EV_READ|EV_WRITE);
    bufferevent_socket_connect_hostname(bev, dns_base, AF_INET, "localhost", 25);

    event_base_dispatch(base);
    return (0);
    }

     
  • Nick Mathewson

    Nick Mathewson - 2011-02-13

    Okay, so unless you're doing something unusual, you're getting the epoll backend (it's listed first).

    The likeliest reason you'd see this behavior, as near as I can tell, is this: when you're connecting, the bufferevent with EV_READ and EV_WRITE enabled is listening for read and write events via the libevent core interface. If the TCP finishes the connection and data arrives quickly, then the socket will be readable and writeable at the same time. (This is where the race happens: does the data arrive at the kernel before or after the kernel has a chance to tell Libevent that the socket is writeable and hence connected?)

    When the same fd is both readable and writeable, Libevent doesn't define the order in which the read and write callbacks get invoked. So if the read callback is invokved first, the bufferevent's bufferevent_readcb function will read data and call your read callback first, and if the write callback bufferevent_writecb is invoked first, your connected callback will get called first.

    For 2.0 stable, I think the best solution here is to document the behavior. It's undesirable, but there should be an easy workaround: If you don't want a read callback to get triggered before the connect event callback, don't do bufferevent_enable(bev, EV_READ) until the connection is complete.

    For 2.1, I think we should fix the behavior. The right fix is probably something like checking whether the "connecting" flag is set when we get a call to bufferevent_readcb, and if so handling the connected event from bufferevent_readcb. (Or something like that.)

     
  • Sam Liddicott

    Sam Liddicott - 2015-07-10

    I can re-produce this using libevent 2.0.22, with this as the server:

    $ while date | nc -l 9999 ; do echo . DONE ; sleep 1 ; done

    this as the client:

    $ while sleep 2 ; do nc localhost 10000 ; echo -n . ; done

    and this as the invocation of sslsplit:

    $ sslsplit -D tcp 127.0.0.1 10000 127.0.0.1 9999

    If I further modify sslsplit so that EV_READ is disabled until after the connection callback, then I find I get another error which is that sometimes the write callback occurs before the connection callback. (Maybe once in a few hundred times)

    This early writecb callback also occurs on a different thread than the one performing the connect.

    I think this is because the localhost connect is quick enough to complete in bufferevent_sock.c AFTER calling evutil_socket_connect() but before setting bufev_p->connecting.

    I think the main mistake is to call bufferevent_setfd(bev, fd); before setting bufev_p->connecting, as this allows the write callback to occur (on another thread) and not catch the connecting flag.

    A more surprising side effect is that the write callback (or event callback if we fix this) will occur on another thread BEFORE the connecting threads call to bufferevent_socket_connect() even returns.

    Of course this can be justified, and the caller should be sure that all other contexts used by the callback are established before calling bufferevent_socket_connect().

    However this early callback occurs even if BEV_OPT_DEFER_CALLBACKS was set which certainly adds to the surprise. I can see how in that case the author might not expect the callback to occur until AFTER the callback currently being executed returns to the event loop (even if a different thread is then used).

    It also leaves a difficulty where the callback of the new connection is using the same context structs as the callback that made the new connection, the author now needs to manage locking on those during setup phase.

    I'm also concerned about pointer aliasing in bufferevent_socket_connect() as bev and bufev_p point to the same struct. Has any thought been given to that, possibly it is a cause of some of this trouble?

    (I suppose it could actually be this same mechanism by which the read callbacks were happening early too, if on another thread).

     
    Last edit: Sam Liddicott 2015-07-10

Get latest updates about Open Source Projects, Conferences and News.

Sign up for the SourceForge newsletter:





No, thanks