#344 timeout_process infinite loop and robustness

[Naizhi Li]

I recently hit an infinite loop in timeout_process for the following loop:

  while ((ev = min_heap_top(&base->timeheap))) {
    if (evutil_timercmp(&ev->ev_timeout, &now, >))
      break;

    /* delete this event from the I/O queues */
    event_del_internal(ev);

    event_debug(("timeout_process: call %p",
       ev->ev_callback));
    event_active_nolock(ev, EV_TIMEOUT, 1);
  }

The problem was that the specific event at top of min_heap was missing EVLIST_TIMEOUT flag. Therefore it's not removed from event_del_internal(). After more debugging we found the root cause - a threading issue on our side (not in libevent) - that caused problem with the min_heap and resulted in this inconsistent state.

That said, I think this code can be improved to handle this inconsistent state better. Infinite loops are hard to detect and debug. Instead of relying on event_del_internal() to remove it from the min_heap list, it can be popped out directly from here. i.e.:

  while ((ev = min_heap_top(&base->timeheap))) {
    if (evutil_timercmp(&ev->ev_timeout, &now, >))
      break;

    /* pop it from the heap */
    min_heap_pop(&base->timeheap);
    ev->ev_flags &= ~EVLIST_TIMEOUT;

    /* delete this event from the I/O queues */
    event_del_internal(ev);

    event_debug(("timeout_process: call %p",
       ev->ev_callback));
    event_active_nolock(ev, EV_TIMEOUT, 1);
  }

It is also slightly more performant since min_heap_pop() does less checks than min_heap_erase().

Thanks,
Naizhi

[Asgari Golwala]