#184 evhttp_error_cb causing SIGABRT

[Jason Toffaletti]

I have an http proxy server that uses evhttp to accept and proxy http requests to several backends. The server will randomly die with SIGABRT. I finally caught one of these in gdb, this is the backtrace:

(gdb) bt
#0 0x00007fe94b8aa4b5 in raise () from /lib/libc.so.6
#1 0x00007fe94b8adf50 in abort () from /lib/libc.so.6
#2 0x00000000004f2235 in event_exit (errcode=16467) at log.c:79
#3 0x00000000004f22da in event_errx (eval=-559030611, fmt=<value optimized out>) at log.c:136
#4 0x00000000004f99dd in evhttp_error_cb (bufev=<value optimized out>, what=<value optimized out>, arg=<value optimized out>) at http.c:1153
#5 0x0000000000502e8e in bufferevent_readcb (fd=19, event=1, arg=0x7fe940004520) at bufferevent_sock.c:191
#6 0x00000000004e8ad8 in event_process_active_single_queue (base=0x1bdd950, flags=<value optimized out>) at event.c:1288
#7 event_process_active (base=0x1bdd950, flags=<value optimized out>) at event.c:1355
#8 event_base_loop (base=0x1bdd950, flags=<value optimized out>) at event.c:1550

I use evhttp_connection_set_timeout to set a 10 second timeout for the outgoing connections to the backend.

[Jason Toffaletti]

I think this might be my fault. I was mistakenly setting Connection: close header on the backend connections and then trying to reuse them.

[Jason Toffaletti]

removing Connection: close header didn't help.

[Jason Toffaletti]

I examined the code that was being "protected" by the assert and decided it wouldn't be dangerous to execute it. This patch seems to have fixed my SIGABRT problems:

diff --git a/import/libevent-2.0.7-rc/http.c b/import/libevent-2.0.7-rc/http.c
index df6b103..a2d4bdf 100644
--- a/import/libevent-2.0.7-rc/http.c
+++ b/import/libevent-2.0.7-rc/http.c
@@ -1150,9 +1150,11 @@ evhttp_error_cb(struct bufferevent *bufev, short what, void *arg)
* reset the connection so that it becomes
* disconnected.
*/
- EVUTIL_ASSERT(evcon->state == EVCON_IDLE);
- evhttp_connection_reset(evcon);
- return;
+ /*EVUTIL_ASSERT(evcon->state == EVCON_IDLE);*/
+ if (evcon->state == EVCON_IDLE) {
+ evhttp_connection_reset(evcon);
+ return;
+ }
}

if (what & BEV_EVENT_TIMEOUT) {

[Nick Mathewson]

Hm. I'm trying to reproduce this, but not having a lot of luck. Can you write a unit test?

I'm also trying to figure out why we added that assert in the first place. Some forensics to figure out what the reason was and why it no longer applies would be awesome.

[Nick Mathewson]

This might be fixed in 2.0.9-rc by commit 0faaa395921e3da1da52 . Please try it out?

[Jason Toffaletti]

I'll try to get to this soon, I've been on vacation for three weeks.

[Jason Toffaletti]

I've upgraded to 2.0.9-rc and I have not seen this reproduced *yet* however it is difficult to be certain right now because my app is exiting from "http.c:685: Assertion req != NULL failed in evhttp_connection_fail" as reported in:

https://sourceforge.net/tracker/index.php?func=detail&aid=3078187&group_id=50884&atid=461324

[Leonid Evdokimov]

Ok, it seems, that I see same issue in libevent-2.0.12

Assertion:
libevent: http.c:1366: Assertion evcon->state == EVCON_IDLE failed in evhttp_error_cb

Backtrace:

#0 0x0000000032d1fa3c in kill () from /lib/libc.so.7
No symbol table info available.
#1 0x0000000032d1e888 in abort () from /lib/libc.so.7
No symbol table info available.
#2 0x00000000326ce04c in event_exit () from /usr/local/lib/event2/libevent-2.0.so.6
No symbol table info available.
#3 0x00000000326ce0fc in event_errx () from /usr/local/lib/event2/libevent-2.0.so.6
No symbol table info available.
#4 0x00000000326d85bd in evhttp_error_cb () from /usr/local/lib/event2/libevent-2.0.so.6
No symbol table info available.
#5 0x00000000326d85ff in evhttp_connection_cb () from /usr/local/lib/event2/libevent-2.0.so.6
No symbol table info available.
#6 0x00000000326c8f81 in bufferevent_writecb () from /usr/local/lib/event2/libevent-2.0.so.6
No symbol table info available.
#7 0x00000000326c11eb in event_base_loop () from /usr/local/lib/event2/libevent-2.0.so.6
No symbol table info available.
#8 0x0000000000457d75 in mainloop (evbase=0x33007300, moirae=@0x7fffffffdfd0) at src/main.cc:153
__PRETTY_FUNCTION__ = "void<unnamed>::mainloop(event_base*, Shutdown&)"

I have full core file, but I have no symbols (although the library is not stripped), so gathering information from the core is real PITA.
Please, tell me what information from the core do you need?