#48 seg fault in db_postgres

trunk
closed-out-of-date
modules (454)
2
2009-04-13
2008-11-13
Brent
No

Partial output:

*** glibc detected *** opensips: free(): invalid next size (fast): 0x0000000001166c70 ***
======= Backtrace: =========
/lib64/libc.so.6[0x2b92149b8684]
/lib64/libc.so.6(cfree+0x8c)[0x2b92149bbccc]
/usr/lib64/libpq.so.5(PQsendQuery+0x69)[0x2b921642a819]
/usr/lib64/opensips/modules/db_postgres.so[0x2b921620f0df]
opensips(db_do_query+0x335)[0x4b0da1]
/usr/lib64/opensips/modules/db_postgres.so(db_postgres_query+0x8a)[0x2b921620fd59]
/usr/lib64/opensips/modules/auth_db.so[0x2b92181ae5f1]
/usr/lib64/opensips/modules/auth_db.so[0x2b92181ae0f1]
/usr/lib64/opensips/modules/auth_db.so(proxy_authorize+0x2a)[0x2b92181adec8]
opensips(do_action+0x24d4)[0x410de0]
opensips(run_action_list+0x32)[0x40e1fa]
opensips[0x44d7aa]
opensips(eval_expr+0xc8)[0x451633]
opensips(eval_expr+0x1ac)[0x451717]
opensips(eval_expr+0x1e0)[0x45174b]
opensips(do_action+0x1caf)[0x4105bb]
opensips(run_action_list+0x32)[0x40e1fa]
opensips(do_action+0x1de0)[0x4106ec]
opensips(run_action_list+0x32)[0x40e1fa]
opensips[0x40e46e]
opensips(do_action+0x103b)[0x40f947]
opensips(run_action_list+0x32)[0x40e1fa]
opensips(do_action+0x1de0)[0x4106ec]
opensips(run_action_list+0x32)[0x40e1fa]
opensips[0x40e46e]
opensips(run_top_route+0x61)[0x40e526]
opensips(receive_msg+0x51f)[0x445da6]
opensips(udp_rcv_loop+0x45c)[0x47af5b]
opensips[0x4219ad]
opensips(main+0x1c74)[0x423ccf]
/lib64/libc.so.6(__libc_start_main+0xf4)[0x2b92149648b4]
opensips[0x40e119]

CentOS 5.2
OpenSIPS: latest from svn (about an hour ago)
PostgreSQL libs: 8.3.4 (built against the same)
DB in writeback mode

Causes OpenSIPS to crash every time within a few minutes of starting. I can reproduce the crash by repeatedly reregistering a hard or soft phone, making a call from a successfully registered device, or anything else that touches the database. Reference build and config is OpenSER 1.2, which does not have the problem.

The core dump is too large to attach so I've posted it here:

http://www.sendspace.com/file/1pv989

Discussion

  • Nobody/Anonymous

    After fiddling with this some more, it appears that I can only reproduce this consistently if fork=yes in opensips.cfg. If I don't daemonize opensips (run it bare, rather than through the init script) and let all the output go to the console, it no longer crashes.

    Please let me know if there's any other useful information that I can provide.

     
  • Sergio Gutierrez

    Hello Brent.

    Is that backtrace taken with gdb? The syntax does not loke familiar.
    In case it does not, could you take the backtrace with gdb?

    Regards.

    Sergio.

     
  • Brent

    Brent - 2008-11-18

    That backtrace is the output from opensips itself, with debugging set to 9. I'm not a gdb jedi. Can you give me instructions or point me to what I need to do to generate what you're asking for? Is it just a matter of running the binary in the debugger and then grabbing the output when it crashes?

     
  • Brent

    Brent - 2008-11-18

    Is this what you need?

    (gdb) bt
    #0 0x00002b9e74dc7155 in raise () from /lib64/libc.so.6
    #1 0x00002b9e74dc8bf0 in abort () from /lib64/libc.so.6
    #2 0x00002b9e74e013db in __libc_message () from /lib64/libc.so.6
    #3 0x00002b9e74e08684 in _int_free () from /lib64/libc.so.6
    #4 0x00002b9e74e0bccc in free () from /lib64/libc.so.6
    #5 0x00002b9e768797e9 in PQclear () from /usr/lib64/libpq.so.5
    #6 0x00002b9e7665fbe9 in free_query (_con=0x779fe8) at dbase.c:320
    #7 0x00002b9e7666037d in db_postgres_store_result (_con=0x779fe8, _r=0x7fff368d58b8)
    at dbase.c:475
    #8 0x00000000004b0634 in db_do_query ()
    #9 0x00002b9e7665fcc9 in db_postgres_query (_h=0x779fe8, _k=0x7fff368d5830, _op=0x0,
    _v=0x7fff368d57f0, _c=0x7802e0, _n=1, _nc=2, _o=0x0, _r=0x7fff368d58b8) at dbase.c:363
    #10 0x00002b9e785fd5b1 in get_ha1 (_username=0x780948, _domain=0x7fff368d58d0,
    _table=0x7fff368d58c0, _ha1=0x7fff368d58f0 "", res=0x7fff368d58b8) at authorize.c:101
    #11 0x00002b9e785fd0b1 in authorize (_m=0x77f328, _realm=0x779f30, _table=0x7737a8 "subscriber",
    _hftype=HDR_AUTHORIZATION_T) at authorize.c:244
    #12 0x00002b9e785fd9fd in www_authorize (_m=0x77f328, _realm=0x779f30 "\001",
    _table=0x7737a8 "subscriber") at authorize.c:287
    #13 0x00000000004105f0 in do_action ()
    #14 0x000000000040da0a in run_action_list ()
    #15 0x000000000044cfba in eval_elem ()
    #16 0x0000000000450e43 in eval_expr ()
    #17 0x0000000000450f27 in eval_expr ()
    #18 0x0000000000450f5b in eval_expr ()
    #19 0x000000000040fdcb in do_action ()
    #20 0x000000000040da0a in run_action_list ()
    #21 0x000000000040dc7e in run_actions ()
    #22 0x000000000040f157 in do_action ()
    #23 0x000000000040da0a in run_action_list ()
    #24 0x000000000044cfba in eval_elem ()
    #25 0x0000000000450e43 in eval_expr ()
    #26 0x0000000000450f5b in eval_expr ()
    #27 0x000000000040fdcb in do_action ()
    #28 0x000000000040da0a in run_action_list ()
    #29 0x000000000040fefc in do_action ()
    #30 0x000000000040da0a in run_action_list ()
    #31 0x000000000040dc7e in run_actions ()
    #32 0x000000000040dd36 in run_top_route ()
    #33 0x00000000004455b6 in receive_msg ()
    #34 0x000000000047a76b in udp_rcv_loop ()
    #35 0x0000000000421501 in main_loop ()
    #36 0x00000000004234df in main ()

     
  • Sergio Gutierrez

    Hello Brent.

    I was reviewing your report. Please, could you post OpenSIPS log before Segmentation Fault?

    Regards.

    Sergio.

     
  • Brent

    Brent - 2009-02-13

    Hmmm ... That was some time ago. Let me see if I can reproduce it and put up a log file.

     
  • Nobody/Anonymous

    Brent, do you feel like getting the patch from issue 2593088 and to try with it? It seems like the same issue.

    Keep in mind that it will make a ton of connections to the database (in my case they were 70).

     
  • Brent

    Brent - 2009-02-13

    Doesn't really sound like a fix ... and 70 connections isn't acceptable. I think I'll look for things that have changed since openser 1.3.2, which doesn't have this issue.

     
  • Bogdan-Andrei Iancu

    • assigned_to: nobody --> bogdan_iancu
     
  • Bogdan-Andrei Iancu

    hi brent,

    there were some fixes in 1.5 over the postgres module - can you update and test ?

    or, is this still happening ?

    regards,
    bogdan

     
  • Brent

    Brent - 2009-03-16

    I haven't tried it in a while--just been running 1.3.2. I'll give the latest version a try when I have a second.

     
  • Bogdan-Andrei Iancu

    • priority: 5 --> 2
     
  • Brent

    Brent - 2009-04-12

    I've been running 1.5 with the setup that produced the crash consistently before for about an hour now without incident. That's not long, but 1.4 was crashing within a few minutes.

    I'll follow up if the problem surfaces again.

     
  • Bogdan-Andrei Iancu

    All fixes that were done on postgres for 1.5.0 were also ported on 1.4 branch, so if you update to 1.4.5 (latest on 1.4 branch) it should be also fixed.

    Regards,
    Bogdan

     
  • Bogdan-Andrei Iancu

    • status: open --> closed-out-of-date
     

Get latest updates about Open Source Projects, Conferences and News.

Sign up for the SourceForge newsletter:





No, thanks