Learn how easy it is to sync an existing GitHub or Google Code repo to a SourceForge project! See Demo

Close

#328 mem leak

1.6.x
closed-out-of-date
core (110)
7
2015-04-06
2010-12-02
thrillerbee
No

Errors in the syslog this time right before the crash:
/usr/local/sbin/opensips[27044]: ERROR:db_flatstore:new_flat_id: no pkg memory left
kernel: [5508366.582447] opensips[27044]: segfault at 10 ip 7fa7ff74c21f sp 7fffdc101700 error 4 in db_flatstore.so[7fa7ff749000+5000]

I do use the flat_rotate MI command with the acc module for cdr collection. The backtrace is attached.

mem dumps are here:
http://www.starviewconnect.com/tmp/core1_mem_dump_20101112.gz
http://www.starviewconnect.com/tmp/core2_mem_dump_20101112.gz

Discussion

  • Hi,

    Indeed, the logs show a leak and I suspect it is from db_flatstore module - unfortunately you do not have properly compiled the memory debugging support, so it hard to pinpoint the guilty code.
    I tried to reproduce ACC via flatstore, but not able to get a leak at all...

    So, please compile the memory debugger as per http://www.opensips.org/Resources/DocsTsMem (be sure you put DBG_QM_MALLOC and take out F_MALLOC ) and recompile + reinstall + restart opensips.

    In your cfg use:
    memlog=10
    memdump=1
    This will generate only a memory dump without any additional memory logs at rutime.

    You can take a mem dump at runtime by sending a SIGUSR1 signal to an opensips process,
    Regards,
    Bogdan

     
    • priority: 5 --> 7
    • assigned_to: nobody --> bogdan_iancu
    • status: open --> open-accepted
     
  • Your opensips is still not compiled with mem debugging :(...The dump from memory debugger looks like:
    0(17665) Memory status (shm):
    0(17665) qm_status (0xb5a7e000):
    0(17665) heap size= 33554432
    0(17665) used= 1592952, used+overhead=1811564, free=31742868
    0(17665) max used (+overhead)= 1811564
    0(17665) dumping all alloc'ed. fragments:
    0(17665) 0. N address=0xb5ab240c frag=0xb5ab23f4 size=4 used=1
    0(17665) alloc'd from mem/shm_mem.c: shm_mem_init_mallocs(199)
    0(17665) start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
    0(17665) 1. N address=0xb5ab2440 frag=0xb5ab2428 size=4 used=1
    0(17665) alloc'd from timer.c: init_timer(52)
    0(17665) start check=f0f0f0f0, end check= c0c0c0c0, abcdefed

    Just to check the flags you use, please upload your Makefile.defs and post the output of "opensips -V" .

    Thanks and regards,
    Bogdan

     
  • thrillerbee
    thrillerbee
    2010-12-14

    Makefile.defs is attached

    opensips -V
    version: opensips 1.6.3-notls (x86_64/linux)
    flags: STATS: Off, USE_IPV6, USE_TCP, DISABLE_NAGLE, USE_MCAST, SHM_MEM, SHM_MMAP, PKG_MALLOC, FAST_LOCK-ADAPTIVE_WAIT
    ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535
    poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
    svnrevision: 2:7290M
    @(#) $Id: main.c 6169 2009-09-22 12:48:37Z bogdan_iancu $
    main.c compiled on 22:47:42 Dec 8 2010 with gcc 4.3.2

    Relevant config:
    debug=2
    memlog=10
    memdump=1

     
  • fixed makefile.defs

     
    Attachments
  • your makfile.defs was not correct (makefile syntax errors) - use the attached one (I fixed the file you uploaded).

     
  • thrillerbee
    thrillerbee
    2010-12-21

    /usr/local/sbin/opensips[13045]: Memory status (shm):
    /usr/local/sbin/opensips[13045]: qm_status (0x7f1c48c82000):
    /usr/local/sbin/opensips[13045]: heap size= 536870912
    /usr/local/sbin/opensips[13045]: used= 13760, used+overhead=21459048, free=515411864
    /usr/local/sbin/opensips[13045]: max used (+overhead)= 75080016
    /usr/local/sbin/opensips[13045]: dumping all alloc'ed. fragments:
    /usr/local/sbin/opensips[13045]: 0. N address=0x7f1c48cb7518 frag=0x7f1c48cb74e8 size=8 used=1
    /usr/local/sbin/opensips[13045]: alloc'd from mem/shm_mem.c: shm_mem_init_mallocs(196)
    /usr/local/sbin/opensips[13045]: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
    /usr/local/sbin/opensips[13045]: 396. N address=0x7f1c48f682b0 frag=0x7f1c48f68280 size=40 used=1
    /usr/local/sbin/opensips[13045]: alloc'd from lb_data.c: load_lb_data(52)
    /usr/local/sbin/opensips[13045]: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
    /usr/local/sbin/opensips[13045]: 536. N address=0x7f1c48f6d708 frag=0x7f1c48f6d6d8 size=6192 used=1
    /usr/local/sbin/opensips[13045]: alloc'd from hash.c: new_subnet_table(276)
    /usr/local/sbin/opensips[13045]: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
    /usr/local/sbin/opensips[13045]: 537. N address=0x7f1c48f6ef98 frag=0x7f1c48f6ef68 size=6192 used=1
    /usr/local/sbin/opensips[13045]: alloc'd from hash.c: new_subnet_table(276)
    /usr/local/sbin/opensips[13045]: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
    /usr/local/sbin/opensips[13045]: 538. N address=0x7f1c48f70828 frag=0x7f1c48f707f8 size=8 used=1
    /usr/local/sbin/opensips[13045]: alloc'd from address.c: init_address(320)
    /usr/local/sbin/opensips[13045]: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
    /usr/local/sbin/opensips[13045]: 539. N address=0x7f1c48f70890 frag=0x7f1c48f70860 size=48 used=1
    /usr/local/sbin/opensips[13045]: alloc'd from hash.c: subnet_table_insert(318)
    /usr/local/sbin/opensips[13045]: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
    /usr/local/sbin/opensips[13045]: 540. N address=0x7f1c48f70920 frag=0x7f1c48f708f0 size=8 used=1
    /usr/local/sbin/opensips[13045]: alloc'd from hash.c: subnet_table_insert(329)
    /usr/local/sbin/opensips[13045]: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
    /usr/local/sbin/opensips[13045]: 586. N address=0x7f1c48f72ce8 frag=0x7f1c48f72cb8 size=960 used=1
    /usr/local/sbin/opensips[13045]: alloc'd from core_stats.c: init_pkg_stats(182)
    /usr/local/sbin/opensips[13045]: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
    /usr/local/sbin/opensips[13045]: 587. N address=0x7f1c48f73108 frag=0x7f1c48f730d8 size=160 used=1
    /usr/local/sbin/opensips[13045]: alloc'd from core_stats.c: init_pkg_stats(183)
    /usr/local/sbin/opensips[13045]: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
    /usr/local/sbin/opensips[13045]: 81089. N address=0x7f1c4a0cdc60 frag=0x7f1c4a0cdc30 size=24 used=1
    /usr/local/sbin/opensips[13045]: alloc'd from proxy.c: hostent_shm_cpy(83)
    /usr/local/sbin/opensips[13045]: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
    /usr/local/sbin/opensips[13045]: 128616. N address=0x7f1c4b2f1c90 frag=0x7f1c4b2f1c60 size=72 used=1
    /usr/local/sbin/opensips[13045]: alloc'd from t_fwd.c: shm_clone_proxy(233)
    /usr/local/sbin/opensips[13045]: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
    /usr/local/sbin/opensips[13045]: 176525. N address=0x7f1c4c8317e8 frag=0x7f1c4c8317b8 size=16 used=1
    /usr/local/sbin/opensips[13045]: alloc'd from mem/shm_mem.c: _shm_resize(112)
    /usr/local/sbin/opensips[13045]: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
    /usr/local/sbin/opensips[13045]: 201764. N address=0x7f1c4cdffb50 frag=0x7f1c4cdffb20 size=16 used=1
    /usr/local/sbin/opensips[13045]: alloc'd from mem/shm_mem.c: _shm_resize(112)
    /usr/local/sbin/opensips[13045]: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
    /usr/local/sbin/opensips[13045]: 214714. N address=0x7f1c4d06ed58 frag=0x7f1c4d06ed28 size=16 used=1
    /usr/local/sbin/opensips[13045]: alloc'd from mem/shm_mem.c: _shm_resize(112)
    /usr/local/sbin/opensips[13045]: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed

     
  • thrillerbee
    thrillerbee
    2010-12-21

    my apologies for the previous comment; here it is cleaned up:

    Memory status (shm):
    qm_status (0x7f1c48c82000):
    heap size= 536870912
    used= 13760, used+overhead=21459048, free=515411864
    max used (+overhead)= 75080016
    dumping all alloc'ed. fragments:
    0. N address=0x7f1c48cb7518 frag=0x7f1c48cb74e8 size=8 used=1
    alloc'd from mem/shm_mem.c: shm_mem_init_mallocs(196)
    start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
    396. N address=0x7f1c48f682b0 frag=0x7f1c48f68280 size=40 used=1
    alloc'd from lb_data.c: load_lb_data(52)
    start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
    536. N address=0x7f1c48f6d708 frag=0x7f1c48f6d6d8 size=6192 used=1
    alloc'd from hash.c: new_subnet_table(276)
    start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
    537. N address=0x7f1c48f6ef98 frag=0x7f1c48f6ef68 size=6192 used=1
    alloc'd from hash.c: new_subnet_table(276)
    start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
    538. N address=0x7f1c48f70828 frag=0x7f1c48f707f8 size=8 used=1
    alloc'd from address.c: init_address(320)
    start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
    539. N address=0x7f1c48f70890 frag=0x7f1c48f70860 size=48 used=1
    alloc'd from hash.c: subnet_table_insert(318)
    start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
    540. N address=0x7f1c48f70920 frag=0x7f1c48f708f0 size=8 used=1
    alloc'd from hash.c: subnet_table_insert(329)
    start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
    586. N address=0x7f1c48f72ce8 frag=0x7f1c48f72cb8 size=960 used=1
    alloc'd from core_stats.c: init_pkg_stats(182)
    start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
    587. N address=0x7f1c48f73108 frag=0x7f1c48f730d8 size=160 used=1
    alloc'd from core_stats.c: init_pkg_stats(183)
    start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
    81089. N address=0x7f1c4a0cdc60 frag=0x7f1c4a0cdc30 size=24 used=1
    alloc'd from proxy.c: hostent_shm_cpy(83)
    start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
    128616. N address=0x7f1c4b2f1c90 frag=0x7f1c4b2f1c60 size=72 used=1
    alloc'd from t_fwd.c: shm_clone_proxy(233)
    start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
    176525. N address=0x7f1c4c8317e8 frag=0x7f1c4c8317b8 size=16 used=1
    alloc'd from mem/shm_mem.c: _shm_resize(112)
    start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
    201764. N address=0x7f1c4cdffb50 frag=0x7f1c4cdffb20 size=16 used=1
    alloc'd from mem/shm_mem.c: _shm_resize(112)
    start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
    214714. N address=0x7f1c4d06ed58 frag=0x7f1c4d06ed28 size=16 used=1
    alloc'd from mem/shm_mem.c: _shm_resize(112)
    start check=f0f0f0f0, end check= c0c0c0c0, abcdefed

     
  • Is this a runtime dump or a shutdown one ?

    Regards,
    Bogdan

     
  • thrillerbee
    thrillerbee
    2010-12-22

    The server crashed (so, on shutdown).

     
  • a shutdown dump is not relevant - try to make a runtime dump by sending SIGUSR1 signal to opensips - wait for couple of hours of run and make the dump.

    Thanks and regards,
    Bogdan

     
  • thrillerbee
    thrillerbee
    2010-12-28

    Bogdan,

    Here's a new memdump after sending SIGUSR1 to opensips while it was out of memory and failing to route calls (so, during runtime): http://www.starviewconnect.com/tmp/mem_dump_20101228.zip

    Thanks!

     
  • Please let me know if I can provide any further information. This is a critical issue for my network. Thanks.

     
  • Hi,

    What is the status on this ?

    Regards,
    Bogdan

     
    • status: open-accepted --> closed-out-of-date