#551 Stuck Transactions/Memory Leak with B2B top hiding

1.8.x
closed-out-of-date
modules (454)
8
2015-04-07
2012-08-14
Ryan Bullock
No

I am experiencing a memory leak when using the B2B modules with the internal "top hiding" scenario. It appears to be related to struck/un-freed transactions, as the opensips mi will report many thousands of inuse_transactions when the reality is only a handful. An example is we currently show 15000+ inuse_transactions while only a handful of calls are actually in progress. These transactions appear to build up over time.

Opensips information:
version: opensips 1.8.0-notls (x86_64/linux)
flags: STATS: Off, USE_IPV6, USE_TCP, DISABLE_NAGLE, USE_MCAST, SHM_MEM, SHM_MMAP, PKG_MALLOC, DBG_QM_MALLOC, FAST_LOCK-ADAPTIVE_WAIT
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535
poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
svnrevision: 2:9164M
@(#) $Id: main.c 8772 2012-03-08 11:16:13Z bogdan_iancu $
main.c compiled on 12:24:36 Aug 3 2012 with gcc 4.4.6

I currently have the memdump option enabled and am waiting for another buildup of transactions to gather the memdump information. I can supply access to the dump and the opensips configuration file via email.

Discussion

  • Ryan,

    I will try to see if I can reproduce this. You can simple topo hiding scenario, with successful calls, nothing special, right ?

    Regards,
    Bogdan

     
    • priority: 5 --> 8
    • assigned_to: nobody --> bogdan_iancu
     
  • Hello Ryan,

    I cannot seem to reproduce your topology hiding bug.
    Can you privately send me your OpenSIPS configuration file, via email ? Also, access to the memory dump would be useful.

    Regards,
    Vlad

     
  • Nick Altmann
    Nick Altmann
    2012-08-14

    Vlad,

    I also expect problems with memory. Look at screenshot.
    I cannot collect statistics now, but memory leak is there.
    Every day I restart opensips to free memory.

     
  • Ryan Bullock
    Ryan Bullock
    2012-08-14

    Only differences I can think of:
    We use the 'b2bl_from_spec_param'.
    We are storing the b2b states in a database.
    We do modify some headers in the local_route.
    We do sequential routing through the b2b, as in we have a proxy that hunts and sends its requests through the b2b2.

    I will send you the configuration shortly, it is pretty straight forward.

    I will not have the memdump until tonight, when I can force the dump out. Once I have it I will post it someplace where I can get you access and send that via email.

    Is their anything else I can get that would help with troubleshooting this?

     
  • Nick Altmann
    Nick Altmann
    2012-08-14

    > We use the 'b2bl_from_spec_param'.
    I use it too.

    > We are storing the b2b states in a database.
    I store it in pgsql too.

    > We do modify some headers in the local_route.
    I also modify headers in local_route

    > We do sequential routing through the b2b, as in we have a proxy that hunts and sends its requests through the b2b2.
    I'm too.

     
  • Ryan Bullock
    Ryan Bullock
    2012-08-15

    Config file was sent to both Vlad and Bogdan via email earlier today and I just sent login information to obtain the memdump as well. If you did not receive the emails please let me know and I can try re-sending.

    Thanks for the help.

    Regards,

    Ryan

     
  • Jim OBrien
    Jim OBrien
    2012-08-20

    I may be seeing this same behavior as well. I had thought this was related to TLS / many client sessions all connecting at once as things failed over from one server to another. Are you both running TCP? I can see that Ryan is not running TLS from the description above. We use opensips as a proxy; not b2bua, it is doing far end NAT...

    The symptom that I see is that as tm:UAS_transactions increases, so does tm:inuse_transactions, following it very closely. Eventually when this happens, the processes run out of memory / things go bad.

     
  • Just uploaded a patch here - it is not a fix, but rather something to help me with the fix - trying to collect more info.

    Please apply this patch, run your b2b and send me logs (debug 3 is ok). I'm looking for messages on CRITICAL level with "Transaction not replied" + any previous messages from the same process.

    Thanks and regards,

     
  • Ryan Bullock
    Ryan Bullock
    2012-09-24

    Hey Bogdan,

    I tried the patch, but the compiler complained that tm might be uninitialized and it caused a segfault when I tried to run it. I have the core dump if you would like a backtrace.

     
  • First attempt for a fix

     
    Attachments
  • Hi Ryan,

    Just uploaded a patch with a first attempt for a fix - couple you give it a try (with cautious :) )

    Thanks and regards,
    Bogdan

     
  • Ryan Bullock
    Ryan Bullock
    2012-10-12

    Adding the patch to my list of things to test. Might be a bit before I get to it.

     
  • Ryan Bullock
    Ryan Bullock
    2012-10-22

    Sorry for the delay in updating this.

    Still showing memory and transactions climb after this patch, so it seems there are still some leaks present.

    Would a new memory dump be helpful?

    Let me know if there is anything I can do to help.

    Regards,

    Ryan

     
    • status: open --> closed-out-of-date