Stuck Transactions/Memory Leak with B2B top hiding

Brought to you by: bogdan_iancu, dan_pascu, liviuchircu, razvancrainea, vladut-paiu

#551 Stuck Transactions/Memory Leak with B2B top hiding

Milestone: 1.8.x

Status: closed-out-of-date

Owner: Bogdan-Andrei Iancu

Labels: modules (454)

Priority: 8

Updated: 2015-04-07

Created: 2012-08-14

Creator: Ryan Bullock

Private: No

I am experiencing a memory leak when using the B2B modules with the internal "top hiding" scenario. It appears to be related to struck/un-freed transactions, as the opensips mi will report many thousands of inuse_transactions when the reality is only a handful. An example is we currently show 15000+ inuse_transactions while only a handful of calls are actually in progress. These transactions appear to build up over time.

Opensips information:
version: opensips 1.8.0-notls (x86_64/linux)
flags: STATS: Off, USE_IPV6, USE_TCP, DISABLE_NAGLE, USE_MCAST, SHM_MEM, SHM_MMAP, PKG_MALLOC, DBG_QM_MALLOC, FAST_LOCK-ADAPTIVE_WAIT
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535
poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
svnrevision: 2:9164M
@(#) $Id: main.c 8772 2012-03-08 11:16:13Z bogdan_iancu $
main.c compiled on 12:24:36 Aug 3 2012 with gcc 4.4.6

I currently have the memdump option enabled and am waiting for another buildup of transactions to gather the memdump information. I can supply access to the dump and the opensips configuration file via email.

Discussion

Bogdan-Andrei Iancu - 2012-08-14

Ryan,

I will try to see if I can reproduce this. You can simple topo hiding scenario, with successful calls, nothing special, right ?

Regards,
Bogdan

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Bogdan-Andrei Iancu - 2012-08-14

priority: 5 --> 8

assigned_to: nobody --> bogdan_iancu
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Vladut-Stefan Paiu - 2012-08-14

Hello Ryan,

I cannot seem to reproduce your topology hiding bug.
Can you privately send me your OpenSIPS configuration file, via email ? Also, access to the memory dump would be useful.

Regards,
Vlad

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Nick Altmann - 2012-08-14

Vlad,

I also expect problems with memory. Look at screenshot.
I cannot collect statistics now, but memory leak is there.
Every day I restart opensips to free memory.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Nick Altmann - 2012-08-14

Cannot attach file. Look it here:
http://img39.imageshack.us/img39/5275/memorys.png

Also just simple topo hiding, nothing special.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Ryan Bullock - 2012-08-14

Only differences I can think of:
We use the 'b2bl_from_spec_param'.
We are storing the b2b states in a database.
We do modify some headers in the local_route.
We do sequential routing through the b2b, as in we have a proxy that hunts and sends its requests through the b2b2.

I will send you the configuration shortly, it is pretty straight forward.

I will not have the memdump until tonight, when I can force the dump out. Once I have it I will post it someplace where I can get you access and send that via email.

Is their anything else I can get that would help with troubleshooting this?

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Nick Altmann - 2012-08-14

> We use the 'b2bl_from_spec_param'.
I use it too.

> We are storing the b2b states in a database.
I store it in pgsql too.

> We do modify some headers in the local_route.
I also modify headers in local_route

> We do sequential routing through the b2b, as in we have a proxy that hunts and sends its requests through the b2b2.
I'm too.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Ryan Bullock - 2012-08-15

Config file was sent to both Vlad and Bogdan via email earlier today and I just sent login information to obtain the memdump as well. If you did not receive the emails please let me know and I can try re-sending.

Thanks for the help.

Regards,

Ryan

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Jim OBrien - 2012-08-20

I may be seeing this same behavior as well. I had thought this was related to TLS / many client sessions all connecting at once as things failed over from one server to another. Are you both running TCP? I can see that Ryan is not running TLS from the description above. We use opensips as a proxy; not b2bua, it is doing far end NAT...

The symptom that I see is that as tm:UAS_transactions increases, so does tm:inuse_transactions, following it very closely. Eventually when this happens, the processes run out of memory / things go bad.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Bogdan-Andrei Iancu - 2012-09-19

Just uploaded a patch here - it is not a fix, but rather something to help me with the fix - trying to collect more info.

Please apply this patch, run your b2b and send me logs (debug 3 is ok). I'm looking for messages on CRITICAL level with "Transaction not replied" + any previous messages from the same process.

Thanks and regards,

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Ryan Bullock - 2012-09-24

Hey Bogdan,

I tried the patch, but the compiler complained that tm might be uninitialized and it caused a segfault when I tried to run it. I have the core dump if you would like a backtrace.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Bogdan-Andrei Iancu - 2012-10-01

First attempt for a fix

b2b_leak.patch

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Bogdan-Andrei Iancu - 2012-10-01

Hi Ryan,

Just uploaded a patch with a first attempt for a fix - couple you give it a try (with cautious :) )

Thanks and regards,
Bogdan

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Ryan Bullock - 2012-10-12

Adding the patch to my list of things to test. Might be a bit before I get to it.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Ryan Bullock - 2012-10-22

Sorry for the delay in updating this.

Still showing memory and transactions climb after this patch, so it seems there are still some leaks present.

Would a new memory dump be helpful?

Let me know if there is anything I can do to help.

Regards,

Ryan

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Bogdan-Andrei Iancu - 2015-04-07

status: open --> closed-out-of-date
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Log in to post a comment.