#13 continous segfaults

0.2
closed-fixed
Aex Aey
segfault (1)
5
2014-12-10
2014-01-24
avimar
No

Yesterday, pcapsipdump would keep segfaulting. I've been running it for months with only occasional segfaults. I think it was version 0.14. I really have no idea what "changed".

I'm trying v.2 from the tar file, but it's the same issues.
I thought it might be because I was writing to a sshfs folder, but even writing locally it's still doing this. Any suggestions?

pcapsipdump version 0.2

uname -a
Linux sip1 3.8.4-x86_64-linode31 #1 SMP Mon Mar 25 16:00:34 EDT 2013 x86_64 GNU/Linux

lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 10.04.4 LTS
Release: 10.04
Codename: lucid

grep pcap /var/log/syslog

Jan 24 08:53:24 sip1 kernel: pcapsipdump[3664]: segfault at 7f62eb37e000 ip 0000000000402758 sp 00007fff321be060 error 6 in pcapsipdump[400000+5000]
Jan 24 08:57:43 sip1 kernel: pcapsipdump[4924]: segfault at 7facfdb34000 ip 0000000000402758 sp 00007fff675cdbb0 error 6 in pcapsipdump[400000+5000]
Jan 24 08:59:52 sip1 kernel: pcapsipdump[5360]: segfault at 7f03dfe2f000 ip 0000000000402758 sp 00007fff96917cd0 error 6 in pcapsipdump[400000+5000]
Jan 24 09:01:53 sip1 kernel: pcapsipdump[6321]: segfault at 7f9aa3a86000 ip 0000000000402758 sp 00007fffd9945740 error 6 in pcapsipdump[400000+5000]
Jan 24 09:03:54 sip1 kernel: pcapsipdump[6768]: segfault at 7f3253c12000 ip 0000000000402758 sp 00007fff69281fb0 error 6 in pcapsipdump[400000+5000]
Jan 24 09:05:50 sip1 kernel: pcapsipdump[7113]: segfault at 7fbe22f96000 ip 0000000000402758 sp 00007fffc41f8720 error 6 in pcapsipdump[400000+5000]
Jan 24 09:07:51 sip1 kernel: pcapsipdump[7643]: segfault at 7f8452584000 ip 0000000000402758 sp 00007fff8dd961b0 error 6 in pcapsipdump[400000+5000]
Jan 24 09:14:12 sip1 kernel: pcapsipdump[9234]: segfault at 7f5d605dc000 ip 0000000000402758 sp 00007fff90a06fb0 error 6 in pcapsipdump[400000+5000]
Jan 24 09:15:58 sip1 kernel: pcapsipdump[9669]: segfault at 7f5c9181c000 ip 0000000000402758 sp 00007fffe7450ae0 error 6 in pcapsipdump[400000+5000]
Jan 24 09:18:15 sip1 kernel: pcapsipdump[10182]: segfault at 7f0baaee7000 ip 0000000000402758 sp 00007fffde9bd110 error 6 in pcapsipdump[400000+5000]
Jan 24 09:20:14 sip1 kernel: pcapsipdump[10495]: segfault at 7fcb1e996000 ip 0000000000402758 sp 00007fff11326a00 error 6 in pcapsipdump[400000+5000]
Jan 24 09:22:16 sip1 kernel: pcapsipdump[11221]: segfault at 7f92b8456000 ip 0000000000402758 sp 00007fff9a574260 error 6 in pcapsipdump[400000+5000]
Jan 24 09:23:46 sip1 kernel: pcapsipdump[11529]: segfault at 7fb39b649000 ip 0000000000402758 sp 00007fff067bc520 error 6 in pcapsipdump[400000+5000]
Jan 24 09:26:03 sip1 kernel: pcapsipdump[12264]: segfault at 7f86bbbc4000 ip 0000000000402758 sp 00007fff4adb1c60 error 6 in pcapsipdump[400000+5000]
Jan 24 09:28:04 sip1 kernel: pcapsipdump[12732]: segfault at 7f8295845000 ip 0000000000402758 sp 00007fffcb687fe0 error 6 in pcapsipdump[400000+5000]
Jan 24 09:30:05 sip1 kernel: pcapsipdump[12943]: segfault at 7f3e7031a000 ip 0000000000402758 sp 00007fffbebde330 error 6 in pcapsipdump[400000+5000]
Jan 24 09:32:06 sip1 kernel: pcapsipdump[13170]: segfault at 7fa13d7e2000 ip 0000000000402758 sp 00007fff913ed210 error 6 in pcapsipdump[400000+5000]
Jan 24 09:34:08 sip1 kernel: pcapsipdump[13384]: segfault at 7f13feefe000 ip 0000000000402758 sp 00007fff3559aa60 error 6 in pcapsipdump[400000+5000]

Discussion

  • Aex Aey
    Aex Aey
    2014-01-24

    Could you please try to debug this crash?

    ### prepare / compile
    make pcapsipdump-debug
    ulimit -c unlimited
    rm core.*
    ### run
    ./pcapsipdump-debug {all-the-usual-pcapsipdump's-arguments}
    ### debug core
    gdb pcapsipdump-debug core.*
    (gdb) bt
    (gdb) info args
    (gdb) info locals
    (gdb) up
    ### now repeat last 3 steps until "up" would be refused
    (gdb) info args
    (gdb) info locals
    (gdb) up
    ...
    

    One thing to watch out is that if "-d" is specified, core file would be created in that specified dir. You would need to move it to the same dir as debug binary and source are in before running "gdb".

     
  • avimar
    avimar
    2014-01-27

    .. and of course pcapsipdump-debug is not crashing now! I'll run the gdb commands once monit tells me it crashed.
    Thanks!

     
  • avimar
    avimar
    2014-02-26

    Post awaiting moderation.
  • Aex Aey
    Aex Aey
    2014-02-26

    Yep. That's a known problem. What crashes is:

    ct->table[idx].f=NULL;

    At this point idx = 10256, while calltable_max is 10240 (i suppose, that's default). In other words, it runs out of calltable because there are too many simulationous calls to cope with.

    Crash itself is already fixed in SVN (error message is now printed if calltable overflows):

    http://sourceforge.net/code-snapshots/svn/p/pc/pcapsipdump/code/pcapsipdump-code-66-trunk.zip

    ...but probably you want to edit calltable.h to increase calltable_max somewhat, if this many calls are expected.

     
  • avimar
    avimar
    2014-02-27

    How many calls is the default? I'm pretty sure I'm not running 10240 calls at a time...

    The current fix in SVN just means it will just ignore and not save the overflow calls, rather than crashing?

    I see two numbers:
    #define calltable_max 10240
    #define calltable_max_ip_per_call 4

    Which do I need to increase? What are the consequences of increasing it?

    Thanks!!

     
    Last edit: avimar 2014-02-27
  • Aex Aey
    Aex Aey
    2014-02-27

    Q: How many calls is the default?
    A: 10240

    Q: I'm pretty sure I'm not running 10240 calls at a time...
    A: What about recent calls? Call could still be considered "current" for slightly more than 315 seconds after last packet was seen. If still not a plausible explanation, table cleanup algorighm could be broken, I'll look into it.

    Q: The current fix in SVN just means it will just ignore and not save the overflow calls, rather than crashing?
    A: correct

    Q: Which do I need to increase?
    A: calltable_max

    Q: What are the consequences of increasing it?
    A:
    1) More simultanious calls would be allowed
    2) More RAM would be used
    3) More CPU would be used (table search algorighm is rubbish and does not scale well at all)

    ...if after increasing calltable_max, CPU is hogging, try to decrease calltable_max_ip_per_call to 2 (although this would break Re-INVITE calls)

     
  • avimar
    avimar
    2014-02-28

    Thanks for the thorough Q&A answers. You're a star!!

    Logs show ~20 concurrent calls = 40 legs with RTP during the time of the crashes.
    There might have been another ~30 calls (60 legs) of just SIP.

    Hmm, if I bridge, and it's hung up - so it keeps that call for 315 seconds in memory... I'm only able to pull up about 50 CDRs around this time.
    pcapsipdump was spinning -- going on and off, so it wasn't a table cleanup issue, per se, but perhaps it was counting... sip clients registering, pings... something else?

    If/when it spins again, I'll try upping the table - I think I can spare a bit more ram and cpu, it doesn't seem to use much at all.
    Thanks!

     
  • avimar
    avimar
    2014-02-28

    Ah! I figured I might as well see what pcapsipdump saved before crashing.
    It was friendly-scanner packets, about 1020 of them before seg-faulting each time.

    Iptables has been instructed to drop these, but in promiscuous mode, I presume pcapsipdump gets the packets untouched by iptables...

     
  • Aex Aey
    Aex Aey
    2014-11-17

    • status: open --> closed-fixed
    • assigned_to: Aex Aey
     
  • avimar
    avimar
    2014-11-24

    So.. I'm getting bombarded again with "User-Agent: friendly-scanner"

    I don't see them in my freeswitch siptrace, and it seems that iptables is correctly banning it.
    But I'm still seeing a TON of new files of calls.
    I tried -p to NOT use promiscuous mode, but that doesn't seem to make any difference.

    Does pcapsipdump have any way to not record these call? pcapsipdump no longer crashes with the patch from SVN- but is there a way for iptables blocked packets to not clog up my logs?

    Thanks!