Learn how easy it is to sync an existing GitHub or Google Code repo to a SourceForge project! See Demo

Close

#561 crash in dialog module

1.8.x
open
modules (454)
5
2012-10-21
2012-09-27
Flavio Goncalves
No

Error in log,

Sep 27 15:15:10 kernel: opensips[11386]: segfault at 7ffc00000020 ip 00007ffca377fcdc sp 00007fff0b587420 error 4 in dialog.so[7ffca374f000+4b000]

Backtrace:
#0 is_dlg_in_profile (msg=<value optimized out>, profile=0x7ffc80ee2c28, value=0x0) at dlg_profile.c:824
824 if (linker->profile==profile) {
Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.80.el6_3.5.x86_64 keyutils-libs-1.4-4.el6.x86_64 krb5-libs-1.9-33.el6_3.2.x86_64 libcom_err-1.41.12-12.el6.x86_64 librabbitmq-0.1-0.2.hgfb6fca832fd2.el6.x86_64 libselinux-2.0.94-5.3.el6.x86_64 mysql-libs-5.1.61-4.el6.x86_64 nss-softokn-freebl-3.12.9-11.el6.x86_64 openssl-1.0.0-25.el6_3.1.x86_64 pcre-7.8-4.el6.x86_64 zlib-1.2.3-27.el6.x86_64
(gdb) backtrace
#0 is_dlg_in_profile (msg=<value optimized out>, profile=0x7ffc80ee2c28, value=0x0) at dlg_profile.c:824
#1 0x00007ffca37573b2 in w_is_in_profile (msg=0x7ffca6912490, profile=0x7ffc80ee2c28 "\250,\356\200\374\177", value=<value optimized out>) at dialog.c:1019
#2 0x0000000000411ca6 in do_action (a=0x7ffca67ecc30, msg=0x7ffca6912490) at action.c:1483
#3 0x000000000040fd02 in run_action_list (a=<value optimized out>, msg=0x7ffca6912490) at action.c:143
#4 0x000000000048939f in eval_elem (e=0x7ffca67ecd08, msg=0x7ffca6912490, val=0x0) at route.c:1438
#5 0x000000000048ad2d in eval_expr (e=0x7ffca67ecd08, msg=0x7ffca6912490, val=0x0) at route.c:1783
#6 0x000000000048ae42 in eval_expr (e=0x7ffca67ecd58, msg=0x7ffca6912490, val=0x0) at route.c:1790
#7 0x000000000048acf3 in eval_expr (e=0x7ffca67ecda8, msg=0x7ffca6912490, val=0x0) at route.c:1804
#8 0x0000000000411abc in do_action (a=0x7ffca67ecf68, msg=0x7ffca6912490) at action.c:899
#9 0x000000000040fd02 in run_action_list (a=<value optimized out>, msg=0x7ffca6912490) at action.c:143
#10 0x0000000000414175 in do_action (a=0x7ffca67ed118, msg=0x7ffca6912490) at action.c:916
#11 0x000000000040fd02 in run_action_list (a=<value optimized out>, msg=0x7ffca6912490) at action.c:143
#12 0x0000000000414175 in do_action (a=0x7ffca67f0568, msg=0x7ffca6912490) at action.c:916
#13 0x000000000040fd02 in run_action_list (a=<value optimized out>, msg=0x7ffca6912490) at action.c:143
#14 0x0000000000414175 in do_action (a=0x7ffca67f0718, msg=0x7ffca6912490) at action.c:916
#15 0x000000000040fd02 in run_action_list (a=<value optimized out>, msg=0x7ffca6912490) at action.c:143
#16 0x0000000000413856 in run_actions (a=0x7ffca67b1178, msg=0x7ffca6912490) at action.c:123
#17 do_action (a=0x7ffca67b1178, msg=0x7ffca6912490) at action.c:573
#18 0x000000000040fd02 in run_action_list (a=<value optimized out>, msg=0x7ffca6912490) at action.c:143
#19 0x00000000004164b8 in run_actions (a=0x7ffca67b0d28, msg=0x7ffca6912490) at action.c:123
#20 run_top_route (a=0x7ffca67b0d28, msg=0x7ffca6912490) at action.c:189
#21 0x000000000047351e in receive_msg (buf=<value optimized out>, len=<value optimized out>, rcv_info=0x7fff0b589e30) at receive.c:165
#22 0x00000000004ccd91 in udp_rcv_loop () at udp_server.c:424
#23 0x00000000004303aa in main_loop (argc=<value optimized out>, argv=<value optimized out>) at main.c:876
#24 main (argc=<value optimized out>, argv=<value optimized out>) at main.c:1520

Discussion

  • Apparently, the problem was caused by garbage being saved in a profile with value \250,\356\200\374\177. The problem always happen after a lot of malformed messages and I'm saving $fd in the profile with value domain. I believe the crash occurred when listing this profile to display current calls per domain in the dashboard. The solution was to check if $fd was not null before saving to an avp and checking if the avp has a value of one of the local domains. I will continue monitoring to check if this solved the problem. A check before saving values to profiles can be valuable.

     
  • The problem is actually the function is_in_profile. For no specific reason it is crashing once per day. It is checking a profile without value. The workaround was to remove the lines with this function (is_in_profile) and change the scripting logic.

     
    • assigned_to: nobody --> vladut-paiu
     
  • Hello,

    This seems like a memory corruption somewhere in the dialog module. The profile name is '\250,\356\200\374\177' , which is obviously wrong.
    Have you tried to run OpenSIPS compiled with memory debugging, and see if OpenSIPS complains about any buffer overflows/underflows ?

    Also, can you try to elaborate a little what steps did you take in order to avoid the problem ?

    Regards,
    Vlad

     
  • What I did to solve the problem was to remove the function is_in_profile(prepaid) prepaid was a profile with no value. I was using is_in_profile to mark prepaid calls for an external system. Now the external system receives this information using rabbitmq. I changed the logic to use events instead of dialog profiles to mark prepaid calls. This was the workaround to the problem. It happened in only one platform. I suspect this was caused by malformed request, becauses I have seen many warnings in the log. I captured all the sip requests until the crash to a file (ramdrive), but there was nothing being received in the moment of the crash calling my attention. Unfortunately I don't have the file anymore (~4GB).