#622 Load balancer - corrupt data when loading table

1.10.x
open
modules (454)
5
2014-05-05
2013-03-06
Umberto Mautone
No

Tested on 1.9.x

Data table:

id | group_id | dst_uri | resources | probe_mode
----+----------+-------------------+-----------+------------
19 | 1 | sip:192.168.1.5 | pstn=100 | 0
20 | 1 | sip:192.168.1.8 | pstn=100 | 0
21 | 2 | sip:192.168.1.100 | pstn=2500 | 0

I added this line in "lb_db_load_data()" to log the actual table fields being loaded:

LM_ERR("Fields: %d %d %s %s\n", id, group, uri, resource);

The first row loads correctly but subsequent rows seem to lose a pointer alignment somewhere:

Mar 6 10:01:53 localhost opensips[1796]: ERROR:load_balancer:lb_db_load_data: Fields: 19 1 sip:192.168.1.5 pstn=100
Mar 6 10:01:53 localhost opensips[1796]: ERROR:load_balancer:lb_db_load_data: Fields: 20 1 sip:192.168.1.8 ¥Ý9w#177
Mar 6 10:01:53 localhost opensips[1796]: ERROR:load_balancer:parse_resources_list: resource must has value!
Mar 6 10:01:53 localhost opensips[1796]: ERROR:load_balancer:add_lb_dsturi: failed to parse resourse string < ¥Ý9w#177>
Mar 6 10:01:53 localhost opensips[1796]: ERROR:load_balancer:lb_db_load_data: failed to add destination 1 -> skipping
Mar 6 10:01:53 localhost opensips[1796]: ERROR:load_balancer:lb_db_load_data: Fields: 21 2 sip:192.168.1.100 192.168.1.5
Mar 6 10:01:53 localhost opensips[1796]: ERROR:load_balancer:parse_resources_list: resource must has value!
Mar 6 10:01:53 localhost opensips[1796]: ERROR:load_balancer:add_lb_dsturi: failed to parse resourse string <192.168.1.5>
Mar 6 10:01:53 localhost opensips[1796]: ERROR:load_balancer:lb_db_load_data: failed to add destination 1 -> skipping

Also tested in 1.8.2 and the problem is not present. The bug seems to be only in the 1.9.x branch.

Discussion

  • Hi Umberto,

    The code for loading LB data did not change from 1.8 to 1.9....so it is a bit odd. I was not able to reproduce it :(.

    What DB engine are you using ? Also, could you try running in full debug (set debug=4).

    Thanks and regards,
    Bogdan

     
    • assigned_to: nobody --> bogdan_iancu
     
  • I have both 1.8.2 and 1.9.0 running on the same box. If I start up 1.8.2, it only loads the first record and I have to call "opensipsctl fifo lb_reload" to force all profiles to be loaded and they load correctly.

    When I start 1.9.0, it tries to load all profiles but some get mangled with garbage. Even issuing a reload will load the exact same records mangled at the exact same place.

    I'm using db_unixodbc connected to PostgreSQL.

    I'll try the same load with debug level set to 4 and paste the result.

     
  • Sorry for taking so long. This is using PostgreSQL 9.2 with unixODBC on a 64 bit box. Here is the output with debug=4 on startup:

    ===================================
    Apr 5 16:57:24 localhost opensips: WARNING:core:warn: warning in config file /usr//etc/opensips/opensips.cfg, line 44, column 13-16: tls support not compiled in
    Apr 5 16:57:24 localhost opensips: INFO:core:init_tcp: using epoll_lt as the TCP io watch method (auto detected)
    Apr 5 16:57:24 localhost opensips[13284]: NOTICE:core:main: version: opensips 1.9.0-notls (x86_64/linux)
    Apr 5 16:57:24 localhost opensips[13284]: INFO:core:main: using 32 Mb shared memory
    Apr 5 16:57:24 localhost opensips[13284]: INFO:core:main: using 2 Mb private memory per process
    Apr 5 16:57:24 localhost opensips[13284]: INFO:core:evi_publish_event: Registered event <E_CORE_THRESHOLD(0)>
    Apr 5 16:57:24 localhost opensips[13284]: INFO:core:evi_publish_event: Registered event <E_CORE_SHM_THRESHOLD(1)>
    Apr 5 16:57:24 localhost opensips[13284]: INFO:core:evi_publish_event: Registered event <E_CORE_PKG_THRESHOLD(2)>
    Apr 5 16:57:24 localhost opensips[13284]: NOTICE:signaling:mod_init: initializing module ...
    Apr 5 16:57:24 localhost opensips[13284]: INFO:sl:mod_init: Initializing StateLess engine
    Apr 5 16:57:24 localhost opensips[13284]: INFO:tm:mod_init: TM - initializing...
    Apr 5 16:57:24 localhost opensips[13284]: INFO:rr:mod_init: rr - initializing
    Apr 5 16:57:24 localhost opensips[13284]: INFO:maxfwd:mod_init: initializing...
    Apr 5 16:57:24 localhost opensips[13284]: INFO:sipmsgops:mod_init: initializing...
    Apr 5 16:57:24 localhost opensips[13284]: INFO:dialog:mod_init: Dialog module - initializing
    Apr 5 16:57:24 localhost opensips[13284]: INFO:load_balancer:mod_init: Load-Balancer module - initializing
    Apr 5 16:57:24 localhost opensips[13284]: ERROR:load_balancer:parse_resources_list: resource must has value!
    Apr 5 16:57:24 localhost opensips[13284]: ERROR:load_balancer:add_lb_dsturi: failed to parse resourse string <HÿóÇ*#177>
    Apr 5 16:57:24 localhost opensips[13284]: ERROR:load_balancer:lb_db_load_data: failed to add destination 3 -> skipping
    Apr 5 16:57:24 localhost opensips[13284]: ERROR:core:parse_uri: bad uri, state 0 parsed: <72.0> (4) / <1.2.3.179> (12)
    Apr 5 16:57:24 localhost opensips[13284]: ERROR:load_balancer:add_lb_dsturi: bad uri [1.2.3.179] for destination
    Apr 5 16:57:24 localhost opensips[13284]: ERROR:load_balancer:lb_db_load_data: failed to add destination 3 -> skipping
    Apr 5 16:57:24 localhost opensips[13284]: INFO:sipmsgops:hname_fixup: using hdr type name <X-Auth-IP>
    Apr 5 16:57:24 localhost opensips[13284]: INFO:sipmsgops:hname_fixup: using hdr type (9) instead of <Route>
    Apr 5 16:57:24 localhost rsyslogd-2177: imuxsock begins to drop messages from pid 13284 due to rate-limiting
    Apr 5 16:57:24 localhost opensips: INFO:core:daemonize: pre-daemon process exiting with 0
    ===================================

    And here is the output when I issue an "lb_reload":

    ===================================
    Apr 5 16:57:39 localhost opensips[13285]: INFO:load_balancer:mi_lb_reload: "lb_reload" MI command received!
    Apr 5 16:57:39 localhost opensips[13285]: ERROR:load_balancer:parse_resources_list: resource must has value!
    Apr 5 16:57:39 localhost opensips[13285]: ERROR:load_balancer:add_lb_dsturi: failed to parse resourse string < ôóÇ*#177>
    Apr 5 16:57:39 localhost opensips[13285]: ERROR:load_balancer:lb_db_load_data: failed to add destination 3 -> skipping
    Apr 5 16:57:39 localhost opensips[13285]: ERROR:core:parse_uri: bad uri, state 0 parsed: <72.0> (4) / <1.2.3.179> (12)
    Apr 5 16:57:39 localhost opensips[13285]: ERROR:load_balancer:add_lb_dsturi: bad uri [1.2.3.179] for destination
    Apr 5 16:57:39 localhost opensips[13285]: ERROR:load_balancer:lb_db_load_data: failed to add destination 3 -> skipping

     
  • This bug as tested is happening on CentOS 6.x and Ubuntu 12.xx. The only common component between the two is PostgreSQL.

     
  • Hello,
    I'm using 1.10.1 from the official yum repo (http://yum.opensips.org/1.10/releases/el/6/x86_64/opensips-yum-releases-1.10-1.el6.noarch.rpm)

    this bug is still present for odbc connections to mysql.

    mysql> select * from load_balancer;
    +----+----------+--------------------+-------------------+------------+-------------+
    | id | group_id | dst_uri | resources | probe_mode | description |
    +----+----------+--------------------+-------------------+------------+-------------+
    | 1 | 1 | sip:10.0.0.95:5060 | pstn=500;sip=500 | 2 | |
    | 3 | 1 | sip:10.0.0.83:5060 | sip=500 | 2 | |
    +----+----------+--------------------+-------------------+------------+-------------+

    after a lb_reload I can see the first destination loaded correctly, but not the second.

    opensipsctl fifo lb_list
    Destination:: sip:10.0.0.95:5060 id=1 group=1 enabled=yes auto-re=on
    Resource:: pstn max=500 load=0
    Resource:: sip max=500 load=0

    log file:

    Apr 22 10:49:29 /usr/sbin/opensips[2036]: INFO:load_balancer:mi_lb_reload: "lb_reload" MI command received!
    Apr 22 10:49:29 /usr/sbin/opensips[2036]: ERROR:core:parse_uri: bad uri, state 0 parsed: <10.0> (4) / <10.0.0.95> (9)
    Apr 22 10:49:29 /usr/sbin/opensips[2036]: ERROR:load_balancer:add_lb_dsturi: bad uri [10.0.0.95] for destination
    Apr 22 10:49:29 /usr/sbin/opensips[2036]: ERROR:load_balancer:lb_db_load_data: failed to add destination 1 -> skipping

    don't understand why it complains about 10.0.0.95 in the log file when that's the one loaded.
    Problem is solved changing unixodbc connection for mysql, using the same table content.

     
  • You say it perfectly works if not using unixodbc layer, but going directly for mysql/postgres ?

    Thanks,
    Bogdan

     
  • yes, using mysql module directly solves the problem.
    I'll try to reproduce on a clean vm later and post the procedure I use for the full installation, hopefully you'll be able to reproduce as well and debug the problem.

     
  • managed to reproduce the problem on a clean centos 6.5 64bit

    here's the procedure I followed:
    install epel repo, update and install mysql and odbc from base (epel is needed for some dependencies)
    install http://yum.opensips.org/1.10/releases/el/6/x86_64/opensips-yum-releases-1.10-1.el6.noarch.rpm
    then all opensips packages version 1.10.1

    rpm -qa|grep mysql
    opensips-mysql-1.10.1-1.el6.x86_64
    mysql-5.1.73-3.el6_5.x86_64
    mysql-connector-odbc-5.1.5r1144-7.el6.x86_64
    mysql-server-5.1.73-3.el6_5.x86_64
    mysql-libs-5.1.73-3.el6_5.x86_64

    enable mysql in /etc/opensips/opensipsctlrc

    opensipsdbctl create # answer yes to all

    opensips.cfg
    http://pastebin.com/7svHDh99
    only change to the default conf is

    loadmodule "dialog.so"
    #loadmodule "db_mysql.so"
    loadmodule "db_unixodbc.so"
    loadmodule "load_balancer.so"
    #modparam("load_balancer", "db_url", "mysql://opensips:opensipsrw@localhost/opensips")
    modparam("load_balancer", "db_url", "unixodbc://opensips:opensipsrw@localhost/OpenSIPS")

    configure odbc.ini

    [OpenSIPS]
    Description = MySQL OpenSIPS
    Driver = MySQL
    Socket = /var/lib/mysql/mysql.sock
    Server = localhost
    Database = opensips
    Option = 3

    insert 2 hosts in the load_balancer table:

    mysql -uopensips -popensipsrw opensips -e "INSERT INTO load_balancer VALUES(1,1,'sip:10.0.0.2:5060','pstn=500;sip=500',2,'');"
    mysql -uopensips -popensipsrw opensips -e "INSERT INTO load_balancer VALUES(2,1,'sip:10.0.0.3:5060','pstn=500',2,'');"
    mysql -uopensips -popensipsrw opensips -e "SELECT * FROM load_balancer;"

    +----+----------+-------------------+------------------+------------+-------------+
    | id | group_id | dst_uri | resources | probe_mode | description |
    +----+----------+-------------------+------------------+------------+-------------+
    | 1 | 1 | sip:10.0.0.2:5060 | pstn=500;sip=500 | 2 | |
    | 2 | 1 | sip:10.0.0.3:5060 | pstn=500 | 2 | |
    +----+----------+-------------------+------------------+------------+-------------+

    start opensips with odbc enabled:
    /etc/init.d/opensips restart
    opensipsctl fifo lb_reload

    log:

    Apr 23 08:32:45 localhost /usr/sbin/opensips[1919]: INFO:load_balancer:mi_lb_reload: "lb_reload" MI command received!
    Apr 23 08:32:45 localhost /usr/sbin/opensips[1919]: ERROR:core:parse_uri: bad uri, state 0 parsed: <�^�#> (4) / <�^�#R#177> (6)
    Apr 23 08:32:45 localhost /usr/sbin/opensips[1919]: ERROR:load_balancer:add_lb_dsturi: bad uri [�^�#R#177] for destination
    Apr 23 08:32:45 localhost /usr/sbin/opensips[1919]: ERROR:load_balancer:lb_db_load_data: failed to add destination 1 -> skipping

    replace odbc with mysql and everything works.
    Hope this can be helpfull.

    Thanks.

     
  • Liviu Chircu
    Liviu Chircu
    2014-05-05

    Hello,

    This issue was fixed on the master, 1.11, 1.10 and 1.8 branches - thank you for the detailed reporting!

    Best regards,
    Liviu

     
    Last edit: Liviu Chircu 2014-05-05