#998 Murmur stops accepting new connections once a ceiling is reached until restart

1.2.4
closed
nobody
None
1.2.4
2
2013-06-14
2013-06-13
Solo Drakban
No

When murmur receives over 1,000 clients it will stop accepting new incoming connections. Existing connections remain, but new connections won't authenticate even if the connection count drops below the threshold that triggered the event.

If clients are allowed to drain to 0 and the voice thread restarts then clients can reconnect to the server successfully until the ceiling it hit again.

We are currently investigating a possible FD_SETSIZE issue in Qt as being the root cause.

Discussion

  • Solo Drakban
    Solo Drakban
    2013-06-13

    Also this is not a ulimit issue, the process has 8k file handles available to it as confirmed via /proc/<pid>/limits and lsof reveals used file handles roughly in the same count as the number of connections.

     
  • Solo Drakban
    Solo Drakban
    2013-06-13

    This is a self-compiled version against git (1.2.4 tag currently) under Amazon Linux (2013.03) linked against the following libraries:

    ldd /opt/murmur/murmurd

    linux-vdso.so.1 =>  (0x00007fff1b476000)
    libprotobuf.so.7 => /usr/local/lib/libprotobuf.so.7 (0x00007f3d8549b000)
    libcap.so.2 => /lib64/libcap.so.2 (0x00007f3d85296000)
    libQtDBus.so.4 => /usr/local/Trolltech/Qt-4.8.4/lib/libQtDBus.so.4 (0x00007f3d85016000)
    libssl.so.10 => /usr/lib64/libssl.so.10 (0x00007f3d84db1000)
    libcrypto.so.10 => /lib64/libcrypto.so.10 (0x00007f3d849ec000)
    libQtSql.so.4 => /usr/local/Trolltech/Qt-4.8.4/lib/libQtSql.so.4 (0x00007f3d847ab000)
    libQtXml.so.4 => /usr/local/Trolltech/Qt-4.8.4/lib/libQtXml.so.4 (0x00007f3d84567000)
    libQtNetwork.so.4 => /usr/local/Trolltech/Qt-4.8.4/lib/libQtNetwork.so.4 (0x00007f3d84213000)
    libQtCore.so.4 => /usr/local/Trolltech/Qt-4.8.4/lib/libQtCore.so.4 (0x00007f3d83d2b000)
    libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f3d83b0f000)
    libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00007f3d83805000)
    libm.so.6 => /lib64/libm.so.6 (0x00007f3d83582000)
    libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f3d8336d000)
    libc.so.6 => /lib64/libc.so.6 (0x00007f3d82fe0000)
    libattr.so.1 => /lib64/libattr.so.1 (0x00007f3d82ddb000)
    libgssapi_krb5.so.2 => /lib64/libgssapi_krb5.so.2 (0x00007f3d82b98000)
    libkrb5.so.3 => /lib64/libkrb5.so.3 (0x00007f3d828b2000)
    libcom_err.so.2 => /lib64/libcom_err.so.2 (0x00007f3d826af000)
    libk5crypto.so.3 => /lib64/libk5crypto.so.3 (0x00007f3d82484000)
    libdl.so.2 => /lib64/libdl.so.2 (0x00007f3d8227f000)
    libz.so.1 => /lib64/libz.so.1 (0x00007f3d82068000)
    librt.so.1 => /lib64/librt.so.1 (0x00007f3d81e5f000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f3d85790000)
    libkrb5support.so.0 => /lib64/libkrb5support.so.0 (0x00007f3d81c54000)
    libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x00007f3d81a50000)
    libresolv.so.2 => /lib64/libresolv.so.2 (0x00007f3d81835000)
    libselinux.so.1 => /usr/lib64/libselinux.so.1 (0x00007f3d81613000)
    
     
  • Solo Drakban
    Solo Drakban
    2013-06-14

    This bug can be closed. It was discovered via help on IRC that Qt wasn't compiled against glib2 which was preventing select() from addressing more than 1024 sockets.

     
  • Mikkel Krautz
    Mikkel Krautz
    2013-06-14

    • status: open --> closed
     
  • Mikkel Krautz
    Mikkel Krautz
    2013-06-14

    Closing.