#998 Murmur stops accepting new connections once a ceiling is reached until restart

1.2.4
closed
nobody
None
1.2.4
2
2013-06-14
2013-06-13
No

When murmur receives over 1,000 clients it will stop accepting new incoming connections. Existing connections remain, but new connections won't authenticate even if the connection count drops below the threshold that triggered the event.

If clients are allowed to drain to 0 and the voice thread restarts then clients can reconnect to the server successfully until the ceiling it hit again.

We are currently investigating a possible FD_SETSIZE issue in Qt as being the root cause.

Discussion

  • Solo Drakban

    Solo Drakban - 2013-06-13

    Also this is not a ulimit issue, the process has 8k file handles available to it as confirmed via /proc/<pid>/limits and lsof reveals used file handles roughly in the same count as the number of connections.

     
  • Solo Drakban

    Solo Drakban - 2013-06-13

    This is a self-compiled version against git (1.2.4 tag currently) under Amazon Linux (2013.03) linked against the following libraries:

    ldd /opt/murmur/murmurd

    linux-vdso.so.1 =>  (0x00007fff1b476000)
    libprotobuf.so.7 => /usr/local/lib/libprotobuf.so.7 (0x00007f3d8549b000)
    libcap.so.2 => /lib64/libcap.so.2 (0x00007f3d85296000)
    libQtDBus.so.4 => /usr/local/Trolltech/Qt-4.8.4/lib/libQtDBus.so.4 (0x00007f3d85016000)
    libssl.so.10 => /usr/lib64/libssl.so.10 (0x00007f3d84db1000)
    libcrypto.so.10 => /lib64/libcrypto.so.10 (0x00007f3d849ec000)
    libQtSql.so.4 => /usr/local/Trolltech/Qt-4.8.4/lib/libQtSql.so.4 (0x00007f3d847ab000)
    libQtXml.so.4 => /usr/local/Trolltech/Qt-4.8.4/lib/libQtXml.so.4 (0x00007f3d84567000)
    libQtNetwork.so.4 => /usr/local/Trolltech/Qt-4.8.4/lib/libQtNetwork.so.4 (0x00007f3d84213000)
    libQtCore.so.4 => /usr/local/Trolltech/Qt-4.8.4/lib/libQtCore.so.4 (0x00007f3d83d2b000)
    libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f3d83b0f000)
    libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00007f3d83805000)
    libm.so.6 => /lib64/libm.so.6 (0x00007f3d83582000)
    libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f3d8336d000)
    libc.so.6 => /lib64/libc.so.6 (0x00007f3d82fe0000)
    libattr.so.1 => /lib64/libattr.so.1 (0x00007f3d82ddb000)
    libgssapi_krb5.so.2 => /lib64/libgssapi_krb5.so.2 (0x00007f3d82b98000)
    libkrb5.so.3 => /lib64/libkrb5.so.3 (0x00007f3d828b2000)
    libcom_err.so.2 => /lib64/libcom_err.so.2 (0x00007f3d826af000)
    libk5crypto.so.3 => /lib64/libk5crypto.so.3 (0x00007f3d82484000)
    libdl.so.2 => /lib64/libdl.so.2 (0x00007f3d8227f000)
    libz.so.1 => /lib64/libz.so.1 (0x00007f3d82068000)
    librt.so.1 => /lib64/librt.so.1 (0x00007f3d81e5f000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f3d85790000)
    libkrb5support.so.0 => /lib64/libkrb5support.so.0 (0x00007f3d81c54000)
    libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x00007f3d81a50000)
    libresolv.so.2 => /lib64/libresolv.so.2 (0x00007f3d81835000)
    libselinux.so.1 => /usr/lib64/libselinux.so.1 (0x00007f3d81613000)
    
     
  • Solo Drakban

    Solo Drakban - 2013-06-14

    This bug can be closed. It was discovered via help on IRC that Qt wasn't compiled against glib2 which was preventing select() from addressing more than 1024 sockets.

     
  • Mikkel Krautz

    Mikkel Krautz - 2013-06-14
    • status: open --> closed
     
  • Mikkel Krautz

    Mikkel Krautz - 2013-06-14

    Closing.