RE: [Quickfix-developers] crashes while under moderate message load

[Sean K. <sea...@pi...>]

The file descriptor limit in linux is separate from the FD_SETSIZE value =
of 1024.  In other words, you could set the nofile limit for a user to =
10,240 and still have the FD_SET limitation of 1024.  There is a hack to =
get around the FD_SET hard limit, but as far as I've found it is not =
recommended.  The FD_SET is a sized array that is indexed based on the =
created socket file descriptor.  Once you've exceeded 1024 fds created =
by your process, you will get unexpected behavior.  This will be most =
apparent when a lot of sockets are being created at the same time.
=20
Our solution to a similar problem here has been to utilize the database =
store, decreasing the fds per session to 1 for the socket itself.  =
Another possible solution could be to run multiple acceptors (of course, =
then you have to do some kind of load balancing to hit different ports).
=20
--Sean

-----Original Message-----
From: James Reed [mailto:jam...@gm...]
Sent: Tuesday, March 21, 2006 12:07 PM
To: Sean Kirkpatrick; qui...@li...
Subject: Re: [Quickfix-developers] crashes while under moderate message =
load


Hello Sean,

We are using a File store for the message store. Our app makes a call to =
setrlimit and sets the fdlimit to 2048. Currently our app is running =
with 1301 fds being used without problems.

Thanks.


On 3/21/06, Sean Kirkpatrick < sea...@pi...> =
wrote:=20

What are you using for the message store?  If the acceptor is using =
files, then I believe 4 file descriptors are created per session =
(seqnums, session, body, and header).  Throw a socket in per session and =
you are most likely going over the 1024 FD_SET limit with 288 =
sessions...

=20
--Sean


-----Original Message-----
From: qui...@li... [mailto: =
qui...@li...]On Behalf Of James Reed
Sent: Tuesday, March 21, 2006 11:50 AM
To: qui...@li...
Subject: [Quickfix-developers] crashes while under moderate message load


Hello all,

We've experienced unexplained crashes where no core dumps are generated =
nor is there anything in the logs to indicate what the problem is. We =
have only noticed that the crashes tend to occur when a large number of =
clients attempt to connect at once or if a significant portion of the =
connected clients try to send messages in a short span of time.

We are using QuickFIX 1.10.2, with g++ 3.2.3, on RHEL AS Release 3. Our =
application uses one Initiator and one Acceptor. The Acceptor is =
configured with 288 Sessions. Our application is also configured to use =
a StdOutLogger to minimize usage of file descriptors. The fdlimit for =
the user account running the process is 2048. I know there is not much =
to go on from the information presented, but does anyone have any ideas =
about what could possibly happening here?

Thanks.

We've run the application in a debugger and found the following upon =
crashing:

Stack Traces

Incident #1:


#0  0x00d22b60 in pthread_detach () from /lib/tls/libpthread.so.0
#1  0x00a4dc3b in FIX::thread_detach (thread=3D0) at Utility.cpp:344
#2  0x00a07334 in FIX::ThreadedSocketAcceptor::removeThread =
(this=3D0x837d210, s=3D1216)
    at stl_map.h:221
#3  0x00a07453 in FIX::ThreadedSocketAcceptor::socketThread =
(p=3D0x6ea61a10)
    at ThreadedSocketConnection.h:51
#4  0x00d21dec in start_thread () from /lib/tls/libpthread.so.0
#5  0x005a8a2a in clone () from /lib/tls/libc.so.6

Incident #2:
=20

#0  0x00a8bb60 in pthread_detach () from /lib/tls/libpthread.so.0
#1  0x00de7c3b in FIX::thread_detach (thread=3D0) at Utility.cpp:344
#2  0x00da1334 in FIX::ThreadedSocketAcceptor::removeThread =
(this=3D0xb6a4f448, s=3D1170)
    at stl_map.h:221
#3  0x00da1453 in FIX::ThreadedSocketAcceptor::socketThread =
(p=3D0xa95f8a10)
    at ThreadedSocketConnection.h:51
#4  0x00a8adec in start_thread () from /lib/tls/libpthread.so.0
#5  0x00595a2a in clone () from /lib/tls/libc.so.6

Incident #3:
=20

#0  0x0027e6d7 in std::string::_Rep::_M_grab () from =
/usr/lib/libstdc++.so.5
#1  0x0027e81c in std::basic_string<char, std::char_traits<char>, =
std::allocator<char> >::basic_string () from /usr/lib/libstdc++.so.5
#2  0x08077b56 in FieldBase (this=3D0x92e9b68, _ctor_arg=3D@0x9379608) =
at SessionID.h:45
#3  0x08077afa in StringField (this=3D0x92e9b68, _ctor_arg=3D@0x9379608) =
at SessionID.h:45
#4  0x08077a2e in BeginString (this=3D0x92e9b68, _ctor_arg=3D@0x9379608) =
at SessionID.h:45
#5  0x0807782f in SessionID (this=3D0x92e9b68, _ctor_arg=3D@0x9379608) =
at SessionID.h:119
#6  0x00f158bb in std::_Rb_tree<FIX::SessionID, FIX::SessionID, =
std::_Identity<FIX::SessionID>, std::less<FIX::SessionID>, =
std::allocator<FIX::SessionID> >::_M_copy (
    this=3D0xb75dea20, __x=3D0x93795f8, __p=3D0x92e9af8) at new:89
#7  0x00f15df0 in _Rb_tree (this=3D0xb75dea20, __x=3D@0xbfff97ec) at =
stl_tree.h:648
#8  0x00f14189 in FIX::Initiator::isLoggedOn (this=3D0xbfff97d0) at =
stl_set.h:131
#9  0x00f19961 in FIX::SocketInitiator::onStart (this=3D0xbfff97d0)
    at SocketInitiator.cpp:86
#10 0x00f142e7 in FIX::Initiator::startThread (p=3D0xb75de8c0) at =
Initiator.cpp:243
#11 0x0058fdec in start_thread () from /lib/tls/libpthread.so.0
#12 0x00393a2a in clone () from /lib/tls/libc.so.6