Thread: Re: [Quickfix-users] Two more questions about QuickFix
Brought to you by:
orenmnero
Menu
▾
▴
quickfix-users
|
From: Ajay K. <Aja...@tr...> - 2008-02-08 18:31:42
|
Dan, You should also take a look at the application specific processing logic in the fromApp() method and determine how long it takes to run, because it too could be contributing to the slow processing of FIX messages sent by your counter party. - Ajay -----Original Message----- From: qui...@li... [mailto:qui...@li...] On Behalf Of Dan Guo Sent: Friday, February 08, 2008 1:03 PM To: qui...@li... Subject: [Quickfix-users] Two more questions about QuickFix QuickFIX Documentation: http://www.quickfixengine.org/quickfix/doc/html/index.html QuickFIX Support: http://www.quickfixengine.org/services.html Hi Everyone! we notice two issues and want to hear whether anyone knows what is going on. 1. when i sent FIX msgs at higher rate (10 msg per second), there seems a longer latency for the msg to hit market. i have strong suspecion QuickFix responsible for the longer latency, as i isolated other segments' contributions to the overall latency. can anyone comment on this? 2. our broker's network engineers noticed there is TCP back-pressure alert on their site. It basically means the FIX messages got queued on our broker's side. Either because our network connection is not adequate or (and) our QuickFix code cannot process them fast enough. did anyone in this list encounter the similiar issue? any comments? Thank you in advance. Dan ------------------------------------------------------------------------ - This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ _______________________________________________ Quickfix-users mailing list Qui...@li... https://lists.sourceforge.net/lists/listinfo/quickfix-users |
|
From: Dan G. <dua...@ya...> - 2008-02-18 04:03:22
|
John,
thanks. here we did a tcp dump and found this - two orders were sent consecutively back to back and the 2nd order is 6mill seconds after 1st one, on socket level. If that holds, 160 orders/s is the best we can do. we certainly think this is not acceptable. we wonder
1. if there is any way to improve;
2. if this is what others see;
3. if there is any configuration error on our part.
thanks,
Dan
----- Original Message ----
From: John Haldi <jh...@ca...>
To: Dan Guo <dua...@ya...>
Sent: Friday, February 8, 2008 1:52:04 PM
Subject: RE: [Quickfix-users] Two more questions about QuickFix
Dan,
1. On sending, you should log when you are actually calling the SendToTarget method and determine how long the delay is between calls within your code. Once you call SendToTarget, quickfix will take over and manage the socket connection to the broker. Depending on your line bandwidth and latency, as well as their ability to process things, this could be a bottleneck. If however you are seeing delays between successive calls to SendToTarget, the delay is in your code. QuickFix will manage any line delays by handling its own queue internally, so it won't cause any noticeable delays between successive calls to SendToTarget. You could also consider logging a millisecond timestamp both before you call SendToTarget and immediately after the method returns, in order to get a sense of the time quickfix is taking to release your thread.
2. When your broker is sending messages to you, quickfix ought to handle any delays in your code by transparently creating and managing an inbound queue on your side. You clear the queue by handling each onMessage event. If your code is the bottleneck, you won't react to the onMessage event until your code has finished processing the previous message. One way to get around this problem (if that is indeed what you are experiencing) is to create your own managed queue. Essentially you limit your code in the onMessage event to grabbing the message and putting it onto your own managed queue. Then, on a separate thread, you have a queue processor which clears the queue. This gives you the flexibility to easily monitor any queue length in your managed queue.
If however your broker is in fact seeing a bottleneck on their side in their ability to send you messages, it is likely a problem with your network connection to the broker. You might want to investigate the TCP traffic and determine if there is a problem between you and the broker. Incorrect net card drivers or router configuration can cause the TCP protocol to progressively reduce the frame size in an attempt to determine an optimal sending packet size. If this shrinks enough, it will manifest itself in seemingly slow network connectivity as the protocol takes each of your messages and breaks them into tiny chunks and then has to reassemble them on the other side before passing them up to your application.
If it helps you to get a benchmark for what to expect, on our systems we typically see about a 1,000 max message per second throughput using the single threaded socket initiator and socket acceptor object, configured for file (not DB) logging and with the .body files turned on. This is on a system using high speed dedicated lines between us and the exchanges/brokers in question. We have not gone further in our attempts to maximize throughput yet, as we only do a few hundred thousand orders per day. On a 4xCPU, 4GB RAM server running right now, I see CPU load at about 2% on our engine and its running about 20 messages per second right now.
Hope this helps,
John
-----Original Message-----
From: qui...@li... [mailto:qui...@li...] On Behalf Of Dan Guo
Sent: Friday, February 08, 2008 1:03 PM
To: qui...@li...
Subject: [Quickfix-users] Two more questions about QuickFix
QuickFIX Documentation: http://www.quickfixengine.org/quickfix/doc/html/index.html
QuickFIX Support: http://www.quickfixengine.org/services.html
Hi Everyone!
we notice two issues and want to hear whether anyone knows what is going on.
1. when i sent FIX msgs at higher rate (10 msg per second), there seems a longer latency for the msg to hit market.
i have strong suspecion QuickFix responsible for the longer latency, as i isolated other segments' contributions to the overall latency.
can anyone comment on this?
2. our broker's network engineers noticed there is TCP back-pressure alert on their site. It basically means the FIX messages got queued on our broker's side. Either because our network connection is not adequate or (and) our QuickFix code cannot process them fast enough.
did anyone in this list encounter the similiar issue? any comments?
Thank you in advance.
Dan
-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
Quickfix-users mailing list
Qui...@li...
https://lists.sourceforge.net/lists/listinfo/quickfix-users
No virus found in this incoming message.
Checked by AVG Free Edition.
Version: 7.5.516 / Virus Database: 269.19.21/1266 - Release Date: 2/8/2008 10:06 AM
No virus found in this outgoing message.
Checked by AVG Free Edition.
Version: 7.5.516 / Virus Database: 269.19.21/1266 - Release Date: 2/8/2008 10:06 AM
|
|
From: <or...@qu...> - 2008-03-28 15:31:36
|
<html><body><div>Do you have any more information about what you are doing? This doesn't really give a whole lot to comment on. How is this test setup?</div> <div> </div> <div>--oren</div> <BLOCKQUOTE style="PADDING-LEFT: 8px; MARGIN-LEFT: 8px; BORDER-LEFT: blue 2px solid" webmail="1">-------- Original Message --------<BR>Subject: Re: [Quickfix-users] Two more questions about QuickFix<BR>From: yihan <yi...@ho...><BR>Date: Fri, March 28, 2008 8:55 am<BR>To: <a href="mailto:qui...@li...">qui...@li...</a><BR><BR>QuickFIX Documentation: <A href="http://www.quickfixengine.org/quickfix/doc/html/index.html" target=_blank><a href="http://www.quickfixengine.org/quickfix/doc/html/index.html">http://www.quickfixengine.org/quickfix/doc/html/index.html</a></A><BR>QuickFIX Support: <A href="http://www.quickfixengine.org/services.html" target=_blank><a href="http://www.quickfixengine.org/services.html">http://www.quickfixengine.org/services.html</a></A><BR><BR><BR>Do you know the performace of quickfix ? How many messages per second ? Right<BR>now I didn't put any code in function fromApp and those callbacks (command<BR>out mine code), but I noticed there is above 4 sec delay once I got traffic<BR>above 400 msg/sec. Any ideas about how to improve the performance of<BR>quickfix?<BR><BR>yihan<BR><BR><BR>Djalma Rosa dos Santos Filho wrote:<BR>> <BR>> QuickFIX Documentation:<BR>> <A href="http://www.quickfixengine.org/quickfix/doc/html/index.html" target=_blank><a href="http://www.quickfixengine.org/quickfix/doc/html/index.html">http://www.quickfixengine.org/quickfix/doc/html/index.html</a></A><BR>> QuickFIX Support: <A href="http://www.quickfixengine.org/services.html" target=_blank><a href="http://www.quickfixengine.org/services.html">http://www.quickfixengine.org/services.html</a></A><BR>> <BR>> <BR>> Yes, I have this kind of problem when dealing with hundreds or thousands<BR>> of<BR>> messages per second and most of the times with applications in C# using<BR>> the<BR>> QuickFIX .NET API.<BR>> <BR>> I agree that 10 msg/sec should not be so hard for QF.<BR>> <BR>> On Feb 8, 2008 8:20 PM, Oren Miller <<A onclick="Popup.composeWindow('pcompose.php?sendto=oren%40quickfixengine.org'); return false;" href="#Compose">oren<B></B>@quickfixengine.org</A>> wrote:<BR>> <BR>>> QuickFIX Documentation:<BR>>> <A href="http://www.quickfixengine.org/quickfix/doc/html/index.html" target=_blank><a href="http://www.quickfixengine.org/quickfix/doc/html/index.html">http://www.quickfixengine.org/quickfix/doc/html/index.html</a></A><BR>>> QuickFIX Support: <A href="http://www.quickfixengine.org/services.html" target=_blank><a href="http://www.quickfixengine.org/services.html">http://www.quickfixengine.org/services.html</a></A><BR>>><BR>>><BR>>> Yeah, please keep in mind that as long as you are in your callback,<BR>>> QuickFIX cannot pull the next message. Everything that goes into fromApp<BR>>> becomes part of the message processing, and if the code in there takes<BR>>> time,<BR>>> then QuickFIX is essentially blocked from processing on that thread. If<BR>>> you<BR>>> must do a lot of processing, then stick your message in a queue and get<BR>>> out<BR>>> of there. In the end this probably won't solve your real problem because<BR>>> it<BR>>> indicates you cannot handle the messages as fast as they are being sent<BR>>> to<BR>>> you. If you are building a real time system, this will be an issues for<BR>>> you<BR>>> regardless.<BR>>> --oren<BR>>><BR>>> On Feb 8, 2008, at 3:18 PM, Ted Graham wrote:<BR>>><BR>>> QuickFIX Documentation:<BR>>> <A href="http://www.quickfixengine.org/quickfix/doc/html/index.html" target=_blank><a href="http://www.quickfixengine.org/quickfix/doc/html/index.html">http://www.quickfixengine.org/quickfix/doc/html/index.html</a></A><BR>>> QuickFIX Support: <A href="http://www.quickfixengine.org/services.html" target=_blank><a href="http://www.quickfixengine.org/services.html">http://www.quickfixengine.org/services.html</a></A><BR>>><BR>>> You are talking about 10 msg/sec. At that sending rate, QFJ will have NO<BR>>> trouble keeping up. Look in your code to find what is wrong, QFJ can<BR>>> handle<BR>>> significantly higher message rates that what you are describing.<BR>>><BR>>><BR>>><BR>>> As for C++ QF vs .NET or Java, I did a fair amount of testing and saw<BR>>> negligible differences in speed between C++ QF w/ JNI and the 100% Java<BR>>> QFJ. QFJ may actually be faster, depending on your scenario.<BR>>><BR>>><BR>>><BR>>> Ted<BR>>><BR>>><BR>>> ------------------------------<BR>>><BR>>> *From:* <A onclick="Popup.composeWindow('pcompose.php?sendto=quickfix-users-bounces%40lists.sourceforge.net'); return false;" href="#Compose">quickfix-users-bounces<B></B>@lists.sourceforge.net</A> [<BR>>> mailto:<A onclick="Popup.composeWindow('pcompose.php?sendto=quickfix-users-bounces%40lists.sourceforge.net'); return false;" href="#Compose">quickfix-users-bounces<B></B>@lists.sourceforge.net</A><<A onclick="Popup.composeWindow('pcompose.php?sendto=quickfix-users-bounces%40lists.sourceforge.net'); return false;" href="#Compose">quickfix-users-bounces<B></B>@lists.sourceforge.net</A>>]<BR>>> *On Behalf Of *ying shi<BR>>> *Sent:* Friday, February 08, 2008 1:11 PM<BR>>> *To:* <A onclick="Popup.composeWindow('pcompose.php?sendto=Quickfix-users%40lists.sourceforge.net'); return false;" href="#Compose">Quickfix-users<B></B>@lists.sourceforge.net</A><BR>>> *Subject:* Re: [Quickfix-users] Two more questions about QuickFix<BR>>><BR>>><BR>>><BR>>> Dan:<BR>>><BR>>><BR>>><BR>>> Djalma's analysis is not bad. And I suggest that if you have enough time<BR>>> to trace the bottleneck, you'd better integrate<BR>>><BR>>> QF's source code into your app and build together, so that you can debug<BR>>> and profile inside its code to find it.<BR>>><BR>>><BR>>><BR>>> I would also suggest use C++ QF lib rather than .Net or Java ones.<BR>>><BR>>><BR>>><BR>>> On 08/02/2008, *Djalma Rosa dos Santos Filho* <<A onclick="Popup.composeWindow('pcompose.php?sendto=drsantosfilho%40gmail.com'); return false;" href="#Compose">drsantosfilho<B></B>@gmail.com</A>><BR>>> wrote:<BR>>><BR>>> QuickFIX Documentation:<BR>>> <A href="http://www.quickfixengine.org/quickfix/doc/html/index.html" target=_blank><a href="http://www.quickfixengine.org/quickfix/doc/html/index.html">http://www.quickfixengine.org/quickfix/doc/html/index.html</a></A><BR>>> QuickFIX Support: <A href="http://www.quickfixengine.org/services.html" target=_blank><a href="http://www.quickfixengine.org/services.html">http://www.quickfixengine.org/services.html</a></A><BR>>><BR>>><BR>>><BR>>> Hi,<BR>>><BR>>> This really looks like the TCP buffer is getting full too fast. If<BR>>> possible, try to set a larger TCP window or find and improve the<BR>>> bottleneck<BR>>> in your application or in QuickFIX.<BR>>><BR>>> With my experience, I've noticed that even with a very fast application<BR>>> routine in FromApp, QuickFIX is always faster when sending and much<BR>>> slower<BR>>> when receiving and I believe that this is the main reason for the issue<BR>>> you<BR>>> are reporting, which is not so uncommon for applications that use QF and<BR>>> ocassionally are required to process at a higher rate.<BR>>><BR>>> In part this is a TCP issue, but I believe that the message receiving<BR>>> process in QuickFIX must be optimized at some point in the long road<BR>>> where<BR>>> the bytes travel in the engine.<BR>>><BR>>> {Recv}->{Parser}->{Validation}->[.NET Interop]->[Message<BR>>> Cracker]->{Application Code}<BR>>><BR>>> I have some suspicions, but without a deeper investigation and some<BR>>> profiling I would not dare to cast the first stone.<BR>>><BR>>> Djalma<BR>>><BR>>> On Feb 8, 2008 4:31 PM, Ajay Kamdar <<A onclick="Popup.composeWindow('pcompose.php?sendto=Ajay.Kamdar%40tradeweb.com'); return false;" href="#Compose">Ajay.Kamdar<B></B>@tradeweb.com</A>> wrote:<BR>>><BR>>> QuickFIX Documentation:<BR>>> <A href="http://www.quickfixengine.org/quickfix/doc/html/index.html" target=_blank><a href="http://www.quickfixengine.org/quickfix/doc/html/index.html">http://www.quickfixengine.org/quickfix/doc/html/index.html</a></A><BR>>> QuickFIX Support: <A href="http://www.quickfixengine.org/services.html" target=_blank><a href="http://www.quickfixengine.org/services.html">http://www.quickfixengine.org/services.html</a></A><BR>>><BR>>><BR>>><BR>>> Dan,<BR>>><BR>>> You should also take a look at the application specific processing logic<BR>>> in the fromApp() method and determine how long it takes to run, because<BR>>> it too could be contributing to the slow processing of FIX messages sent<BR>>> by your counter party.<BR>>><BR>>> - Ajay<BR>>><BR>>><BR>>> -----Original Message-----<BR>>> From: <A onclick="Popup.composeWindow('pcompose.php?sendto=quickfix-users-bounces%40lists.sourceforge.net'); return false;" href="#Compose">quickfix-users-bounces<B></B>@lists.sourceforge.net</A><BR>>> [mailto:<A onclick="Popup.composeWindow('pcompose.php?sendto=quickfix-users-bounces%40lists.sourceforge.net'); return false;" href="#Compose">quickfix-users-bounces<B></B>@lists.sourceforge.net</A>] On Behalf Of Dan<BR>>> Guo<BR>>> Sent: Friday, February 08, 2008 1:03 PM<BR>>> To: <A onclick="Popup.composeWindow('pcompose.php?sendto=quickfix-users%40lists.sourceforge.net'); return false;" href="#Compose">quickfix-users<B></B>@lists.sourceforge.net</A><BR>>> Subject: [Quickfix-users] Two more questions about QuickFix<BR>>><BR>>> QuickFIX Documentation:<BR>>> <A href="http://www.quickfixengine.org/quickfix/doc/html/index.html" target=_blank><a href="http://www.quickfixengine.org/quickfix/doc/html/index.html">http://www.quickfixengine.org/quickfix/doc/html/index.html</a></A><BR>>> QuickFIX Support: <A href="http://www.quickfixengine.org/services.html" target=_blank><a href="http://www.quickfixengine.org/services.html">http://www.quickfixengine.org/services.html</a></A><BR>>><BR>>> Hi Everyone!<BR>>><BR>>> we notice two issues and want to hear whether anyone knows what is going<BR>>> on.<BR>>><BR>>> 1. when i sent FIX msgs at higher rate (10 msg per second), there seems<BR>>> a longer latency for the msg to hit market.<BR>>><BR>>> i have strong suspecion QuickFix responsible for the longer latency,<BR>>> as i isolated other segments' contributions to the overall latency.<BR>>><BR>>> can anyone comment on this?<BR>>><BR>>> 2. our broker's network engineers noticed there is TCP back-pressure<BR>>> alert on their site. It basically means the FIX messages got queued on<BR>>> our broker's side. Either because our network connection is not adequate<BR>>> or (and) our QuickFix code cannot process them fast enough.<BR>>><BR>>> did anyone in this list encounter the similiar issue? any comments?<BR>>><BR>>> Thank you in advance.<BR>>><BR>>> Dan<BR>>><BR>>> ------------------------------------------------------------------------<BR>>> -<BR>>> This <a href="http://SF.net">SF.net</a> email is sponsored by: Microsoft<BR>>> Defy all challenges. Microsoft(R) Visual Studio 2008.<BR>>> <A href="http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/" target=_blank><a href="http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/">http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/</a></A><BR>>> _______________________________________________<BR>>> Quickfix-users mailing list<BR>>> <A onclick="Popup.composeWindow('pcompose.php?sendto=Quickfix-users%40lists.sourceforge.net'); return false;" href="#Compose">Quickfix-users<B></B>@lists.sourceforge.net</A><BR>>> <A href="https://lists.sourceforge.net/lists/listinfo/quickfix-users" target=_blank><a href="https://lists.sourceforge.net/lists/listinfo/quickfix-users">https://lists.sourceforge.net/lists/listinfo/quickfix-users</a></A><BR>>><BR>>> -------------------------------------------------------------------------<BR>>> This <a href="http://SF.net">SF.net</a> email is sponsored by: Microsoft<BR>>> Defy all challenges. Microsoft(R) Visual Studio 2008.<BR>>> <A href="http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/" target=_blank><a href="http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/">http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/</a></A><BR>>> _______________________________________________<BR>>> Quickfix-users mailing list<BR>>> <A onclick="Popup.composeWindow('pcompose.php?sendto=Quickfix-users%40lists.sourceforge.net'); return false;" href="#Compose">Quickfix-users<B></B>@lists.sourceforge.net</A><BR>>> <A href="https://lists.sourceforge.net/lists/listinfo/quickfix-users" target=_blank><a href="https://lists.sourceforge.net/lists/listinfo/quickfix-users">https://lists.sourceforge.net/lists/listinfo/quickfix-users</a></A><BR>>><BR>>><BR>>><BR>>><BR>>> -------------------------------------------------------------------------<BR>>> This <a href="http://SF.net">SF.net</a> email is sponsored by: Microsoft<BR>>> Defy all challenges. Microsoft(R) Visual Studio 2008.<BR>>> <A href="http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/" target=_blank><a href="http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/">http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/</a></A><BR>>> _______________________________________________<BR>>> Quickfix-users mailing list<BR>>> <A onclick="Popup.composeWindow('pcompose.php?sendto=Quickfix-users%40lists.sourceforge.net'); return false;" href="#Compose">Quickfix-users<B></B>@lists.sourceforge.net</A><BR>>> <A href="https://lists.sourceforge.net/lists/listinfo/quickfix-users" target=_blank><a href="https://lists.sourceforge.net/lists/listinfo/quickfix-users">https://lists.sourceforge.net/lists/listinfo/quickfix-users</a></A><BR>>><BR>>><BR>>> -------------------------------------------------------------------------<BR>>> This <a href="http://SF.net">SF.net</a> email is sponsored by: Microsoft<BR>>> Defy all challenges. Microsoft(R) Visual Studio 2008.<BR>>><BR>>> <A href="http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/_______________________________________________" target=_blank><a href="http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/_______________________________________________">http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/_______________________________________________</a></A><BR>>> Quickfix-users mailing list<BR>>> <A onclick="Popup.composeWindow('pcompose.php?sendto=Quickfix-users%40lists.sourceforge.net'); return false;" href="#Compose">Quickfix-users<B></B>@lists.sourceforge.net</A><BR>>> <A href="https://lists.sourceforge.net/lists/listinfo/quickfix-users" target=_blank><a href="https://lists.sourceforge.net/lists/listinfo/quickfix-users">https://lists.sourceforge.net/lists/listinfo/quickfix-users</a></A><BR>>><BR>>><BR>>><BR>>> -------------------------------------------------------------------------<BR>>> This <a href="http://SF.net">SF.net</a> email is sponsored by: Microsoft<BR>>> Defy all challenges. Microsoft(R) Visual Studio 2008.<BR>>> <A href="http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/" target=_blank><a href="http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/">http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/</a></A><BR>>> _______________________________________________<BR>>> Quickfix-users mailing list<BR>>> <A onclick="Popup.composeWindow('pcompose.php?sendto=Quickfix-users%40lists.sourceforge.net'); return false;" href="#Compose">Quickfix-users<B></B>@lists.sourceforge.net</A><BR>>> <A href="https://lists.sourceforge.net/lists/listinfo/quickfix-users" target=_blank><a href="https://lists.sourceforge.net/lists/listinfo/quickfix-users">https://lists.sourceforge.net/lists/listinfo/quickfix-users</a></A><BR>>><BR>>><BR>> <BR>> -------------------------------------------------------------------------<BR>> This <a href="http://SF.net">SF.net</a> email is sponsored by: Microsoft<BR>> Defy all challenges. Microsoft(R) Visual Studio 2008.<BR>> <A href="http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/" target=_blank><a href="http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/">http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/</a></A><BR>> _______________________________________________<BR>> Quickfix-users mailing list<BR>> <A onclick="Popup.composeWindow('pcompose.php?sendto=Quickfix-users%40lists.sourceforge.net'); return false;" href="#Compose">Quickfix-users<B></B>@lists.sourceforge.net</A><BR>> <A href="https://lists.sourceforge.net/lists/listinfo/quickfix-users" target=_blank><a href="https://lists.sourceforge.net/lists/listinfo/quickfix-users">https://lists.sourceforge.net/lists/listinfo/quickfix-users</a></A><BR>> <BR>> <BR><BR>-- <BR>View this message in context: <A href="http://www.nabble.com/Two-more-questions-about-QuickFix-tp15360781p16349797.html" target=_blank><a href="http://www.nabble.com/Two-more-questions-about-QuickFix-tp15360781p16349797.html">http://www.nabble.com/Two-more-questions-about-QuickFix-tp15360781p16349797.html</a></A><BR>Sent from the QuickFIX - User mailing list archive at <a href="http://Nabble.com">Nabble.com</a>.<BR><BR><BR>-------------------------------------------------------------------------<BR>Check out the new <a href="http://SourceForge.net">SourceForge.net</a> Marketplace.<BR>It's the best place to buy or sell services for<BR>just about anything Open Source.<BR><A href="http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace" target=_blank><a href="http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace">http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace</a></A><BR>_______________________________________________<BR>Quickfix-users mailing list<BR><A onclick="Popup.composeWindow('pcompose.php?sendto=Quickfix-users%40lists.sourceforge.net'); return false;" href="#Compose">Quickfix-users<B></B>@lists.sourceforge.net</A><BR><A href="https://lists.sourceforge.net/lists/listinfo/quickfix-users" target=_blank><a href="https://lists.sourceforge.net/lists/listinfo/quickfix-users">https://lists.sourceforge.net/lists/listinfo/quickfix-users</a></A><BR></BLOCKQUOTE></body></html> |
|
From: Djalma R. d. S. F. <drs...@gm...> - 2008-02-08 19:40:31
|
Hi,
This really looks like the TCP buffer is getting full too fast. If possible,
try to set a larger TCP window or find and improve the bottleneck in your
application or in QuickFIX.
With my experience, I've noticed that even with a very fast application
routine in FromApp, QuickFIX is always faster when sending and much slower
when receiving and I believe that this is the main reason for the issue you
are reporting, which is not so uncommon for applications that use QF and
ocassionally are required to process at a higher rate.
In part this is a TCP issue, but I believe that the message receiving
process in QuickFIX must be optimized at some point in the long road where
the bytes travel in the engine.
{Recv}->{Parser}->{Validation}->[.NET Interop]->[Message
Cracker]->{Application Code}
I have some suspicions, but without a deeper investigation and some
profiling I would not dare to cast the first stone.
Djalma
On Feb 8, 2008 4:31 PM, Ajay Kamdar <Aja...@tr...> wrote:
> QuickFIX Documentation:
> http://www.quickfixengine.org/quickfix/doc/html/index.html
> QuickFIX Support: http://www.quickfixengine.org/services.html
>
> Dan,
>
> You should also take a look at the application specific processing logic
> in the fromApp() method and determine how long it takes to run, because
> it too could be contributing to the slow processing of FIX messages sent
> by your counter party.
>
> - Ajay
>
> -----Original Message-----
> From: qui...@li...
> [mailto:qui...@li...] On Behalf Of Dan
> Guo
> Sent: Friday, February 08, 2008 1:03 PM
> To: qui...@li...
> Subject: [Quickfix-users] Two more questions about QuickFix
>
> QuickFIX Documentation:
> http://www.quickfixengine.org/quickfix/doc/html/index.html
> QuickFIX Support: http://www.quickfixengine.org/services.html
>
> Hi Everyone!
>
> we notice two issues and want to hear whether anyone knows what is going
> on.
>
> 1. when i sent FIX msgs at higher rate (10 msg per second), there seems
> a longer latency for the msg to hit market.
>
> i have strong suspecion QuickFix responsible for the longer latency,
> as i isolated other segments' contributions to the overall latency.
>
> can anyone comment on this?
>
> 2. our broker's network engineers noticed there is TCP back-pressure
> alert on their site. It basically means the FIX messages got queued on
> our broker's side. Either because our network connection is not adequate
> or (and) our QuickFix code cannot process them fast enough.
>
> did anyone in this list encounter the similiar issue? any comments?
>
> Thank you in advance.
>
> Dan
>
> ------------------------------------------------------------------------
> -
> This SF.net email is sponsored by: Microsoft
> Defy all challenges. Microsoft(R) Visual Studio 2008.
> http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
> _______________________________________________
> Quickfix-users mailing list
> Qui...@li...
> https://lists.sourceforge.net/lists/listinfo/quickfix-users
>
> -------------------------------------------------------------------------
> This SF.net email is sponsored by: Microsoft
> Defy all challenges. Microsoft(R) Visual Studio 2008.
> http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
> _______________________________________________
> Quickfix-users mailing list
> Qui...@li...
> https://lists.sourceforge.net/lists/listinfo/quickfix-users
>
|
|
From: ying s. <sy8...@gm...> - 2008-02-08 20:11:15
|
Dan: Djalma's analysis is not bad. And I suggest that if you have enough time to trace the bottleneck, you'd better integrate QF's source code into your app and build together, so that you can debug and profile inside its code to find it. I would also suggest use C++ QF lib rather than .Net or Java ones. On 08/02/2008, Djalma Rosa dos Santos Filho <drs...@gm...> wrote: > > QuickFIX Documentation: > http://www.quickfixengine.org/quickfix/doc/html/index.html > QuickFIX Support: http://www.quickfixengine.org/services.html > > > Hi, > > This really looks like the TCP buffer is getting full too fast. If > possible, try to set a larger TCP window or find and improve the bottleneck > in your application or in QuickFIX. > > With my experience, I've noticed that even with a very fast application > routine in FromApp, QuickFIX is always faster when sending and much slower > when receiving and I believe that this is the main reason for the issue you > are reporting, which is not so uncommon for applications that use QF and > ocassionally are required to process at a higher rate. > > In part this is a TCP issue, but I believe that the message receiving > process in QuickFIX must be optimized at some point in the long road where > the bytes travel in the engine. > > {Recv}->{Parser}->{Validation}->[.NET Interop]->[Message > Cracker]->{Application Code} > > I have some suspicions, but without a deeper investigation and some > profiling I would not dare to cast the first stone. > > Djalma > > > On Feb 8, 2008 4:31 PM, Ajay Kamdar <Aja...@tr...> wrote: > > > QuickFIX Documentation: > > http://www.quickfixengine.org/quickfix/doc/html/index.html > > QuickFIX Support: http://www.quickfixengine.org/services.html > > > > > > Dan, > > > > You should also take a look at the application specific processing logic > > in the fromApp() method and determine how long it takes to run, because > > it too could be contributing to the slow processing of FIX messages sent > > by your counter party. > > > > - Ajay > > > > -----Original Message----- > > From: qui...@li... > > [mailto:qui...@li...] On Behalf Of Dan > > Guo > > Sent: Friday, February 08, 2008 1:03 PM > > To: qui...@li... > > Subject: [Quickfix-users] Two more questions about QuickFix > > > > QuickFIX Documentation: > > http://www.quickfixengine.org/quickfix/doc/html/index.html > > QuickFIX Support: http://www.quickfixengine.org/services.html > > > > Hi Everyone! > > > > we notice two issues and want to hear whether anyone knows what is going > > on. > > > > 1. when i sent FIX msgs at higher rate (10 msg per second), there seems > > a longer latency for the msg to hit market. > > > > i have strong suspecion QuickFix responsible for the longer latency, > > as i isolated other segments' contributions to the overall latency. > > > > can anyone comment on this? > > > > 2. our broker's network engineers noticed there is TCP back-pressure > > alert on their site. It basically means the FIX messages got queued on > > our broker's side. Either because our network connection is not adequate > > or (and) our QuickFix code cannot process them fast enough. > > > > did anyone in this list encounter the similiar issue? any comments? > > > > Thank you in advance. > > > > Dan > > > > ------------------------------------------------------------------------ > > - > > This SF.net email is sponsored by: Microsoft > > Defy all challenges. Microsoft(R) Visual Studio 2008. > > http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ > > _______________________________________________ > > Quickfix-users mailing list > > Qui...@li... > > https://lists.sourceforge.net/lists/listinfo/quickfix-users > > > > > > ------------------------------------------------------------------------- > > This SF.net email is sponsored by: Microsoft > > Defy all challenges. Microsoft(R) Visual Studio 2008. > > http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ > > _______________________________________________ > > Quickfix-users mailing list > > Qui...@li... > > https://lists.sourceforge.net/lists/listinfo/quickfix-users > > > > > > > ------------------------------------------------------------------------- > This SF.net email is sponsored by: Microsoft > Defy all challenges. Microsoft(R) Visual Studio 2008. > http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ > _______________________________________________ > Quickfix-users mailing list > Qui...@li... > https://lists.sourceforge.net/lists/listinfo/quickfix-users > > |
|
From: Ted G. <tg...@Co...> - 2008-02-08 21:18:31
|
You are talking about 10 msg/sec. At that sending rate, QFJ will have NO trouble keeping up. Look in your code to find what is wrong, QFJ can handle significantly higher message rates that what you are describing. As for C++ QF vs .NET or Java, I did a fair amount of testing and saw negligible differences in speed between C++ QF w/ JNI and the 100% Java QFJ. QFJ may actually be faster, depending on your scenario. Ted _____ From: qui...@li... [mailto:qui...@li...] On Behalf Of ying shi Sent: Friday, February 08, 2008 1:11 PM To: Qui...@li... Subject: Re: [Quickfix-users] Two more questions about QuickFix Dan: Djalma's analysis is not bad. And I suggest that if you have enough time to trace the bottleneck, you'd better integrate QF's source code into your app and build together, so that you can debug and profile inside its code to find it. I would also suggest use C++ QF lib rather than .Net or Java ones. On 08/02/2008, Djalma Rosa dos Santos Filho <drs...@gm...> wrote: QuickFIX Documentation: http://www.quickfixengine.org/quickfix/doc/html/index.html QuickFIX Support: http://www.quickfixengine.org/services.html Hi, This really looks like the TCP buffer is getting full too fast. If possible, try to set a larger TCP window or find and improve the bottleneck in your application or in QuickFIX. With my experience, I've noticed that even with a very fast application routine in FromApp, QuickFIX is always faster when sending and much slower when receiving and I believe that this is the main reason for the issue you are reporting, which is not so uncommon for applications that use QF and ocassionally are required to process at a higher rate. In part this is a TCP issue, but I believe that the message receiving process in QuickFIX must be optimized at some point in the long road where the bytes travel in the engine. {Recv}->{Parser}->{Validation}->[.NET Interop]->[Message Cracker]->{Application Code} I have some suspicions, but without a deeper investigation and some profiling I would not dare to cast the first stone. Djalma On Feb 8, 2008 4:31 PM, Ajay Kamdar <Aja...@tr...> wrote: QuickFIX Documentation: http://www.quickfixengine.org/quickfix/doc/html/index.html QuickFIX Support: http://www.quickfixengine.org/services.html Dan, You should also take a look at the application specific processing logic in the fromApp() method and determine how long it takes to run, because it too could be contributing to the slow processing of FIX messages sent by your counter party. - Ajay -----Original Message----- From: qui...@li... [mailto:qui...@li...] On Behalf Of Dan Guo Sent: Friday, February 08, 2008 1:03 PM To: qui...@li... Subject: [Quickfix-users] Two more questions about QuickFix QuickFIX Documentation: http://www.quickfixengine.org/quickfix/doc/html/index.html QuickFIX Support: http://www.quickfixengine.org/services.html Hi Everyone! we notice two issues and want to hear whether anyone knows what is going on. 1. when i sent FIX msgs at higher rate (10 msg per second), there seems a longer latency for the msg to hit market. i have strong suspecion QuickFix responsible for the longer latency, as i isolated other segments' contributions to the overall latency. can anyone comment on this? 2. our broker's network engineers noticed there is TCP back-pressure alert on their site. It basically means the FIX messages got queued on our broker's side. Either because our network connection is not adequate or (and) our QuickFix code cannot process them fast enough. did anyone in this list encounter the similiar issue? any comments? Thank you in advance. Dan ------------------------------------------------------------------------ - This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ _______________________________________________ Quickfix-users mailing list Qui...@li... https://lists.sourceforge.net/lists/listinfo/quickfix-users ------------------------------------------------------------------------- This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ _______________________________________________ Quickfix-users mailing list Qui...@li... https://lists.sourceforge.net/lists/listinfo/quickfix-users ------------------------------------------------------------------------- This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ _______________________________________________ Quickfix-users mailing list Qui...@li... https://lists.sourceforge.net/lists/listinfo/quickfix-users |
|
From: Oren M. <or...@qu...> - 2008-02-08 22:19:02
|
Yeah, please keep in mind that as long as you are in your callback, QuickFIX cannot pull the next message. Everything that goes into fromApp becomes part of the message processing, and if the code in there takes time, then QuickFIX is essentially blocked from processing on that thread. If you must do a lot of processing, then stick your message in a queue and get out of there. In the end this probably won't solve your real problem because it indicates you cannot handle the messages as fast as they are being sent to you. If you are building a real time system, this will be an issues for you regardless. --oren On Feb 8, 2008, at 3:18 PM, Ted Graham wrote: > QuickFIX Documentation: http://www.quickfixengine.org/quickfix/doc/ > html/index.html > QuickFIX Support: http://www.quickfixengine.org/services.html > > You are talking about 10 msg/sec. At that sending rate, QFJ will > have NO trouble keeping up. Look in your code to find what is > wrong, QFJ can handle significantly higher message rates that what > you are describing. > > > > As for C++ QF vs .NET or Java, I did a fair amount of testing and > saw negligible differences in speed between C++ QF w/ JNI and the > 100% Java QFJ. QFJ may actually be faster, depending on your > scenario. > > > > Ted > > > > From: qui...@li... [mailto:quickfix- > use...@li...] On Behalf Of ying shi > Sent: Friday, February 08, 2008 1:11 PM > To: Qui...@li... > Subject: Re: [Quickfix-users] Two more questions about QuickFix > > > > Dan: > > > > Djalma's analysis is not bad. And I suggest that if you have enough > time to trace the bottleneck, you'd better integrate > > QF's source code into your app and build together, so that you can > debug and profile inside its code to find it. > > > > I would also suggest use C++ QF lib rather than .Net or Java ones. > > > > On 08/02/2008, Djalma Rosa dos Santos Filho > <drs...@gm...> wrote: > > QuickFIX Documentation: http://www.quickfixengine.org/quickfix/doc/ > html/index.html > QuickFIX Support: http://www.quickfixengine.org/services.html > > > > > Hi, > > This really looks like the TCP buffer is getting full too fast. If > possible, try to set a larger TCP window or find and improve the > bottleneck in your application or in QuickFIX. > > With my experience, I've noticed that even with a very fast > application routine in FromApp, QuickFIX is always faster when > sending and much slower when receiving and I believe that this is > the main reason for the issue you are reporting, which is not so > uncommon for applications that use QF and ocassionally are required > to process at a higher rate. > > In part this is a TCP issue, but I believe that the message > receiving process in QuickFIX must be optimized at some point in > the long road where the bytes travel in the engine. > > {Recv}->{Parser}->{Validation}->[.NET Interop]->[Message Cracker]-> > {Application Code} > > I have some suspicions, but without a deeper investigation and some > profiling I would not dare to cast the first stone. > > Djalma > > > On Feb 8, 2008 4:31 PM, Ajay Kamdar <Aja...@tr...> wrote: > > QuickFIX Documentation: http://www.quickfixengine.org/quickfix/doc/ > html/index.html > QuickFIX Support: http://www.quickfixengine.org/services.html > > > > Dan, > > You should also take a look at the application specific processing > logic > in the fromApp() method and determine how long it takes to run, > because > it too could be contributing to the slow processing of FIX messages > sent > by your counter party. > > - Ajay > > > -----Original Message----- > From: qui...@li... > [mailto:qui...@li...] On Behalf Of Dan > Guo > Sent: Friday, February 08, 2008 1:03 PM > To: qui...@li... > Subject: [Quickfix-users] Two more questions about QuickFix > > QuickFIX Documentation: > http://www.quickfixengine.org/quickfix/doc/html/index.html > QuickFIX Support: http://www.quickfixengine.org/services.html > > Hi Everyone! > > we notice two issues and want to hear whether anyone knows what is > going > on. > > 1. when i sent FIX msgs at higher rate (10 msg per second), there > seems > a longer latency for the msg to hit market. > > i have strong suspecion QuickFix responsible for the longer latency, > as i isolated other segments' contributions to the overall latency. > > can anyone comment on this? > > 2. our broker's network engineers noticed there is TCP back-pressure > alert on their site. It basically means the FIX messages got queued on > our broker's side. Either because our network connection is not > adequate > or (and) our QuickFix code cannot process them fast enough. > > did anyone in this list encounter the similiar issue? any comments? > > Thank you in advance. > > Dan > > ---------------------------------------------------------------------- > -- > - > This SF.net email is sponsored by: Microsoft > Defy all challenges. Microsoft(R) Visual Studio 2008. > http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ > _______________________________________________ > Quickfix-users mailing list > Qui...@li... > https://lists.sourceforge.net/lists/listinfo/quickfix-users > > ---------------------------------------------------------------------- > --- > This SF.net email is sponsored by: Microsoft > Defy all challenges. Microsoft(R) Visual Studio 2008. > http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ > _______________________________________________ > Quickfix-users mailing list > Qui...@li... > https://lists.sourceforge.net/lists/listinfo/quickfix-users > > > > > ---------------------------------------------------------------------- > --- > This SF.net email is sponsored by: Microsoft > Defy all challenges. Microsoft(R) Visual Studio 2008. > http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ > _______________________________________________ > Quickfix-users mailing list > Qui...@li... > https://lists.sourceforge.net/lists/listinfo/quickfix-users > > > > ---------------------------------------------------------------------- > --- > This SF.net email is sponsored by: Microsoft > Defy all challenges. Microsoft(R) Visual Studio 2008. > http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ > _______________________________________________ > Quickfix-users mailing list > Qui...@li... > https://lists.sourceforge.net/lists/listinfo/quickfix-users |
×
Want the latest updates on software, tech news, and AI?
Get latest updates about software, tech news, and AI from SourceForge directly in your inbox once a month.
Thanks for helping keep SourceForge clean.
X