Thread: [Quickfix-developers] Missing FIX messages in Initiator?
Brought to you by:
orenmnero
Menu
▾
▴
quickfix-developers
|
From: Edde <edd...@gm...> - 2005-07-21 16:46:46
|
Hi Guys, During the last couple of weeks we've been experiencing problems in the FIX communication with our broker firm. Based upon the QuickFix logs of the FIX transmissons the problem is only appearing when we're sending FIX messages to the broker firm (Initiator --> Acceptor). Usually the problem is that it takes a long time for our messages to arrive at the broker firm (sometimes 30s or more) which is very frustrating when you for example want to insert an order to the market. We've been doing extensive network monitoring and we can't find any problems in the network connection at either end. We're running all FIX traffic through a VPN (SecuRemote) and we're suspecting that this might have something to do with the problem but we're not sure. Does anyone on this list have any clues to what might be the problem here? And then yesterday and again today the Acceptor actually terminated the connection since it didn't recieve any messages from our Initiator. I've put together a list of the relevant FIX messages (truncated after Tag 35). "EDDE" is the Initiator (QuickFIX) and "FIP" is the acceptor (I think they're using a self developped FIX Engine): *EDDE -> FIP 09:06:53.349 - 8=3DFIX.4.29=3D17335=3DD (InsertOrde= rSingle) *FIP -> EDDE 09:06:59.739 - 8=3DFIX.4.29=3D34435=3D8 (ExecutionR= eport) *EDDE -> FIP 09:07:23.067 - 8=3DFIX.4.29=3D5135=3D0 (Heartbeat) *EDDE -> FIP 09:07:24.270 - 8=3DFIX.4.29=3D18835=3DD (InsertOrde= rSingle) *EDDE -> FIP 09:07:29.333 - 8=3DFIX.4.29=3D18835=3DD (InsertOrde= rSingle) *FIP -> EDDE 09:07:30.708 - 8=3DFIX.4.29=3D5235=3D0 (Heartbeat) *EDDE -> FIP 09:07:37.677 - 8=3DFIX.4.29=3D17935=3DD (InsertOrde= rSingle) *EDDE -> FIP 09:07:42.677 - 8=3DFIX.4.29=3D17735=3DD (InsertOrde= rSingle) *EDDE -> FIP 09:07:47.677 - 8=3DFIX.4.29=3D17735=3DD (InsertOrde= rSingle) *FIP -> EDDE 09:08:00.708 - 8=3DFIX.4.2=019=3D78=0135=3D1=0134=3D113=0149=3DFIP=0152=3D20050719-07:07:5= 9.001=0156=3DEDDE=01112=3D20050719-07:07:59.000=0110=3D057=01 (TestRequest) *EDDE -> FIP 09:08:00.708 - 8=3DFIX.4.2=019=3D77=0135=3D0=0134=3D96=0149=3DEDDE=0152=3D20050719-07:08:0= 0.708=0156=3DFIP=01112=3D20050719-07:07:59.000=0110=3D018=01 (Response) *EDDE -> FIP 09:08:00.958 - 8=3DFIX.4.29=3D19035=3DD (InsertOrde= rSingle) *EDDE -> FIP 09:08:05.958 - 8=3DFIX.4.29=3D17635=3DD (InsertOrde= rSingle) *EDDE -> FIP 09:08:18.005 - 8=3DFIX.4.29=3D17635=3DD (InsertOrde= rSingle) *FIP -> EDDE 09:08:30.708 - 8=3DFIX.4.2=019=3D109=0135=3D5=0134=3D114=0149=3DFIP=0152=3D20050719-07:08:= 29.001=0156=3DEDDE=0158=3DDidn't receive message for a long time. Disconnected.=0110=3D005=01 (Logout) *EDDE -> FIP 09:08:30.708 - 8=3DFIX.4.29=3D5235=3D5 (Logout) *EDDE -> FIP 09:08:49.817 - 8=3DFIX.4.29=3D6435=3DA (Logon failu= re) *EDDE -> FIP 09:09:09.145 - 8=3DFIX.4.29=3D6435=3DA (Logon) *FIP -> EDDE 09:09:10.474 - 8=3DFIX.4.29=3D6435=3DA (Logon) After this we're doing the proper resends off messages etc. Based on the logs the communication didn't work in the EDDE --> FIP direction between 09:06:55 and 09:09:10 which is just over 2 mins. The point is that the communication works flawlessly in one direction (FIP --> EDDE) while all the messages going the other direction just doesn't reach the target. From what I understand the FIX protocol doesn't acknowledge message delivery so this is ok. What I'm trying to figure out is where these messages go? Shouldn't there be a socket exception or something if the message can't be delivered to the target socket? Another important part of this is that we're running two sessions in the system, one for MarketData and one for Order handling. The problem only occurs on the Order session while the communication on the MarketData session works just fine. I think this rules out any problems with the actual network but I'm not sure. Our system is running on Windows XP using a Java application and the QuickFIX/JNI solution (1.9.4). We're using VPN (SecuRemote) to assure a safe connection with our broker. If anyone have any ideas what may cause these problems please let me know. Thanks, /Eddie Ps. I'm very excited about the pure Java implementation of QuickFIX. Great work!!! |
|
From: Caleb E. <cal...@gm...> - 2005-07-21 17:27:47
|
On 7/21/05, Edde <edd...@gm...> wrote: > Another important part of this is that we're running two sessions in > the system, one for MarketData and one for Order handling. The problem > only occurs on the Order session while the communication on the > MarketData session works just fine. I think this rules out any > problems with the actual network but I'm not sure. This is all going through the same VPN tunnel? What is the bandwidth of that connection? Perhaps the market data session is using so much of your bandwidth that its "crowding out" your Order session. --=20 Caleb Epstein caleb dot epstein at gmail dot com |
|
From: Joerg T. <Joe...@ma...> - 2005-07-21 17:52:05
|
Caleb Epstein wrote:
> On 7/21/05, Edde <edd...@gm...> wrote:
>
>>Another important part of this is that we're running two sessions in
>>the system, one for MarketData and one for Order handling. The problem
>>only occurs on the Order session while the communication on the
>>MarketData session works just fine. I think this rules out any
>>problems with the actual network but I'm not sure.
>
> This is all going through the same VPN tunnel? What is the bandwidth
> of that connection? Perhaps the market data session is using so much
> of your bandwidth that its "crowding out" your Order session.
Another common issues is the MTU of the physical interface. Sometimes the end-to-end MTU
cannot be discovered correctly. If the MTU is too large, the VPN connection can get stuck
on large packet size. In a terminal session you notice this, if commands with only a few
lines of output go well, but larger output as created by vi, ls of large dirs or less gets
stuck.
In summary, I do not think its QF related, but has to do with your VPN connection.
BTW, which QF version do you use?
Cheers, Jörg
--
Joerg Thoennes
http://macd.com
Tel.: +49 (0)241 44597-24 Macdonald Associates GmbH
Fax : +49 (0)241 44597-10 Lothringer Str. 52, D-52070 Aachen
|
|
From: Edde <edd...@gm...> - 2005-07-22 08:13:37
|
Hi J=F6rg, Thanks for the quick response! > Another common issues is the MTU of the physical interface. Sometimes the= end-to- >end MTU cannot be discovered correctly. If the MTU is too large, the VPN connection >can get stuck on large packet size. In a terminal session you notice this, if commands >with only a few lines of output go well, but larger output as created by vi, ls of large dirs >or less gets stuck. By MTU you mean, Maximum Transmission Unit? (I had to look it up on Google = ;-)) So, does this mean that the VPN connection can be limited to a certain MTU and if a unit arrives with a bigger size the VPN turns into a bottleneck? =20 > In summary, I do not think its QF related, but has to do with your VPN co= nnection. That's my conclusion as well and also Caleb seems to think so. > BTW, which QF version do you use? I'm running a Java application so we're using the QuickFIX/JNI solution. QuickFIX Version 1.9.4. I plan to migrate to QuickFIX/J in the near future. Thanks, /Eddie >=20 > Cheers, J=F6rg >=20 > -- > Joerg Thoennes > http://macd.com > Tel.: +49 (0)241 44597-24 Macdonald Associates GmbH > Fax : +49 (0)241 44597-10 Lothringer Str. 52, D-52070 Aachen > |
|
From: Joerg T. <Joe...@ma...> - 2005-07-22 08:39:21
|
Edde wrote:
> By MTU you mean, Maximum Transmission Unit? (I had to look it up on Google ;-))
Sorry, I was a bit in a hurry. Probably you also read some explanations about MTU...
> So, does this mean that the VPN connection can be limited to a certain
> MTU and if a unit arrives with a bigger size the VPN turns into a
> bottleneck?
My knowledge is rather vague here, any network export may correct me if I tell nonsense...
An IP packet crosses different networks (ethernet, ADSL, ATM etc etc) and every network
has a restriction known as MTU. There are is also something like MTU discovery to get MTU
which fits all crossing nets. Otherwise, some intermediate routers may have to fragment
the packet (and later re-assemble it). Nowadays, in a world of firewalls and security
concerns this does not seem to work very well. E.g. if the network admin disables all ICMP
messages (MTU discovery is an ICMP message).
A VPN is a virtual network on top of a physical network. I.e. all packets are wrapped into
network headers (and trailers) for this virtual network. But the physical space for
payload (the MTU) stays the same, so the MTU possible over the virtual network is physical
MTU - virtual header size - virtual trailer size.
If your normal MTU is 1500 (ethernet), and your VPN wrapper creates e.g. 200 bytes of
overhead per packet, setting the MTU to 1300 (1500-200) may cure the kind of problems
described.
Actually this is a hack to workaround bad router and network implementations somewhere on
the way through the internet, but it works for us.
Cheers, Jörg
--
Joerg Thoennes
http://macd.com
Tel.: +49 (0)241 44597-24 Macdonald Associates GmbH
Fax : +49 (0)241 44597-10 Lothringer Str. 52, D-52070 Aachen
|
|
From: Edde <edd...@gm...> - 2005-07-22 06:49:15
|
Hi Caleb, Thanks for the quick response! =20 > This is all going through the same VPN tunnel? What is the bandwidth > of that connection? Perhaps the market data session is using so much > of your bandwidth that its "crowding out" your Order session. Yes, at the moment all communication goes through the VPN tunnel. My theory about the communications problem with the Order session is also related to the VPN tunnel and both you and Joerg seems to think so as well. Now all I have to do is convince our broker firm that it most likely is the VPN which are causing these problems. I tried to suggest that we could do a test without the VPN but apparently that's too much of a security risk which I guess is fair enough. However, you raised a good point about both the MarketData and Order traffic going through the same VPN tunnel. As far as I can see there would be no risk of moving the MarketData traffic outside of the VPN would there? I mean this is just market data information, I wouldn't care if anyone is eavesdropping on this communication. I'll check back with my broker and see if we can try this. Thanks! /Eddie >=20 > -- > Caleb Epstein > caleb dot epstein at gmail dot com > |
