Thread: [Quickfix-developers] Waiting for automatic ResendRequest to finish?
Brought to you by:
orenmnero
Menu
▾
▴
quickfix-developers
|
From: Edde <edd...@gm...> - 2006-02-20 13:39:00
|
Hi Guys, I'm using the QuickFix JNI library version 1.9.4 (plan to upgrade when QuickFix/J moves to 1.0.0) and I was wondering if there is a way to detect when all messages have been received from an automatic ResendRequest? I experienced a problem with this last week in our production system when our counterparty's system went down for about an hour. After they had rebooted their system and we could log on again QuickFix did the automatic ResendRequest to get any messages missed during the downtime: 20060216-14:26:00 : 15:26:00.350 - MsgSeqNum too high, expecting 13817 but received 66426 EDDE -> 15:26:00.360 - 8=3D FIX.4.2=019=3D67=0135=3D2=0134=3D32889=0149=3DEDDE=0152=3D20060216-14:26:00= .350=0156=3DXXXX=017=3D13817=0116=3D0=0110=3D235=01 Since it needed to resend 50000 messages this takes some time and while thi= s is in progress our system continues the startup process. I'm planning to rewrite this code ASAP but at the moment what we do after a major crash on either end is to clear all our databases and send a ResendRequest demanding all messages sent in the session: EDDE -> 15:28:10.177 - 8=3D FIX.4.2=019=3D67=0135=3D2=0134=3D32910=0149=3DEDDE=0152=3D20060216-14:28:10= .157=0156=3DXXXX=017=3D1=0116=3D0=0110=3D228=01 The problem this time is that since I'm still receiving messages from the first ResendRequest code I will get an unexpected sequence number as the answer to my latest ResendRequest: XXXX -> 15:28:10.177 - 8=3D FIX.4.2=019=3D375=0135=3D8=0134=3D56802=0143=3DY=0149=3DXXXX=0152=3D2006021= 6-14:28:08.234..... 20060216-14:28:10 : 15:28:10.177 - MsgSeqNum too high, expecting 2 but received 56802 20060216-14:28:10 : 15:28:10.177 - Sent ResendRequest FROM: 2 TO: 0 EDDE -> 15:28:10.177 - 8=3D FIX.4.2=019=3D63=0135=3D2=0134=3D32911=0149=3DEDDE=0152=3D20060216-14:28:10= .177=0156=3DXXXX=017=3D2=0116=3D0=0110=3D017=01 So, for each message I receive between 56802 and 66426 QuickFIX automatically sends a new ResendRequest demanding all messages in the session. Obviously this is not good behaviour and I could work around this if there is a way to know when the first ResendRequest has received all the messages it's expecting. Anyone know if there is a way to detect this? Thanks, /Eddie I've put together a snapshot of the session logs to illustrate the problem (XXXX is my counterparty): 20060216-14:25:56 : 15:25:56.234 - Created session 20060216-14:25:56 : 15:25:56.415 - Connecting to 192.42.172.11 on port 7474 20060216-14:25:56 : 15:25:56.475 - Connection succeeded EDDE -> 15:25:56.565 - 8=3D FIX.4.2=019=3D67=0135=3DA=0134=3D32888=0149=3DEDDE=0152=3D20060216-14:25:56= .495=0156=3DXXXX=0198=3D0=01108=3D180=0110=3D014=01 20060216-14:25:56 : 15:25:56.565 - Initiated logon request XXXX -> 15:26:00.350 - 8=3D FIX.4.2=019=3D67=0135=3DA=0134=3D66426=0149=3DXXXX=0152=3D20060216-14:25:59= .177=0156=3DEDDE=0198=3D0=01108=3D180=0110=3D009=01 20060216-14:26:00 : 15:26:00.350 - Received logon response 20060216-14:26:00 : 15:26:00.350 - MsgSeqNum too high, expecting 13817 but received 66426 EDDE -> 15:26:00.360 - 8=3D FIX.4.2=019=3D67=0135=3D2=0134=3D32889=0149=3DEDDE=0152=3D20060216-14:26:00= .350=0156=3DXXXX=017=3D13817=0116=3D0=0110=3D235=01 20060216-14:26:00 : 15:26:00.360 - Sent ResendRequest FROM: 13817 TO: 0 EDDE -> 15:26:06.389 - 8=3D FIX.4.2=019=3D76=0135=3D1=0134=3D32890=0149=3DEDDE=0152=3D20060216-14:26:06= .389=0156=3DXXXX=01112=3DTestOS, 15:26:06=0110=3D206=01 EDDE -> 15:26:09.393 - 8=3D FIX.4.2=019=3D76=0135=3D1=0134=3D32891=0149=3DEDDE=0152=3D20060216-14:26:09= .393=0156=3DXXXX=01112=3DTestOS, 15:26:09=0110=3D208=01 EDDE -> 15:26:12.397 - 8=3D FIX.4.2=019=3D76=0135=3D1=0134=3D32892=0149=3DEDDE=0152=3D20060216-14:26:12= .397=0156=3DXXXX=01112=3DTestOS, 15:26:12=0110=3D201=01 EDDE -> 15:26:15.402 - 8=3D FIX.4.2=019=3D76=0135=3D1=0134=3D32893=0149=3DEDDE=0152=3D20060216-14:26:15= .402=0156=3DXXXX=01112=3DTestOS, 15:26:15=0110=3D195=01 EDDE -> 15:26:18.406 - 8=3D FIX.4.2=019=3D76=0135=3D1=0134=3D32894=0149=3DEDDE=0152=3D20060216-14:26:18= .406=0156=3DXXXX=01112=3DTestOS, 15:26:18=0110=3D206=01 EDDE -> 15:26:21.410 - 8=3D FIX.4.2=019=3D76=0135=3D1=0134=3D32895=0149=3DEDDE=0152=3D20060216-14:26:21= .410=0156=3DXXXX=01112=3DTestOS, 15:26:21=0110=3D190=01 EDDE -> 15:26:24.415 - 8=3D FIX.4.2=019=3D76=0135=3D1=0134=3D32896=0149=3DEDDE=0152=3D20060216-14:26:24= .415=0156=3DXXXX=01112=3DTestOS, 15:26:24=0110=3D202=01 EDDE -> 15:26:27.419 - 8=3D FIX.4.2=019=3D76=0135=3D1=0134=3D32897=0149=3DEDDE=0152=3D20060216-14:26:27= .419=0156=3DXXXX=01112=3DTestOS, 15:26:27=0110=3D213=01 EDDE -> 15:26:30.423 - 8=3D FIX.4.2=019=3D76=0135=3D1=0134=3D32898=0149=3DEDDE=0152=3D20060216-14:26:30= .423=0156=3DXXXX=01112=3DTestOS, 15:26:30=0110=3D197=01 EDDE -> 15:26:33.428 - 8=3D FIX.4.2=019=3D76=0135=3D1=0134=3D32899=0149=3DEDDE=0152=3D20060216-14:26:33= .428=0156=3DXXXX=01112=3DTestOS, 15:26:33=0110=3D209=01 EDDE -> 15:26:36.432 - 8=3D FIX.4.2=019=3D76=0135=3D1=0134=3D32900=0149=3DEDDE=0152=3D20060216-14:26:36= .432=0156=3DXXXX=01112=3DTestOS, 15:26:36=0110=3D193=01 EDDE -> 15:26:39.436 - 8=3D FIX.4.2=019=3D76=0135=3D1=0134=3D32901=0149=3DEDDE=0152=3D20060216-14:26:39= .436=0156=3DXXXX=01112=3DTestOS, 15:26:39=0110=3D204=01 EDDE -> 15:26:42.441 - 8=3D FIX.4.2=019=3D76=0135=3D1=0134=3D32902=0149=3DEDDE=0152=3D20060216-14:26:42= .441=0156=3DXXXX=01112=3DTestOS, 15:26:42=0110=3D189=01 EDDE -> 15:26:45.445 - 8=3D FIX.4.2=019=3D76=0135=3D1=0134=3D32903=0149=3DEDDE=0152=3D20060216-14:26:45= .445=0156=3DXXXX=01112=3DTestOS, 15:26:45=0110=3D200=01 EDDE -> 15:26:48.449 - 8=3D FIX.4.2=019=3D76=0135=3D1=0134=3D32904=0149=3DEDDE=0152=3D20060216-14:26:48= .449=0156=3DXXXX=01112=3DTestOS, 15:26:48=0110=3D211=01 EDDE -> 15:26:51.454 - 8=3D FIX.4.2=019=3D76=0135=3D1=0134=3D32905=0149=3DEDDE=0152=3D20060216-14:26:51= .454=0156=3DXXXX=01112=3DTestOS, 15:26:51=0110=3D196=01 EDDE -> 15:26:54.458 - 8=3D FIX.4.2=019=3D76=0135=3D1=0134=3D32906=0149=3DEDDE=0152=3D20060216-14:26:54= .458=0156=3DXXXX=01112=3DTestOS, 15:26:54=0110=3D207=01 EDDE -> 15:26:57.462 - 8=3D FIX.4.2=019=3D76=0135=3D1=0134=3D32907=0149=3DEDDE=0152=3D20060216-14:26:57= .462=0156=3DXXXX=01112=3DTestOS, 15:26:57=0110=3D209=01 EDDE -> 15:27:00.467 - 8=3D FIX.4.2=019=3D76=0135=3D1=0134=3D32908=0149=3DEDDE=0152=3D20060216-14:27:00= .467=0156=3DXXXX=01112=3DTestOS, 15:27:00=0110=3D193=01 EDDE -> 15:27:03.471 - 8=3D FIX.4.2=019=3D76=0135=3D1=0134=3D32909=0149=3DEDDE=0152=3D20060216-14:27:03= .471=0156=3DXXXX=01112=3DTestOS, 15:27:03=0110=3D195=01 XXXX -> 15:27:04.112 - 8=3D FIX.4.2=019=3D370=0135=3D8=0134=3D13817=0143=3DY=0149=3DXXXX=0152=3D2006021= 6-14:27:02.934=0156=3DEDDE=01122=3D20060216-09:00:57.849=011=3D311761=016= =3D0.000000=0111=3DISTSE0000170110-OL6825p=0114=3D0.000000=0115=3DSEK=0117= =3D1140088724=0120=3D0=0122=3D4=0131=3D0.000000=0132=3D0.000000=0137=3D1376= 4816=0138=3D199.000000=0139=3D4=0140=3D2=0144=3D98=0148=3DSE0000170110=0154= =3D1=0155=3DAZA=0158=3DUnsolicitedorder cancellation=0159=3D3=01109=3D311761=01150=3D4=01151=3D 0.000000=01198=3D20060216100057986548=01207=3DST=0110=3D030=01 XXXX -> 15:27:04.222 - 8=3D FIX.4.2=019=3D375=0135=3D8=0134=3D13818=0143=3DY=0149=3DXXXX=0152=3D2006021= 6-14:27:02.936=0156=3DEDDE=01122=3D20060216-09:00:58.349=011=3D311761=016= =3D0.000000=0111=3DISTSE0000314502-OL6826p=0114=3D0.000000=0115=3DSEK=0117= =3D1140088725=0120=3D0=0122=3D4=0131=3D0.000000=0132=3D0.000000=0137=3D1376= 4817=0138=3D455.000000=0139=3D4=0140=3D2=0144=3D34.5=0148=3DSE0000314502=01= 54=3D2=0155=3DSALAB=0158=3DUnsolicited order cancellation=0159=3D3=01109=3D311761=01150=3D4=01151=3D 0.000000=01198=3D20060216100058475000=01207=3DST=0110=3D020=01 XXXX -> ETC, ETC... Until the last message 34=3D66426 20060216-14:27:14 : 15:27:14.226 - Received SequenceReset FROM: 20675 TO: 20676 20060216-14:27:14 : 15:27:14.387 - Received SequenceReset FROM: 20766 TO: 20767 20060216-14:27:53 : 15:27:53.653 - Received SequenceReset FROM: 46393 TO: 46394 20060216-14:27:55 : 15:27:55.456 - Received SequenceReset FROM: 47614 TO: 47615 20060216-14:28:02 : 15:28:02.306 - Received SequenceReset FROM: 52242 TO: 52245 20060216-14:28:02 : 15:28:02.306 - Received SequenceReset FROM: 52248 TO: 52263 20060216-14:28:02 : 15:28:02.306 - Received SequenceReset FROM: 52264 TO: 52306 20060216-14:28:02 : 15:28:02.316 - Received SequenceReset FROM: 52319 TO: 52320 20060216-14:28:04 : 15:28:04.789 - Received SequenceReset FROM: 53725 TO: 53726 EDDE -> 15:28:10.177 - 8=3D FIX.4.2=019=3D67=0135=3D2=0134=3D32910=0149=3DEDDE=0152=3D20060216-14:28:10= .157=0156=3DXXXX=017=3D1=0116=3D0=0110=3D228=01 XXXX -> 15:28:10.177 - 8=3D FIX.4.2=019=3D375=0135=3D8=0134=3D56802=0143=3DY=0149=3DXXXX=0152=3D2006021= 6-14:28:08.234=0156=3DEDDE=01122=3D20060216-12:37:06.916=011=3D311761=016= =3D0.000000=0111=3DISTSE0000567752-OL27946p=0114=3D0.000000=0115=3DSEK=0117= =3D1140131699=0120=3D0=0122=3D4=0131=3D0.000000=0132=3D0.000000=0137=3D1378= 7749=0138=3D92.000000=0139=3D0=0140=3D2=0144=3D26.7000007629395=0148=3DSE00= 00567752=0154=3D2=0155=3DNTEKB=0159=3D3=0160=3D20060216-12:37:07=01109=3D31= 1761=01150=3D0=01151=3D 92.000000=01198=3D20060216133708358461=01207=3DST=0110=3D146=01 20060216-14:28:10 : 15:28:10.177 - MsgSeqNum too high, expecting 2 but received 56802 20060216-14:28:10 : 15:28:10.177 - Sent ResendRequest FROM: 2 TO: 0 EDDE -> 15:28:10.177 - 8=3D FIX.4.2=019=3D63=0135=3D2=0134=3D32911=0149=3DEDDE=0152=3D20060216-14:28:10= .177=0156=3DXXXX=017=3D2=0116=3D0=0110=3D017=01 XXXX -> 15:28:10.177 - 8=3D FIX.4.2=019=3D376=0135=3D8=0134=3D56803=0143=3DY=0149=3DXXXX=0152=3D2006021= 6-14:28:08.236=0156=3DEDDE=01122=3D20060216-12:37:06.950=011=3D311761=016= =3D0.000000=0111=3DISTSE0000115446-OL27943p=0114=3D0.000000=0115=3DSEK=0117= =3D1140131700=0120=3D0=0122=3D4=0131=3D0.000000=0132=3D0.000000=0137=3D1378= 7746=0138=3D39.000000=0139=3D4=0140=3D2=0144=3D345.5=0148=3DSE0000115446=01= 54=3D1=0155=3DVOLVB=0158=3DUnsolicited order cancellation=0159=3D3=01109=3D311761=01150=3D4=01151=3D 0.000000=01198=3D20060216133707359386=01207=3DST=0110=3D133=01 20060216-14:28:10 : 15:28:10.177 - MsgSeqNum too high, expecting 2 but received 56803 EDDE -> 15:28:10.177 - 8=3D FIX.4.2=019=3D63=0135=3D2=0134=3D32912=0149=3DEDDE=0152=3D20060216-14:28:10= .177=0156=3DXXXX=017=3D2=0116=3D0=0110=3D018=01 20060216-14:28:10 : 15:28:10.177 - Sent ResendRequest FROM: 2 TO: 0 |
|
From: Dale W. <wil...@oc...> - 2006-02-20 14:16:27
|
Hi Eddie Edde wrote: > Since it needed to resend 50000 messages this takes some time and > while this is in progress our system continues the startup process. > I'm planning to rewrite this code ASAP but at the moment what we do > after a major crash on either end is to clear all our databases and > send a ResendRequest demanding all messages sent in the session: The problem is that a ResendRequest is an administrative-level message and you are attempting to use it for an application-level purpose. TCP/IP will resend packets on occasion as part of it's error recovery, but no one attempts to use this information to "replay" a TCP/IP session. THe same should be true for FIX. Communication stacks work because the layers are kept distinct --- each one serving it's own purpose and interactinint with the layers above and below it through a well specified API. Unfortunately in FIX the two layers, admin and app, are not so clearly delineated and it tempts developers to cross the line. This is not the first time I have seen someone trying to use a ResendRequest for this purpose. QuickFIX takes care of the administrative layer of the protocol for you, but if your application sends its own admin layer messages that QuickFIX doesn't know about, it gets confused (and rightly so!) You might be able to come up with a hack that works for this particular version of QuickFIX, but what ever you come up with there's no guarantee that it will work correctly in future versions of QuickFIX. That gets us to the question of, "what's the *right* way to fix this problem.. Solution #1: read the day's messages fom the store or from the log and reprocess them. . Solution #2: define a "retransmit today's transactions" message. You might define a new custom message type, or consider "twisting" an application level message for this purpose. (How about using "wildcard" order id's in an order status request.? ) You could even define an "End of retransamint" application level message (or field within a message). This, of course, assumes you have some influence on the software at the other end of the connection. Note that "misusing" an application level message is much less of a sin than subverting an Admin message. Among other things it won't confuse QuickFIX. Solution #3: Trick QuickFIX into sending the resend request you need -- reset the expected incoming sequence number to 1 without chaning the next sequence number to send. Do this before loggin in so only one "resed" will happen. Dale -- ----------------------------------------------------- Dale Wilson, Senior Software Engineer Object Computing, Inc. (OCI) http://www.ociweb.com/ http://www.theaceorb.com/ ---------------------------------------------------- |
|
From: Edde <edd...@gm...> - 2006-02-20 15:24:15
|
Hi Dale, Thanks for the quick reply. The reason we've gone through the hassel of resending all the messages is that we're too people working on our trading system. We work from two different locations and a couple of times a day we swap who's doing the actual trading. The current procedure when we swap is that we simply exchange the seqnums file and then since we do the complete ResendRequest the one taking over the trading will process the old messages and reach the current trading "state". However I realize know that I've been on the wrong path by doing this. Your first suggestion about reading the messages from the log seems pretty good to me. Is there a standardized way of doing this? For example: 1) I don't really care about heartbeats etc so is there a specification somewhere which messages are significant when reprocessing? 2) Are there any utility methods in QuickFIX to reproduce a FIX message fro= m a String/character stream or similar? 3) I guess you need to maintain a list of all messages already processed so that you don't get duplicate processing of one or more messages? Thanks for your help, /Eddie On 2/20/06, Dale Wilson <wil...@oc...> wrote: > > Hi Eddie > > Edde wrote: > > > Since it needed to resend 50000 messages this takes some time and > > while this is in progress our system continues the startup process. > > I'm planning to rewrite this code ASAP but at the moment what we do > > after a major crash on either end is to clear all our databases and > > send a ResendRequest demanding all messages sent in the session: > > The problem is that a ResendRequest is an administrative-level message > and you are attempting to use it for an application-level purpose. > TCP/IP will resend packets on occasion as part of it's error recovery, > but no one attempts to use this information to "replay" a TCP/IP > session. THe same should be true for FIX. Communication stacks work > because the layers are kept distinct --- each one serving it's own > purpose and interactinint with the layers above and below it through a > well specified API. > > Unfortunately in FIX the two layers, admin and app, are not so clearly > delineated and it tempts developers to cross the line. This is not the > first time I have seen someone trying to use a ResendRequest for this > purpose. QuickFIX takes care of the administrative layer of the > protocol for you, but if your application sends its own admin layer > messages that QuickFIX doesn't know about, it gets confused (and rightly > so!) > > You might be able to come up with a hack that works for this particular > version of QuickFIX, but what ever you come up with there's no guarantee > that it will work correctly in future versions of QuickFIX. > > That gets us to the question of, "what's the *right* way to fix this > problem.. > > Solution #1: read the day's messages fom the store or from the log and > reprocess them. . > > Solution #2: define a "retransmit today's transactions" message. You > might define a new custom message type, or consider "twisting" an > application level message for this purpose. (How about using "wildcard" > order id's in an order status request.? ) You could even define an > "End of retransamint" application level message (or field within a > message). This, of course, assumes you have some influence on the > software at the other end of the connection. Note that "misusing" an > application level message is much less of a sin than subverting an Admin > message. Among other things it won't confuse QuickFIX. > > Solution #3: Trick QuickFIX into sending the resend request you need > -- reset the expected incoming sequence number to 1 without chaning the > next sequence number to send. Do this before loggin in so only one > "resed" will happen. > > > Dale > > -- > ----------------------------------------------------- > Dale Wilson, Senior Software Engineer > Object Computing, Inc. (OCI) > http://www.ociweb.com/ http://www.theaceorb.com/ > ---------------------------------------------------- > > |
|
From: Dale W. <wil...@oc...> - 2006-02-20 17:04:11
|
Edde wrote: > Hi Dale, > > Thanks for the quick reply. > > The reason we've gone through the hassel of resending all the messages > is that we're too people working on our trading system. We work from > two different locations and a couple of times a day we swap who's > doing the actual trading. A simple solution: Share a disk and put the store directory on the shared disk so both workstations can access it. Dale > The current procedure when we swap is that we simply exchange the > seqnums file and then since we do the complete ResendRequest the one > taking over the trading will process the old messages and reach the > current trading "state". > However I realize know that I've been on the wrong path by doing this. > Your first suggestion about reading the messages from the log seems > pretty good to me. Is there a standardized way of doing this? For example: > > 1) I don't really care about heartbeats etc so is there a > specification somewhere which messages are significant when reprocessing? > 2) Are there any utility methods in QuickFIX to reproduce a FIX > message from a String/character stream or similar? > 3) I guess you need to maintain a list of all messages already > processed so that you don't get duplicate processing of one or more > messages? > > Thanks for your help, > /Eddie > > > On 2/20/06, *Dale Wilson* <wil...@oc... > <mailto:wil...@oc...>> wrote: > > Hi Eddie > > Edde wrote: > > > Since it needed to resend 50000 messages this takes some time and > > while this is in progress our system continues the startup process. > > I'm planning to rewrite this code ASAP but at the moment what we do > > after a major crash on either end is to clear all our databases and > > send a ResendRequest demanding all messages sent in the session: > > The problem is that a ResendRequest is an administrative-level message > and you are attempting to use it for an application-level purpose. > TCP/IP will resend packets on occasion as part of it's error recovery, > but no one attempts to use this information to "replay" a TCP/IP > session. THe same should be true for FIX. Communication stacks work > because the layers are kept distinct --- each one serving it's own > purpose and interactinint with the layers above and below it through a > well specified API. > > Unfortunately in FIX the two layers, admin and app, are not so > clearly > delineated and it tempts developers to cross the line. This > is not the > first time I have seen someone trying to use a ResendRequest for this > purpose. QuickFIX takes care of the administrative layer of the > protocol for you, but if your application sends its own admin layer > messages that QuickFIX doesn't know about, it gets confused (and > rightly > so!) > > You might be able to come up with a hack that works for this > particular > version of QuickFIX, but what ever you come up with there's no > guarantee > that it will work correctly in future versions of QuickFIX. > > That gets us to the question of, "what's the *right* way to fix this > problem.. > > Solution #1: read the day's messages fom the store or from the > log and > reprocess them. . > > Solution #2: define a "retransmit today's transactions" message. You > might define a new custom message type, or consider "twisting" an > application level message for this purpose. (How about using > "wildcard" > order id's in an order status request.? ) You could even define an > "End of retransamint" application level message (or field within a > message). This, of course, assumes you have some influence on the > software at the other end of the connection. Note that "misusing" an > application level message is much less of a sin than subverting an > Admin > message. Among other things it won't confuse QuickFIX. > > Solution #3: Trick QuickFIX into sending the resend request you need > -- reset the expected incoming sequence number to 1 without > chaning the > next sequence number to send. Do this before loggin in so only one > "resed" will happen. > > > Dale > > -- > ----------------------------------------------------- > Dale Wilson, Senior Software Engineer > Object Computing, Inc. (OCI) > http://www.ociweb.com/ http://www.theaceorb.com/ > ---------------------------------------------------- > > -- ----------------------------------------------------- Dale Wilson, Senior Software Engineer Object Computing, Inc. (OCI) http://www.ociweb.com/ http://www.theaceorb.com/ ---------------------------------------------------- |
|
From: Oren M. <or...@qu...> - 2006-02-20 15:58:53
|
I don't see how your current system works. If you are swapping seqnum files, QuickFIX wouldn't pass you the older messages anyway on account of their sequence numbers being too low plus being marked PossDup. QuickFIX will simply throw the old stuff away while only passing you the new stuff. In effect making the extra resend request sort of pointless. If you truly wanted to do this resend request, you would have to reset your incoming sequence number. in any case... 1) The Message class has isAdmin and isApp methods. Anything that returns true for isApp should be processed (or thrown out if isAdmin is true, however you want to look at it) 2) The constructor for the Message class takes in a string. 3) Is your system stateless? Is it ok to just process all messages when it loads up? If not you would essentially have to treat every message like it is marked PossResend and handle appropriately. I would really look into writing a simple gateway so your two applications can share the connection. --oren Edde wrote: > Hi Dale, > > Thanks for the quick reply. > > The reason we've gone through the hassel of resending all the messages > is that we're too people working on our trading system. We work from > two different locations and a couple of times a day we swap who's > doing the actual trading. The current procedure when we swap is that > we simply exchange the seqnums file and then since we do the complete > ResendRequest the one taking over the trading will process the old > messages and reach the current trading "state". > However I realize know that I've been on the wrong path by doing this. > Your first suggestion about reading the messages from the log seems > pretty good to me. Is there a standardized way of doing this? For example: > > 1) I don't really care about heartbeats etc so is there a > specification somewhere which messages are significant when reprocessing? > 2) Are there any utility methods in QuickFIX to reproduce a FIX > message from a String/character stream or similar? > 3) I guess you need to maintain a list of all messages already > processed so that you don't get duplicate processing of one or more > messages? > > Thanks for your help, > /Eddie > > > On 2/20/06, *Dale Wilson* <wil...@oc... > <mailto:wil...@oc...>> wrote: > > Hi Eddie > > Edde wrote: > > > Since it needed to resend 50000 messages this takes some time and > > while this is in progress our system continues the startup process. > > I'm planning to rewrite this code ASAP but at the moment what we do > > after a major crash on either end is to clear all our databases and > > send a ResendRequest demanding all messages sent in the session: > > The problem is that a ResendRequest is an administrative-level message > and you are attempting to use it for an application-level purpose. > TCP/IP will resend packets on occasion as part of it's error recovery, > but no one attempts to use this information to "replay" a TCP/IP > session. THe same should be true for FIX. Communication stacks work > because the layers are kept distinct --- each one serving it's own > purpose and interactinint with the layers above and below it through a > well specified API. > > Unfortunately in FIX the two layers, admin and app, are not so > clearly > delineated and it tempts developers to cross the line. This > is not the > first time I have seen someone trying to use a ResendRequest for this > purpose. QuickFIX takes care of the administrative layer of the > protocol for you, but if your application sends its own admin layer > messages that QuickFIX doesn't know about, it gets confused (and > rightly > so!) > > You might be able to come up with a hack that works for this > particular > version of QuickFIX, but what ever you come up with there's no > guarantee > that it will work correctly in future versions of QuickFIX. > > That gets us to the question of, "what's the *right* way to fix this > problem.. > > Solution #1: read the day's messages fom the store or from the > log and > reprocess them. . > > Solution #2: define a "retransmit today's transactions" message. You > might define a new custom message type, or consider "twisting" an > application level message for this purpose. (How about using > "wildcard" > order id's in an order status request.? ) You could even define an > "End of retransamint" application level message (or field within a > message). This, of course, assumes you have some influence on the > software at the other end of the connection. Note that "misusing" an > application level message is much less of a sin than subverting an > Admin > message. Among other things it won't confuse QuickFIX. > > Solution #3: Trick QuickFIX into sending the resend request you need > -- reset the expected incoming sequence number to 1 without > chaning the > next sequence number to send. Do this before loggin in so only one > "resed" will happen. > > > Dale > > -- > ----------------------------------------------------- > Dale Wilson, Senior Software Engineer > Object Computing, Inc. (OCI) > http://www.ociweb.com/ http://www.theaceorb.com/ > ---------------------------------------------------- > > |
|
From: Edde <edd...@gm...> - 2006-02-21 10:37:44
|
Hi Oren, > I don't see how your current system works. If you are swapping seqnum > files, QuickFIX wouldn't pass you the older messages anyway on account > of their sequence numbers being too low plus being marked PossDup. > QuickFIX will simply throw the old stuff away while only passing you the > new stuff. In effect making the extra resend request sort of > pointless. If you truly wanted to do this resend request, you would > have to reset your incoming sequence number. Let's see if I can try to clear things up a bit. Me and my colleague both work from home about 100km from each other. We both have fairly reliable broadband connections but they're certainly not perfect and dissruptions occur on a weekly basis. Say that my colleague starts with the trading in the morning when the marke= t opens and then at lunch time I take over. Our procedure then is for my colleague to put his seqnums file on a third-party FTP server and before I start up my system I replace my existing seqnums with the one he's put on FTP. When I connect using QuickFIX our counterparty will send me any messages that it has sent during the transition time but these I just discard. Instead I wait until our system has completed it's startup and whe= n all is complete I reset my incoming sequence number (like you said) and do = a complete resend of all messages and thus recreating the whole session. However, I understand from Dale's post that this is abusing the admin layer so a better solution would probably be to store the current state (all trades and active orders) at shutdown and then pass this "state" file along to FTP as well. Then I can recreate the state before connecting to my counterparty and when the connection is made I can simply start processing the messages as they arrive from QuickFIX and not play around with the admi= n layer at all. However, as Joerg pointed out in a later post, I still need a fail-over situation when the reason we swap is that my colleague's internet connectio= n is down. This means he can't put any state files on FTP so I have to recreate the state of the session relying on my counterparty alone. The onl= y way I've come up with to do this is by resending all messages from the star= t of the session but if someone has a better suggestion I'm all ears. > I would really look into writing a simple gateway so your two > applications can share the connection. I've though about this solution as well but the problem is where to put the gateway. If we'd put it at my place or my colleague's we'd rely on both our internet connections to work all the time which is definately not the case. Thanks for helping! Cheers, /Eddie --oren > > Edde wrote: > > > Hi Dale, > > > > Thanks for the quick reply. > > > > The reason we've gone through the hassel of resending all the messages > > is that we're too people working on our trading system. We work from > > two different locations and a couple of times a day we swap who's > > doing the actual trading. The current procedure when we swap is that > > we simply exchange the seqnums file and then since we do the complete > > ResendRequest the one taking over the trading will process the old > > messages and reach the current trading "state". > > However I realize know that I've been on the wrong path by doing this. > > Your first suggestion about reading the messages from the log seems > > pretty good to me. Is there a standardized way of doing this? For > example: > > > > 1) I don't really care about heartbeats etc so is there a > > specification somewhere which messages are significant when > reprocessing? > > 2) Are there any utility methods in QuickFIX to reproduce a FIX > > message from a String/character stream or similar? > > 3) I guess you need to maintain a list of all messages already > > processed so that you don't get duplicate processing of one or more > > messages? > > > > Thanks for your help, > > /Eddie > > > > > > On 2/20/06, *Dale Wilson* <wil...@oc... > > <mailto:wil...@oc...>> wrote: > > > > Hi Eddie > > > > Edde wrote: > > > > > Since it needed to resend 50000 messages this takes some time and > > > while this is in progress our system continues the startup > process. > > > I'm planning to rewrite this code ASAP but at the moment what we > do > > > after a major crash on either end is to clear all our databases > and > > > send a ResendRequest demanding all messages sent in the session: > > > > The problem is that a ResendRequest is an administrative-level > message > > and you are attempting to use it for an application-level purpose. > > TCP/IP will resend packets on occasion as part of it's error > recovery, > > but no one attempts to use this information to "replay" a TCP/IP > > session. THe same should be true for FIX. Communication stacks > work > > because the layers are kept distinct --- each one serving it's own > > purpose and interactinint with the layers above and below it throug= h > a > > well specified API. > > > > Unfortunately in FIX the two layers, admin and app, are not so > > clearly > > delineated and it tempts developers to cross the line. This > > is not the > > first time I have seen someone trying to use a ResendRequest for > this > > purpose. QuickFIX takes care of the administrative layer of the > > protocol for you, but if your application sends its own admin layer > > messages that QuickFIX doesn't know about, it gets confused (and > > rightly > > so!) > > > > You might be able to come up with a hack that works for this > > particular > > version of QuickFIX, but what ever you come up with there's no > > guarantee > > that it will work correctly in future versions of QuickFIX. > > > > That gets us to the question of, "what's the *right* way to fix thi= s > > problem.. > > > > Solution #1: read the day's messages fom the store or from the > > log and > > reprocess them. . > > > > Solution #2: define a "retransmit today's transactions" > message. You > > might define a new custom message type, or consider "twisting" an > > application level message for this purpose. (How about using > > "wildcard" > > order id's in an order status request.? ) You could even define a= n > > "End of retransamint" application level message (or field within a > > message). This, of course, assumes you have some influence on the > > software at the other end of the connection. Note that "misusing" > an > > application level message is much less of a sin than subverting an > > Admin > > message. Among other things it won't confuse QuickFIX. > > > > Solution #3: Trick QuickFIX into sending the resend request you > need > > -- reset the expected incoming sequence number to 1 without > > chaning the > > next sequence number to send. Do this before loggin in so only one > > "resed" will happen. > > > > > > Dale > > > > -- > > ----------------------------------------------------- > > Dale Wilson, Senior Software Engineer > > Object Computing, Inc. (OCI) > > http://www.ociweb.com/ http://www.theaceorb.com/ > > ---------------------------------------------------- > > > > > |
|
From: Edde <edd...@gm...> - 2006-03-15 16:08:53
|
Hi Oren, > > I would really look into writing a simple gateway so your two > applications can share the connection. Thanks for everyone's help with this issue! We've now been in contact with our broker and they have agreed to host one of our server which is great news since we can now rewrite our applications to work as a client/server system. The idea is to have our server (which is hosted at the broker firm) to communicate with the market through FIX and then we can have any number of clients connected to the server. I guess this is a fairly common setup for = a trading system so I was wondering if anyone on this list have any suggestions on what transfer/communications protocol we should use between the client and the server. I guess you can use FIX here as well but since w= e need to transmit other data as well (e.g portfolio information) I'd like to find some other way to communicate. My own idea was to use some XML based protocol like SOAP but my experience in this is limited so I'd gladly appreciate any suggestions from the experts on this list. Our system is 100= % Java based and I'd like to keep it this way if possible. Thanks again! /Eddie --oren > > Edde wrote: > > > Hi Dale, > > > > Thanks for the quick reply. > > > > The reason we've gone through the hassel of resending all the messages > > is that we're too people working on our trading system. We work from > > two different locations and a couple of times a day we swap who's > > doing the actual trading. The current procedure when we swap is that > > we simply exchange the seqnums file and then since we do the complete > > ResendRequest the one taking over the trading will process the old > > messages and reach the current trading "state". > > However I realize know that I've been on the wrong path by doing this. > > Your first suggestion about reading the messages from the log seems > > pretty good to me. Is there a standardized way of doing this? For > example: > > > > 1) I don't really care about heartbeats etc so is there a > > specification somewhere which messages are significant when > reprocessing? > > 2) Are there any utility methods in QuickFIX to reproduce a FIX > > message from a String/character stream or similar? > > 3) I guess you need to maintain a list of all messages already > > processed so that you don't get duplicate processing of one or more > > messages? > > > > Thanks for your help, > > /Eddie > > > > > > On 2/20/06, *Dale Wilson* <wil...@oc... > > <mailto:wil...@oc...>> wrote: > > > > Hi Eddie > > > > Edde wrote: > > > > > Since it needed to resend 50000 messages this takes some time and > > > while this is in progress our system continues the startup > process. > > > I'm planning to rewrite this code ASAP but at the moment what we > do > > > after a major crash on either end is to clear all our databases > and > > > send a ResendRequest demanding all messages sent in the session: > > > > The problem is that a ResendRequest is an administrative-level > message > > and you are attempting to use it for an application-level purpose. > > TCP/IP will resend packets on occasion as part of it's error > recovery, > > but no one attempts to use this information to "replay" a TCP/IP > > session. THe same should be true for FIX. Communication stacks > work > > because the layers are kept distinct --- each one serving it's own > > purpose and interactinint with the layers above and below it throug= h > a > > well specified API. > > > > Unfortunately in FIX the two layers, admin and app, are not so > > clearly > > delineated and it tempts developers to cross the line. This > > is not the > > first time I have seen someone trying to use a ResendRequest for > this > > purpose. QuickFIX takes care of the administrative layer of the > > protocol for you, but if your application sends its own admin layer > > messages that QuickFIX doesn't know about, it gets confused (and > > rightly > > so!) > > > > You might be able to come up with a hack that works for this > > particular > > version of QuickFIX, but what ever you come up with there's no > > guarantee > > that it will work correctly in future versions of QuickFIX. > > > > That gets us to the question of, "what's the *right* way to fix thi= s > > problem.. > > > > Solution #1: read the day's messages fom the store or from the > > log and > > reprocess them. . > > > > Solution #2: define a "retransmit today's transactions" > message. You > > might define a new custom message type, or consider "twisting" an > > application level message for this purpose. (How about using > > "wildcard" > > order id's in an order status request.? ) You could even define a= n > > "End of retransamint" application level message (or field within a > > message). This, of course, assumes you have some influence on the > > software at the other end of the connection. Note that "misusing" > an > > application level message is much less of a sin than subverting an > > Admin > > message. Among other things it won't confuse QuickFIX. > > > > Solution #3: Trick QuickFIX into sending the resend request you > need > > -- reset the expected incoming sequence number to 1 without > > chaning the > > next sequence number to send. Do this before loggin in so only one > > "resed" will happen. > > > > > > Dale > > > > -- > > ----------------------------------------------------- > > Dale Wilson, Senior Software Engineer > > Object Computing, Inc. (OCI) > > http://www.ociweb.com/ http://www.theaceorb.com/ > > ---------------------------------------------------- > > > > > |
|
From: Joerg T. <Joe...@ma...> - 2006-02-20 16:05:15
|
Edde wrote:
> [...]
> So, for each message I receive between 56802 and 66426 QuickFIX
> automatically sends a new ResendRequest demanding all messages in the
> session. Obviously this is not good behaviour and I could work around t=
his
> if there is a way to know when the first ResendRequest has received all=
the
> messages it's expecting.
Hmmm, this reminds me of a behaviour fixed in 1.10.2: This version explic=
itely avoids=20
sending duplicate ResendRequest(N:Infinity) and logs the message "Already=
sent=20
ResendRequest. Not sending another". Maybe a newer version will improve t=
he situation.
While I understand Dales argument, not to abuse the session level for app=
lication level=20
recovery, this scenario could make sense in a fail-over situation.
BTW, a TestRequest with a unique has to be answered by a Heartbeat with t=
his unique id.
If you get this specific heartbeat, you know that the other side must hav=
e processed all=20
messages preceding this TestRequest. Of course, this does not prevent fur=
ther resend=20
processing at any later point of time.
Cheers, J=F6rg
--=20
Joerg Thoennes
http://macd.com
Tel.: +49 (0)241 44597-24 Macdonald Associates GmbH
Fax : +49 (0)241 44597-10 Lothringer Str. 52, D-52070 Aachen
|
|
From: Dale W. <wil...@oc...> - 2006-02-21 14:45:14
|
Edde wrote: > Say that my colleague starts with the trading in the morning when the > market opens and then at lunch time I take over. Our procedure then is > for my colleague to put his seqnums file on a third-party FTP server > and before I start up my system I replace my existing seqnums with the > one he's put on FTP. When I connect using QuickFIX our counterparty > will send me any messages that it has sent during the transition > time but these I just discard. Instead I wait until our system has > completed it's startup and when all is complete I reset my incoming > sequence number (like you said) and do a complete resend of all > messages and thus recreating the whole session. If I read this correctly, you know you're going to do the recovery before you start the application. That means that having the counterparty send the messages sent during transmission time is wasted effort, right? So I suggest that you clear the expected sequence number *before* starting the application. That way QuickFIX will allways request a retransmission from 1 thru infinity, and since QuickFIX requested the retransmissions it will know what to do with them. You still have to address the issue of setting your sending sequence number if your colleague sent any messages after he captured the file he put on the FTP server and before the connection died. Do you have a way to handle that? Dale -- ----------------------------------------------------- Dale Wilson, Senior Software Engineer Object Computing, Inc. (OCI) http://www.ociweb.com/ http://www.theaceorb.com/ ---------------------------------------------------- |
|
From: Edde <edd...@gm...> - 2006-02-21 15:35:18
|
> > > If I read this correctly, you know you're going to do the recovery > before you start the application. That means that having the > counterparty send the messages sent during transmission time is wasted > effort, right? So I suggest that you clear the expected sequence > number *before* starting the application. That way QuickFIX will > allways request a retransmission from 1 thru infinity, and since > QuickFIX requested the retransmissions it will know what to do with them. Yes, that would probably be the best way to do this so I'll reimplement our system to use this when we have internet connection problems. You still have to address the issue of setting your sending sequence > number if your colleague sent any messages after he captured the file he > put on the FTP server and before the connection died. Do you have a way > to handle that? I don't think this can happen since the seqnums are stored after we've logged out from FIX which means no more messages are received. However, if it does happen I guess you can always bump up the seqnums manually just to be able to logon. Thanks, /Eddie Dale > > -- > ----------------------------------------------------- > Dale Wilson, Senior Software Engineer > Object Computing, Inc. (OCI) > http://www.ociweb.com/ http://www.theaceorb.com/ > ---------------------------------------------------- > > |
