Re: [Quickfix-developers] Waiting for automatic ResendRequest to finish?

[Edde <edd...@gm...>]

Hi Oren,



> I don't see how your current system works.  If you are swapping seqnum
> files, QuickFIX wouldn't pass you the older messages anyway on account
> of their sequence numbers being too low plus being marked PossDup.
> QuickFIX will simply throw the old stuff away while only passing you the
> new stuff.  In effect making the extra resend request sort of
> pointless.  If you truly wanted to do this resend request, you would
> have to reset your incoming sequence number.


Let's see if I can try to clear things up a bit. Me and my colleague both
work from home about 100km from each other. We both have fairly reliable
broadband connections but they're certainly not perfect and dissruptions
occur on a weekly basis.
Say that my colleague starts with the trading in the morning when the marke=
t
opens and then at lunch time I take over. Our procedure then is for my
colleague to put his seqnums file on a third-party FTP server and before I
start up my system I replace my existing seqnums with the one he's put on
FTP. When I connect using QuickFIX our counterparty will send me any
messages that it has sent during the transition time but these I just
discard. Instead I wait until our system has completed it's startup and whe=
n
all is complete I reset my incoming sequence number (like you said) and do =
a
complete resend of all messages and thus recreating the whole session.
However, I understand from Dale's post that this is abusing the admin layer
so a better solution would probably be to store the current state (all
trades and active orders) at shutdown and then pass this "state" file along
to FTP as well. Then I can recreate the state before connecting to my
counterparty and when the connection is made I can simply start processing
the messages as they arrive from QuickFIX and not play around with the admi=
n
layer at all.

However, as Joerg pointed out in a later post, I still need a fail-over
situation when the reason we swap is that my colleague's internet connectio=
n
is down. This means he can't put any state files on FTP so I have to
recreate the state of the session relying on my counterparty alone. The onl=
y
way I've come up with to do this is by resending all messages from the star=
t
of the session but if someone has a better suggestion I'm all ears.


> I would really look into writing a simple gateway so your two
> applications can share the connection.


I've though about this solution as well but the problem is where to put the
gateway. If we'd put it at my place or my colleague's we'd rely on both our
internet connections to work all the time which is definately not the case.

Thanks for helping!
Cheers,
/Eddie

--oren
>
> Edde wrote:
>
> > Hi Dale,
> >
> > Thanks for the quick reply.
> >
> > The reason we've gone through the hassel of resending all the messages
> > is that we're too people working on our trading system. We work from
> > two different locations and a couple of times a day we swap who's
> > doing the actual trading. The current procedure when we swap is that
> > we simply exchange the seqnums file and then since we do the complete
> > ResendRequest the one taking over the trading will process the old
> > messages and reach the current trading "state".
> > However I realize know that I've been on the wrong path by doing this.
> > Your first suggestion  about reading the messages from the log seems
> > pretty good to me. Is there a standardized way of doing this? For
> example:
> >
> > 1) I don't really care about heartbeats etc so is there a
> > specification somewhere which messages are significant when
> reprocessing?
> > 2) Are there any utility methods in QuickFIX to reproduce a FIX
> > message from a String/character stream or similar?
> > 3) I guess you need to maintain a list of all messages already
> > processed so that you don't get duplicate processing of one or more
> > messages?
> >
> > Thanks for your help,
> > /Eddie
> >
> >
> > On 2/20/06, *Dale Wilson* <wil...@oc...
> > <mailto:wil...@oc...>> wrote:
> >
> >     Hi Eddie
> >
> >     Edde wrote:
> >
> >     > Since it needed to resend 50000 messages this takes some time and
> >     > while this is in progress our system continues the startup
> process.
> >     > I'm planning to rewrite this code ASAP but at the moment what we
> do
> >     > after a major crash on either end is to clear all our databases
> and
> >     > send a ResendRequest demanding all messages sent in the session:
> >
> >     The problem is that a ResendRequest is an administrative-level
> message
> >     and you are attempting to use it for an application-level purpose.
> >     TCP/IP will resend packets on occasion as part of it's error
> recovery,
> >     but no one attempts to use this information to "replay" a TCP/IP
> >     session.  THe same should be true for FIX.  Communication stacks
> work
> >     because the layers are kept distinct  --- each one serving it's own
> >     purpose and interactinint with the layers above and below it throug=
h
> a
> >     well specified API.
> >
> >     Unfortunately in FIX the two layers, admin and app, are not so
> >     clearly
> >     delineated and it tempts developers to cross the line.  This
> >     is  not the
> >     first time I have seen someone trying to use a ResendRequest for
> this
> >     purpose.  QuickFIX takes care of the administrative layer of the
> >     protocol for you, but if your application sends its own admin layer
> >     messages that QuickFIX doesn't know about, it gets confused (and
> >     rightly
> >     so!)
> >
> >     You might be able to come up with a hack that works for this
> >     particular
> >     version of QuickFIX, but what ever you come up with there's no
> >     guarantee
> >     that it will work correctly in future versions of QuickFIX.
> >
> >     That gets us to the question of, "what's the *right* way to fix thi=
s
> >     problem..
> >
> >     Solution #1:  read the day's messages fom the store or from the
> >     log and
> >     reprocess them.  .
> >
> >     Solution #2: define a "retransmit today's transactions"
> message.  You
> >     might  define a new custom message type, or consider "twisting" an
> >     application level message for this purpose. (How about using
> >     "wildcard"
> >     order id's in an order status request.? )   You could even define a=
n
> >     "End of retransamint" application level message (or field within a
> >     message).  This, of course, assumes you have some influence on the
> >     software at the other end of the connection.  Note that "misusing"
> an
> >     application level message is much less of a sin than subverting an
> >     Admin
> >     message.  Among other things it won't confuse QuickFIX.
> >
> >     Solution #3:  Trick QuickFIX into sending  the resend request you
> need
> >     -- reset the expected incoming sequence number to 1 without
> >     chaning the
> >     next sequence number to send.  Do this before loggin in so only one
> >     "resed" will happen.
> >
> >
> >     Dale
> >
> >     --
> >     -----------------------------------------------------
> >     Dale Wilson, Senior Software Engineer
> >     Object Computing, Inc. (OCI)
> >     http://www.ociweb.com/  http://www.theaceorb.com/
> >     ----------------------------------------------------
> >
> >
>