Re: [Quickfix-developers] Persisting unprocessed messages

[K. F. <kfr...@gm...>]

Hello Simeon!

On Mon, May 23, 2016 at 11:51 AM, Simeon Simeonov
<sim...@gm...> wrote:
>
> Hi, I have a quickfix initiator and I am trying to solve the following problem. If I have a burst of fills, quickfix will receive all the fills and start passing them to the application. The fill messages at this point have been received and are held in memory by quickfix. If the application crashes while processing the messages then what is in memory is lost. When the client restarts it has to re-request the unprocessed messages from the server which might not have them.

To emphasize what Viktor said, you have a fundamental problem
here.  No matter how promptly you persist the messages you
receive, there is always the possibility that some messages will
be lost in flight.

As Viktor said, that is what ResendRequest is for, and for guaranteed
reliability, it is necessary for the server to remember (presumably by
persisting) the messages it has sent, and to honor the ResendRequest.

Let's say your client persists received messages "as soon as" it gets
them.  Let's say the server writes the messages it is sending to its
socket and immediately deletes them from memory (because they've
been sent).  Now let's say your server crashes (for example, a power
failure) before the network has actually delivered the messages to
your application.  Under these assumptions, you have an unrecoverable
error because the messages have been lost for good.  No matter how
quickly you persist the messages you receive, there will always be
such an in-flight window where messages can be lost, so you're better
off implementing a robust solution that solves the problem, rather than
"persisting really fast" in order to reduce the problem to a small, but
non-zero level.

The FIX solution is to require both sides of the FIX conversation to
remember the messages they have sent, and correctly honor the
ResendRequest.

(You could imagine a protocol where the client sends back to the
server an "I have received and persisted this message -- you can
delete it on your end now" message.  But you would still have to
have a way -- some ResendRequest, or equivalent -- to recover any
messages that were in flight when you crashed, before you had a
chance to persist them.)

To recap:  You say "When the client restarts it has to re-request the
unprocessed messages from the server which might not have them."
The fact that "the server ... might not have them" means that your
server is fundamentally broken, and there is no way that the FIX
protocol can fix that for you.

> My question is this - is there a simple way to make quickfix dump messages to disk

Yes, QuickFIX will persist messages to disk and/or a database.

> so that when the application crashes it can pick from where it left off?

Sort of.  It depends what you mean by "from where it left off."
It can pick up from where it left off successfully persisting
messages, but it can't necessarily pick up from where the server
left off writing messages to its socket because of the in-flight
issue described above.

> I have enabled 'PersistMessages=Y' and I have a disk store but it doesn't seem to be picking up from where it left off upon restart.

Presumably the server has written more messages than you have
persisted, as described above.  That is, the client's notion of "where
it left off" does not match that of the server.

> I am not sure what config I need to get this functionality.

There is no client-side configuration that will (or could) solve the
in-flight issue.  You need to have your server persist the messages
it sends and properly honor ResendRequest.  That's the FIX way
to implement this kind of fully legitimate requirement of being able to
recover after a crash.

> Kind regards,
> Simeon


Good luck.


K. Frank