Thread: [Quickfix-developers] Persisting unprocessed messages
Brought to you by:
orenmnero
Menu
▾
▴
quickfix-developers
|
From: Simeon S. <sim...@gm...> - 2016-05-23 15:51:38
|
Hi, I have a quickfix initiator and I am trying to solve the following problem. If I have a burst of fills, quickfix will receive all the fills and start passing them to the application. The fill messages at this point have been received and are held in memory by quickfix. If the application crashes while processing the messages then what is in memory is lost. When the client restarts it has to re-request the unprocessed messages from the server which might not have them. My question is this - is there a simple way to make quickfix dump messages to disk so that when the application crashes it can pick from where it left off? I have enabled 'PersistMessages=Y' and I have a disk store but it doesn't seem to be picking up from where it left off upon restart. I am not sure what config I need to get this functionality. Kind regards, Simeon |
|
From: Viktor P. <pohrebnyak@i.ua> - 2016-05-23 18:57:47
|
Hi, Quickfix extracts/processes messages from network buffer one-by-one and writes them to the disk if configured. There is no way to force behavior you want in the current engine. Cheers, Viktor -- View this message in context: http://quickfix.13857.n7.nabble.com/Persisting-unprocessed-messages-tp6807p6808.html Sent from the QuickFIX - Dev mailing list archive at Nabble.com. |
|
From: Simeon S. <sim...@gm...> - 2016-05-23 19:47:40
|
Thanks Viktor, I imagine this is a common problem that one needs to guard against. What is the common solution? One way would be to take messages from quickfix with minimal processing and persist them at application level then process them in a separate thread/process - this offloads persisting responsibilities to the application. Another way would be to persist the SeqNum of the last processed message and then at login request from the acceptor all messages after this. However, I will be relying on the acceptor supporting this functionality (i.e. keeping a history of messages). Do these methods sound like a reasonable way to deal with this? On 23 May 2016 at 18:36, Viktor Pogrebnyak <pohrebnyak@i.ua> wrote: > QuickFIX Documentation: http://www.quickfixengine.org/quickfix/doc/html/ > > Hi, > > Quickfix extracts/processes messages from network buffer one-by-one and > writes them to the disk if configured. There is no way to force behavior > you > want in the current engine. > > Cheers, > Viktor > > > > -- > View this message in context: > http://quickfix.13857.n7.nabble.com/Persisting-unprocessed-messages-tp6807p6808.html > Sent from the QuickFIX - Dev mailing list archive at Nabble.com. > > > ------------------------------------------------------------------------------ > Mobile security can be enabling, not merely restricting. Employees who > bring their own devices (BYOD) to work are irked by the imposition of MDM > restrictions. Mobile Device Manager Plus allows you to control only the > apps on BYO-devices by containerizing them, leaving personal data > untouched! > https://ad.doubleclick.net/ddm/clk/304595813;131938128;j > _______________________________________________ > Quickfix-developers mailing list > Qui...@li... > https://lists.sourceforge.net/lists/listinfo/quickfix-developers > |
|
From: Viktor P. <pohrebnyak@i.ua> - 2016-05-23 20:21:03
|
Hi, The whole purpose of ResendRequest message is to retrieve messages that weren't processed by the client even when it received them. If service does not keep track of the messages being sent + lacks replay capability then you are in a big trouble, unfortunately. In your case client must not only remember which messages it received but also which were actually processed. Cheers, Viktor -- View this message in context: http://quickfix.13857.n7.nabble.com/Persisting-unprocessed-messages-tp6807p6810.html Sent from the QuickFIX - Dev mailing list archive at Nabble.com. |
|
From: K. F. <kfr...@gm...> - 2016-05-23 22:06:25
|
Hello Simeon! On Mon, May 23, 2016 at 11:51 AM, Simeon Simeonov <sim...@gm...> wrote: > > Hi, I have a quickfix initiator and I am trying to solve the following problem. If I have a burst of fills, quickfix will receive all the fills and start passing them to the application. The fill messages at this point have been received and are held in memory by quickfix. If the application crashes while processing the messages then what is in memory is lost. When the client restarts it has to re-request the unprocessed messages from the server which might not have them. To emphasize what Viktor said, you have a fundamental problem here. No matter how promptly you persist the messages you receive, there is always the possibility that some messages will be lost in flight. As Viktor said, that is what ResendRequest is for, and for guaranteed reliability, it is necessary for the server to remember (presumably by persisting) the messages it has sent, and to honor the ResendRequest. Let's say your client persists received messages "as soon as" it gets them. Let's say the server writes the messages it is sending to its socket and immediately deletes them from memory (because they've been sent). Now let's say your server crashes (for example, a power failure) before the network has actually delivered the messages to your application. Under these assumptions, you have an unrecoverable error because the messages have been lost for good. No matter how quickly you persist the messages you receive, there will always be such an in-flight window where messages can be lost, so you're better off implementing a robust solution that solves the problem, rather than "persisting really fast" in order to reduce the problem to a small, but non-zero level. The FIX solution is to require both sides of the FIX conversation to remember the messages they have sent, and correctly honor the ResendRequest. (You could imagine a protocol where the client sends back to the server an "I have received and persisted this message -- you can delete it on your end now" message. But you would still have to have a way -- some ResendRequest, or equivalent -- to recover any messages that were in flight when you crashed, before you had a chance to persist them.) To recap: You say "When the client restarts it has to re-request the unprocessed messages from the server which might not have them." The fact that "the server ... might not have them" means that your server is fundamentally broken, and there is no way that the FIX protocol can fix that for you. > My question is this - is there a simple way to make quickfix dump messages to disk Yes, QuickFIX will persist messages to disk and/or a database. > so that when the application crashes it can pick from where it left off? Sort of. It depends what you mean by "from where it left off." It can pick up from where it left off successfully persisting messages, but it can't necessarily pick up from where the server left off writing messages to its socket because of the in-flight issue described above. > I have enabled 'PersistMessages=Y' and I have a disk store but it doesn't seem to be picking up from where it left off upon restart. Presumably the server has written more messages than you have persisted, as described above. That is, the client's notion of "where it left off" does not match that of the server. > I am not sure what config I need to get this functionality. There is no client-side configuration that will (or could) solve the in-flight issue. You need to have your server persist the messages it sends and properly honor ResendRequest. That's the FIX way to implement this kind of fully legitimate requirement of being able to recover after a crash. > Kind regards, > Simeon Good luck. K. Frank |
|
From: Michael C. S. <hik...@gm...> - 2016-05-23 22:08:55
|
If the service can not be depended upon to honor a resend request it's not a FIX service and nothing you do on the client side will guarantee against lost data. Unless a TCP socket is closed cleanly you can still lose data. A network card fails for example. What if you persisted the sequence number of the last message processed at the application layer? Then you know which messages to ignore on a replay. Having the application separately be able to detect a duplicate message would be a double safeguard in case the app crashed after processing but just before persistence. On Mon, May 23, 2016 at 4:41 PM Viktor Pogrebnyak <pohrebnyak@i.ua> wrote: > QuickFIX Documentation: http://www.quickfixengine.org/quickfix/doc/html/ > > Hi, > > The whole purpose of ResendRequest message is to retrieve messages that > weren't processed by the client even when it received them. If service does > not keep track of the messages being sent + lacks replay capability then > you > are in a big trouble, unfortunately. In your case client must not only > remember which messages it received but also which were actually processed. > > Cheers, > Viktor > > > > -- > View this message in context: > http://quickfix.13857.n7.nabble.com/Persisting-unprocessed-messages-tp6807p6810.html > Sent from the QuickFIX - Dev mailing list archive at Nabble.com. > > > ------------------------------------------------------------------------------ > Mobile security can be enabling, not merely restricting. Employees who > bring their own devices (BYOD) to work are irked by the imposition of MDM > restrictions. Mobile Device Manager Plus allows you to control only the > apps on BYO-devices by containerizing them, leaving personal data > untouched! > https://ad.doubleclick.net/ddm/clk/304595813;131938128;j > _______________________________________________ > Quickfix-developers mailing list > Qui...@li... > https://lists.sourceforge.net/lists/listinfo/quickfix-developers > |
|
From: Simeon S. <sim...@gm...> - 2016-05-25 16:40:14
|
Yes, ResendRequest is the way to go forward here and even if quickfix had a mechanism of persisting unprocessed (wrt application layer) messages, it will not be a very useful feature. I'll pick the SeqNum from the last persisted message on disk. I know messages are delivered in-order due to TCP so I know if last message I've processed and stored to disk has SeqNum X than I've seen all messages up to and including X. On 23 May 2016 at 23:08, Michael C. Starkie <hik...@gm...> wrote: > QuickFIX Documentation: http://www.quickfixengine.org/quickfix/doc/html/ > > > If the service can not be depended upon to honor a resend request it's not > a FIX service and nothing you do on the client side will guarantee against > lost data. Unless a TCP socket is closed cleanly you can still lose data. A > network card fails for example. > > What if you persisted the sequence number of the last message processed at > the application layer? Then you know which messages to ignore on a replay. > Having the application separately be able to detect a duplicate message > would be a double safeguard in case the app crashed after processing but > just before persistence. > On Mon, May 23, 2016 at 4:41 PM Viktor Pogrebnyak <pohrebnyak@i.ua> wrote: > >> QuickFIX Documentation: http://www.quickfixengine.org/quickfix/doc/html/ >> >> Hi, >> >> The whole purpose of ResendRequest message is to retrieve messages that >> weren't processed by the client even when it received them. If service >> does >> not keep track of the messages being sent + lacks replay capability then >> you >> are in a big trouble, unfortunately. In your case client must not only >> remember which messages it received but also which were actually >> processed. >> >> Cheers, >> Viktor >> >> >> >> -- >> View this message in context: >> http://quickfix.13857.n7.nabble.com/Persisting-unprocessed-messages-tp6807p6810.html >> Sent from the QuickFIX - Dev mailing list archive at Nabble.com. >> >> >> ------------------------------------------------------------------------------ >> Mobile security can be enabling, not merely restricting. Employees who >> bring their own devices (BYOD) to work are irked by the imposition of MDM >> restrictions. Mobile Device Manager Plus allows you to control only the >> apps on BYO-devices by containerizing them, leaving personal data >> untouched! >> https://ad.doubleclick.net/ddm/clk/304595813;131938128;j >> _______________________________________________ >> Quickfix-developers mailing list >> Qui...@li... >> https://lists.sourceforge.net/lists/listinfo/quickfix-developers >> > > > ------------------------------------------------------------------------------ > Mobile security can be enabling, not merely restricting. Employees who > bring their own devices (BYOD) to work are irked by the imposition of MDM > restrictions. Mobile Device Manager Plus allows you to control only the > apps on BYO-devices by containerizing them, leaving personal data > untouched! > https://ad.doubleclick.net/ddm/clk/304595813;131938128;j > _______________________________________________ > Quickfix-developers mailing list > Qui...@li... > https://lists.sourceforge.net/lists/listinfo/quickfix-developers > |
×
Want the latest updates on software, tech news, and AI?
Get latest updates about software, tech news, and AI from SourceForge directly in your inbox once a month.
Thanks for helping keep SourceForge clean.
X