Re: [Quickfixj-users] Seeking Advice: Handling Message Processing Errors and Pausing FIX Messages in QuickFIX/J

[Robert N. <ro...@el...>]

If you cannot heartbeat in time because your main thread is processing the message then you will have problems. You want to minimize the need to logoff, logon again and send resend requests. The other end isn’t going to tolerate that much.

In general when you listen on FIX and you have expensive processing to do you should be using a messaging layer and not doing expensive progressing in the handlers of your FIX callbacks. You want that code to be as light as possible. In general you want to be able to continue listening to FIX messages even when you have resource issues on your side related to processing the messages. Worse case you have to logoff and potentially sleep forever if you cannot “queue” the message for processing later. The idea being that your alerting catches that and procedures are in place to resume processing later. What you don’t want to do is miss messages because the reset takes place whilst you’re down.


> On Mar 17, 2024, at 8:23 AM, Felipe Windmoller <fel...@gm...> wrote:
> 
> QFJ Documentation: http://www.quickfixj.org/documentation/
> QFJ Support: http://www.quickfixj.org/support/
> 
> 
> Dear QuickFIX/J community,
> 
> I am seeking advice on how to manage messages from a drop copy session. Each business message must be processed by a separate application, and I need to ensure that every message is processed without fail.
> 
> I am considering how to handle scenarios where, during the execution of the fromApp() method, calling this external application results in an error. Is it feasible to pause the processing of my FIX messages until the external call is successful? In such a scenario, the pending messages would accumulate. Would this approach be advisable? Additionally, I am curious about the mechanisms available for halting QFJ processing in response to an error encountered within the fromApp() method. Specifically, how can I effectively stop the processing of messages by QFJ at this juncture? Would it involve rejecting the problematic FIX message, thereby necessitating its resending by the FIX counterparty?
> 
> I have contemplated the following strategies:
> 
> 1. Utilizing the QFJ Incoming Log:
> My idea is to configure the QFJ application to solely receive and log FIX messages. A separate application would then periodically (every 1 minute) check this log, guided by a control table that tracks the last sequence number processed. It would then process the messages accordingly.
> Point of Attention:
> As Christoph John has elucidated on StackOverflow <https://stackoverflow.com/a/78167504/11110455>, "QFJ logs messages upon receipt at AbstractIoHandler.java <https://github.com/quickfix-j/quickfixj/blob/d7c9fe52fabdc56f28ace4063f6ac78fe422b511/quickfixj-core/src/main/java/quickfix/mina/AbstractIoHandler.java#L142>. Message parsing and subsequent processing occur later; only then is the message conveyed to the Session class where it undergoes validation, leading to a potential call to fromApp() if validation is successful." Consequently, there's an inherent uncertainty about processing solely valid messages.
> 
> 
> 2. Employing a Message Broker:
> Another approach is to leverage a message broker as an intermediary in the fromApp() method to queue the received messages. This would essentially serve as a buffer, allowing another application to process these messages asynchronously.
> Point of Attention:
> A critical consideration is the strategy to adopt if the message broker becomes unavailable during the execution of a fromApp() method call.
> 
> 
> 3. Storing Pending Messages in a Database Table:
> Similar to using a message broker, this method involves storing the messages received in the fromApp() method in a database table. This table would then act as a buffer until the messages can be processed.
> 
> I greatly appreciate your insights and suggestions on these approaches or any other potential solutions you might recommend.
> 
> Thank you, Felipe Windmoller
> _______________________________________________
> Quickfixj-users mailing list
> Qui...@li...
> https://lists.sourceforge.net/lists/listinfo/quickfixj-users