Re: [Quickfixj-users] Seeking Advice: Handling Message Processing Errors and Pausing FIX Messages in QuickFIX/J

[Felipe W. <fel...@gm...>]

Hello Robert,

Thank you very much for sharing your insights.

Based on your advice, I’ve decided to implement a message broker for
processing my FIX messages. This approach seems to align well with ensuring
that our system remains efficient and responsive, particularly during heavy
processing periods.

I have a query that may have been covered in the QFJ documentation, but I
would appreciate your expertise on it. Could you advise on the recommended
procedure for gracefully logging off a session if the message broker
becomes unavailable?

To manage situations where the message broker is down, I'm considering the
following procedure within the `fromApp()` callback:
1. Attempt to enqueue the FIX message onto the broker.
2. If the attempt fails, retry a predefined number of times (the exact
number of retries would be configurable).
3. Should these attempts to place the message on the broker fail, I plan to
initiate a logoff from the FIX session and then throw an exception to
ensure that QFJ does not mark this message sequence as successfully
processed in its session control.
4. Additionally, trigger an alert to notify our team that there is an issue
with the message broker. This will ensure that we are aware of the problem
immediately and can take action to resolve it as quickly as possible.

Do you think this approach is practical? I'm aiming to strike a balance
between robustness and maintaining session integrity, ensuring we don’t
lose messages or unnecessarily disrupt the session with our FIX
counterparties.

Thank you again for your valuable feedback.

Best regards,
Felipe Windmoller

Em dom., 17 de mar. de 2024 14:02, Robert Nicholson <ro...@el...>
escreveu:

> QFJ Documentation: http://www.quickfixj.org/documentation/
> QFJ Support: http://www.quickfixj.org/support/
>
>
> If you cannot heartbeat in time because your main thread is processing the
> message then you will have problems. You want to minimize the need to
> logoff, logon again and send resend requests. The other end isn’t going to
> tolerate that much.
>
> In general when you listen on FIX and you have expensive processing to do
> you should be using a messaging layer and not doing expensive progressing
> in the handlers of your FIX callbacks. You want that code to be as light as
> possible. In general you want to be able to continue listening to FIX
> messages even when you have resource issues on your side related to
> processing the messages. Worse case you have to logoff and potentially
> sleep forever if you cannot “queue” the message for processing later. The
> idea being that your alerting catches that and procedures are in place to
> resume processing later. What you don’t want to do is miss messages because
> the reset takes place whilst you’re down.
>
>
> On Mar 17, 2024, at 8:23 AM, Felipe Windmoller <fel...@gm...>
> wrote:
>
> QFJ Documentation: http://www.quickfixj.org/documentation/
> QFJ Support: http://www.quickfixj.org/support/
>
>
> Dear QuickFIX/J community,
>
> I am seeking advice on how to manage messages from a drop copy session.
> Each business message must be processed by a separate application, and I
> need to ensure that every message is processed without fail.
>
> I am considering how to handle scenarios where, during the execution of
> the fromApp() method, calling this external application results in an
> error. Is it feasible to pause the processing of my FIX messages until the
> external call is successful? In such a scenario, the pending messages would
> accumulate. Would this approach be advisable? Additionally, I am curious
> about the mechanisms available for halting QFJ processing in response to an
> error encountered within the fromApp() method. Specifically, how can I
> effectively stop the processing of messages by QFJ at this juncture? Would
> it involve rejecting the problematic FIX message, thereby necessitating its
> resending by the FIX counterparty?
>
> I have contemplated the following strategies:
>
>    1. 1. Utilizing the QFJ Incoming Log:
>
>    My idea is to configure the QFJ application to solely receive and log
>    FIX messages. A separate application would then periodically (every 1
>    minute) check this log, guided by a control table that tracks the last
>    sequence number processed. It would then process the messages accordingly.
>    Point of Attention:
>
>    As Christoph John has elucidated on StackOverflow
>    <https://stackoverflow.com/a/78167504/11110455>, *"QFJ logs messages
>    upon receipt at AbstractIoHandler.java
>    <https://github.com/quickfix-j/quickfixj/blob/d7c9fe52fabdc56f28ace4063f6ac78fe422b511/quickfixj-core/src/main/java/quickfix/mina/AbstractIoHandler.java#L142>.
>    Message parsing and subsequent processing occur later; only then is the
>    message conveyed to the Session class where it undergoes validation,
>    leading to a potential call to fromApp() if validation is successful."*
>    Consequently, there's an inherent uncertainty about processing solely valid
>    messages.
>
>
>    2.
>    3. 2. Employing a Message Broker:
>
>    Another approach is to leverage a message broker as an intermediary in
>    the fromApp() method to queue the received messages. This would
>    essentially serve as a buffer, allowing another application to process
>    these messages asynchronously.
>    Point of Attention:
>
>    A critical consideration is the strategy to adopt if the message
>    broker becomes unavailable during the execution of a fromApp() method
>    call.
>
>
>    4.
>    5. 3. Storing Pending Messages in a Database Table:
>
>    Similar to using a message broker, this method involves storing the
>    messages received in the fromApp() method in a database table. This
>    table would then act as a buffer until the messages can be processed.
>
>
>
> I greatly appreciate your insights and suggestions on these approaches or
> any other potential solutions you might recommend.
>
> Thank you, Felipe Windmoller
> _______________________________________________
> Quickfixj-users mailing list
> Qui...@li...
> https://lists.sourceforge.net/lists/listinfo/quickfixj-users
>
>
> _______________________________________________
> Quickfixj-users mailing list
> Qui...@li...
> https://lists.sourceforge.net/lists/listinfo/quickfixj-users
>