Re: [Quickfix-developers] Network disconnect recovery testing

[<daw...@ya...>]

Hi K. Frank,

Really appreciate the comprehensive response. Advice from the trenches is exactly what I need!

So actually for this business case we do NOT want to resend any orders or cancels that didn't get through during downtime. The order submission is fast-paced and the limit prices may be stale by the time we reconnect. Also, cancels are always linked to new orders. This makes the approach simple - any unacked messages can be 'cancelled' on our side. You're right about deleting orders, that's not a good idea. Instead will set them to 'Withdrawn' status.

My plan is to have process which runs 120 seconds after each successful logon [that should be enough time for any missing acks/fills to get back in the resend]. The process checks if there are any orders in 'Sent' status (unacked) and created before last logon time - update these orders to 'Withdrawn' status.

Simple but should do the job. Definitely need to get unacked orders out of the way as they will interfere with new orders being sent (we don't allow more than one open order of the same ticker/side). There's no auto-cancel requirements from the broker so the policy is up to us.

What do you think?


Thanks!
Dermot




----- Original Message -----
>From: K. Frank <kfr...@gm...>
>To: Quickfix Developers List <qui...@li...> 
>Date: 2017/4/17, Mon 21:33
>Subject: Re: [Quickfix-developers] Network disconnect recovery testing
> 
>QuickFIX Documentation: http://www.quickfixengine.org/quickfix/doc/html/
>
>Hi Dermot!
>
>On Mon, Apr 17, 2017 at 5:45 AM,  <daw...@ya...> wrote:
>> ...
>> Hi Mike,
>>
>> Just getting back to orders that are sent (and don't go anywhere) during network downtime - is there a recommended approach to identifying and cleaning these up? I'm thinking to just run a process every minute or so that checks "if currently logged on and order hasn't been acked in the last minute then delete from database". What do you think?
>
>Some words of advice from the trenches ...
>
>Think through your recovery strategy carefully (as you seem to
>be doing).
>
>First, if you send an order and receive an ack, then your counter-
>party has accepted responsibility for that order.  (Hopefully he
>fulfills his responsibility.)
>
>If you try to send an order and don't receive an ack, you don't
>know whether your counter-party has it or not.  (Send order,
>receive and process order, send ack -- network goes down --
>don't receive ack.  Note, in this scenario, your counter-party
>doesn't know -- until and unless you request a gap fill -- that
>you didn't receive the ack.  FIX doesn't ack acks.)  Note, this
>means that you REALLY don't want to delete unacked orders
>from your database and forget about them -- they might be
>live on your counter-party's side.  So you need, at least, to
>put them in some kind of limbo state, and have a protocol
>for cleaning them up (or leaving them live when you get your
>ack after a reconnect and gap fill).
>
>You need to decide on a business protocol for handling network
>outages.  Some counter-parties will let you (or require) that you
>have them (pre-arranged -- not via FIX) auto-cancel any orders
>if connectivity drops.  Note such a "cure" can be worse than the
>problem if your network goes down for a few milliseconds or a
>few seconds or even a few minutes.  It depends on your use case.
>
>Similarly, you need to decide what you want to do with orders
>that you have attempted to issue but haven't sent yet.  Again,
>if the outage was only for a few milliseconds or seconds, you
>might be best to just send them.
>
>The problem is with orders that you might have sent.  You can't
>really not resend them in a gap fill because that could be
>inconsistent with the traffic you counter-party has already seen.
>You could negotiate with your counter-party an "auto-cancel" policy
>for new, previously unseen orders that come in a gap fill.
>
>Let's say you do decide to filter out (somehow) possibly unsent traffic
>when gap filling after an outage.  While you might want to "filter out"
>new orders (or have your counter-party ignore / auto-cancel them)
>you almost certainly do not want to filter out possibly unsent cancel
>requests.
>
>You are right that it makes a lot of sense to monitor whether you
>have connectivity with your counter-party.  Note there are several
>levels to this:
>
>Business "connectivity" -- e.g., receive acks
>FIX connectivity -- heartbeats, isLoggedOn
>Network connectivity -- e.g., a free-standing "ping" watchdog
>
>It does make sense -- to reduce potential gap-fill load and economic
>exposure if you don't manage a timely reconnect -- to pause
>sending orders on your side if you detect possible connectivity
>interruption.  You may or may not wish to queue up such unsent
>traffic to send if you reestablish connectivity after a "short" amount
>of time.  Note, there is real business logic in how you decide to
>handle this.  Would you really not want to send cancel requests
>that you generated and got queued up during a (real or imagined)
>loss of connectivity?
>
>Repeating two key points:
>
>It is well worth designing your recovery strategy with care.
>
>The details of your strategy will depend on your specific business
>use case -- auto-cancellation policies, how long an outage puts
>you into a clean-up mode, rather than recovering and proceeding
>normally (keeping orders live, etc.).
>
>And last:  Test to the extent you can afford and your business requires.
>I have NEVER seen automated recovery work completely correctly in
>institutional environments -- even with reasonably well-tested systems.
>(I'm not saying it can't happen -- I've just never seen it.)
>
>
>> Dermot
>
>
>Good luck.
>
>
>K. Frank
>
>------------------------------------------------------------------------------
>Check out the vibrant tech community on one of the world's most
>engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>_______________________________________________
>Quickfix-developers mailing list
>Qui...@li...
>https://lists.sourceforge.net/lists/listinfo/quickfix-developers
>
>
>