Re: [Quickfix-developers] Network disconnect recovery testing

[K. F. <kfr...@gm...>]

Hello Dermot!

On Tue, Apr 18, 2017 at 9:48 AM,  <daw...@ya...> wrote:
> Hi K. Frank,
>
> Really appreciate the comprehensive response. Advice from the trenches is
> exactly what I need!
>
> So actually for this business case we do NOT want to resend any orders or
> cancels that didn't get through during downtime. The order submission is
> fast-paced and the limit prices may be stale by the time we reconnect.

This is reasonable.

> Also, cancels are always linked to new orders.

> (we don't allow more than one open order of the same ticker/side)

>From this I take it that your trading strategy, roughly speaking,
consists of having outstanding limit orders (possibly on both sides)
for a set of securities, more or less continuously, at least until they
get filled.  You do, however, adjust limit prices, by first cancelling such
an order, and then issuing a new order with a new limit price (rather
than issuing a single cancel-replace request that, in essence, modifies
the limit price of an existing order.

As side note: I would imagine that, if only as a matter of operational risk
control, there would be instances where you would want to cancel an
order without then issuing a new order.  It's your business logic, so
maybe you really don't ever do this, but it would seem prudent to have
this capability.  Anyway, if you do ever have "unpaired" cancels, it would
seem that you would want to resend them in the event of a reconnect and
a resend request from your counter-party.

> This makes the approach simple -
> any unacked messages can be 'cancelled' on our side. You're right about
> deleting orders, that's not a good idea. Instead will set them to
> 'Withdrawn' status.

This seems reasonable.  When QuickFIX reconnects for you and responds
to the resend request from your counter-party, it will give you a chance to
intervene before it actually resends a given message.  It does this through
the Application::toApp() callback that you can override in your MyApp (or
whatever) class that you have derived from Application.  You can then tweak
the messages before they are sent, or -- as in your use case -- throw a
DoNotSend exception, in which case QuickFIX will gap-fill the sequence
number (in effect telling your counter-party that the missed message was
an ignorable admin message), rather than resend the message.

> My plan is to have process which runs 120 seconds after each successful
> logon [that should be enough time for any missing acks/fills to get back in
> the resend].

This should work.  However, as a matter of general practice, using this kind
of magic-number wait time to insure that something has happened should
be avoided when possible.

When you reconnect, your counter-party tells you what sequence number
it's on.  You (i.e., QuickFIX) notice that you've missed some messages so
you (QuickFIX) issue a resend request.  You can then wait until your
counter-party is done with its resend.

I'm not entirely sure how to get QuickFIX to tell you that the resend is done,
though.  The resent messages ought to have their PosDupFlag set, so it
might work to wait until you get a message that was sent post reconnect
because its PosDupFlag isn't set.  (QuickFIX queues up out-of-order
messages for you, so messages should delivered to your application logic
in order after the resend is complete.)  Maybe you could status a known
good order and use its status message as a flag that the resend is complete.

(It would be nice if you could get a callback from QuickFIX or query it directly
somehow to get it to tell you that the resend is complete.  I don't know of a
way to do this, but perhaps there is one.)

> The process checks if there are any orders in 'Sent' status
> (unacked) and created before last logon time - update these orders to
> 'Withdrawn' status.

Yes, so the logic would be something like:

Inspect orders that QuickFIX is resending on your behalf in the toApp callback,
and throw the DoNotSend exception so that you don't resend new orders and
cancel requests that presumably haven't been sent.

Wait until your counter-party's resend is complete (e.g., as outlined
above), and
then update unacked orders in your own order store to your proposed "Withdrawn"
status.

> Simple but should do the job. Definitely need to get unacked orders out of
> the way as they will interfere with new orders being sent (we don't allow
> more than one open order of the same ticker/side). There's no auto-cancel
> requirements from the broker so the policy is up to us.

You still have a small potential issue.  According to you, after
sending the initial
order for a ticker/side, you only send pairs of cancel / new orders.
It's possible
that before a disconnect your cancel makes it through, but your new
order doesn't.

So you will want to have a little bit of logic that recognizes this
situation and lets
you send another new order without it being part of a cancel / new order pair
(because the cancel was already sent and correctly processed).

> What do you think?

If you (try to) send an order, reconnect, get all of your
counter-party's messages
through a proper resend request, and you don't get an ack for that
order, then it's
fair to say that your counter-party has told you that he did not
receive the order.

But relying on your counter-party's silence to communicate this could be less
reliable that getting an affirmative statement that the order was not received.

To be redundantly cautious, you could proactively send status requests for your
unacked orders, and if your counter-party never got them, you should get back
some kind of unknown order message.  I believe that the FIX specification says
you can do this using the ClOrdID (the order id generated on your end and sent
as part of your new order message).  However, bear in mind, some counter-parties
may require that you status the order using the OrdID (the order id assigned by
the counter-party and sent back in the ack).  Well, since you never got an ack,
you wouldn't be able to do it this way.  (As I said, I don't believe
that requiring the
OrdID in place of the ClOrdID is really meets the spec, but it is a
pitfall to watch
out for.)

Similarly, if you tried to send a cancel (in your case, presumably as part of a
cancel / new order pair), but never got a ur-out, you could status the order to
confirm that it is still alive, rather than relying on silence (i.e.,
the absence of
the ur-out) to indicate that the order is alive.

Although this kind of double-check requires building some additional application
logic, it will probably be worth it if you're going to trade any
significant flow
through your application or your application is going to be in service for more
than, say, a few months.

Trust me, stuff happens.

It always seems cheaper not to invest in this kind of additional
development cost.
Cheaper, at least, until you're on the losing side of a trading error
(and automated-
trading trading errors can blow up in a hurry).

> Thanks!
> Dermot


Happy Trading!


K. Frank


> ----- Original Message -----
> From: K. Frank <kfr...@gm...>
> To: Quickfix Developers List <qui...@li...>
> Date: 2017/4/17, Mon 21:33
> Subject: Re: [Quickfix-developers] Network disconnect recovery testing
>
> Hi Dermot!
>
> On Mon, Apr 17, 2017 at 5:45 AM,  <daw...@ya...> wrote:
>> ...
>> Hi Mike,
>>
>> Just getting back to orders that are sent (and don't go anywhere) during
>> network downtime - is there a recommended approach to identifying and
>> cleaning these up? I'm thinking to just run a process every minute or so
>> that checks "if currently logged on and order hasn't been acked in the last
>> minute then delete from database". What do you think?
>
> Some words of advice from the trenches ...
>
> Think through your recovery strategy carefully (as you seem to
> be doing).
> ...