Re: [Quickfix-developers] Network disconnect recovery testing
Brought to you by:
orenmnero
From: <daw...@ya...> - 2017-05-10 17:32:57
|
Hi K. Frank, Firstly apologies for my very tardy response - I got sidetracked on another project and didn't want to send a cursory reply after the effort you put in. So you hit the nail on the head, yes our strategy will be sending limit orders on both sides until they are filled, with limits being changed continuously. I'm using separate new order/cancel pairs rather than cancel-replace because limit orders may be replaced by a pegged algo (and visa versa) and that change isn't supported by the broker via cancel-replace. Also, yes there will be cases where orders are cancelled outright (not individually but as a cancel-all function) and you're right it would make sense to have them included as a resend so I'll use the toApp() to control that logic for those orders. [You still have a small potential issue. According to you, after sending the initial order for a ticker/side, you only send pairs of cancel / new orders. It's possible that before a disconnect your cancel makes it through, but your new order doesn't] - this won't be a problem as the new order will not be triggered until I get the cancel confirm back. [My plan is to have process which runs 120 seconds after each successful logon] - this will work probably 95% of the time but yes, point taken - it's hardly bullet proof. I could possibly use the QuickFix event "resend ..... has been satisfied" to check when a resend has ended but your suggestion of making an order status requests would lay all doubt to rest so I'm going to try that one out. Automated trading errors.. yes I've seen the damage algos can do, especially the volume driven ones! Thanks again for the great advice. I'll probably be back with more on this one! Regards, Dermot ----- Original Message ----- >From: K. Frank <kfr...@gm...> >To: Quickfix Developers List <qui...@li...> >Date: 2017/4/20, Thu 06:15 >Subject: Re: [Quickfix-developers] Network disconnect recovery testing > >QuickFIX Documentation: http://www.quickfixengine.org/quickfix/doc/html/ > >Hello Dermot! > >On Tue, Apr 18, 2017 at 9:48 AM, <daw...@ya...> wrote: >> Hi K. Frank, >> >> Really appreciate the comprehensive response. Advice from the trenches is >> exactly what I need! >> >> So actually for this business case we do NOT want to resend any orders or >> cancels that didn't get through during downtime. The order submission is >> fast-paced and the limit prices may be stale by the time we reconnect. > >This is reasonable. > >> Also, cancels are always linked to new orders. > >> (we don't allow more than one open order of the same ticker/side) > >>From this I take it that your trading strategy, roughly speaking, >consists of having outstanding limit orders (possibly on both sides) >for a set of securities, more or less continuously, at least until they >get filled. You do, however, adjust limit prices, by first cancelling such >an order, and then issuing a new order with a new limit price (rather >than issuing a single cancel-replace request that, in essence, modifies >the limit price of an existing order. > >As side note: I would imagine that, if only as a matter of operational risk >control, there would be instances where you would want to cancel an >order without then issuing a new order. It's your business logic, so >maybe you really don't ever do this, but it would seem prudent to have >this capability. Anyway, if you do ever have "unpaired" cancels, it would >seem that you would want to resend them in the event of a reconnect and >a resend request from your counter-party. > >> This makes the approach simple - >> any unacked messages can be 'cancelled' on our side. You're right about >> deleting orders, that's not a good idea. Instead will set them to >> 'Withdrawn' status. > >This seems reasonable. When QuickFIX reconnects for you and responds >to the resend request from your counter-party, it will give you a chance to >intervene before it actually resends a given message. It does this through >the Application::toApp() callback that you can override in your MyApp (or >whatever) class that you have derived from Application. You can then tweak >the messages before they are sent, or -- as in your use case -- throw a >DoNotSend exception, in which case QuickFIX will gap-fill the sequence >number (in effect telling your counter-party that the missed message was >an ignorable admin message), rather than resend the message. > >> My plan is to have process which runs 120 seconds after each successful >> logon [that should be enough time for any missing acks/fills to get back in >> the resend]. > >This should work. However, as a matter of general practice, using this kind >of magic-number wait time to insure that something has happened should >be avoided when possible. > >When you reconnect, your counter-party tells you what sequence number >it's on. You (i.e., QuickFIX) notice that you've missed some messages so >you (QuickFIX) issue a resend request. You can then wait until your >counter-party is done with its resend. > >I'm not entirely sure how to get QuickFIX to tell you that the resend is done, >though. The resent messages ought to have their PosDupFlag set, so it >might work to wait until you get a message that was sent post reconnect >because its PosDupFlag isn't set. (QuickFIX queues up out-of-order >messages for you, so messages should delivered to your application logic >in order after the resend is complete.) Maybe you could status a known >good order and use its status message as a flag that the resend is complete. > >(It would be nice if you could get a callback from QuickFIX or query it directly >somehow to get it to tell you that the resend is complete. I don't know of a >way to do this, but perhaps there is one.) > >> The process checks if there are any orders in 'Sent' status >> (unacked) and created before last logon time - update these orders to >> 'Withdrawn' status. > >Yes, so the logic would be something like: > >Inspect orders that QuickFIX is resending on your behalf in the toApp callback, >and throw the DoNotSend exception so that you don't resend new orders and >cancel requests that presumably haven't been sent. > >Wait until your counter-party's resend is complete (e.g., as outlined >above), and >then update unacked orders in your own order store to your proposed "Withdrawn" >status. > >> Simple but should do the job. Definitely need to get unacked orders out of >> the way as they will interfere with new orders being sent (we don't allow >> more than one open order of the same ticker/side). There's no auto-cancel >> requirements from the broker so the policy is up to us. > >You still have a small potential issue. According to you, after >sending the initial >order for a ticker/side, you only send pairs of cancel / new orders. >It's possible >that before a disconnect your cancel makes it through, but your new >order doesn't. > >So you will want to have a little bit of logic that recognizes this >situation and lets >you send another new order without it being part of a cancel / new order pair >(because the cancel was already sent and correctly processed). > >> What do you think? > >If you (try to) send an order, reconnect, get all of your >counter-party's messages >through a proper resend request, and you don't get an ack for that >order, then it's >fair to say that your counter-party has told you that he did not >receive the order. > >But relying on your counter-party's silence to communicate this could be less >reliable that getting an affirmative statement that the order was not received. > >To be redundantly cautious, you could proactively send status requests for your >unacked orders, and if your counter-party never got them, you should get back >some kind of unknown order message. I believe that the FIX specification says >you can do this using the ClOrdID (the order id generated on your end and sent >as part of your new order message). However, bear in mind, some counter-parties >may require that you status the order using the OrdID (the order id assigned by >the counter-party and sent back in the ack). Well, since you never got an ack, >you wouldn't be able to do it this way. (As I said, I don't believe >that requiring the >OrdID in place of the ClOrdID is really meets the spec, but it is a >pitfall to watch >out for.) > >Similarly, if you tried to send a cancel (in your case, presumably as part of a >cancel / new order pair), but never got a ur-out, you could status the order to >confirm that it is still alive, rather than relying on silence (i.e., >the absence of >the ur-out) to indicate that the order is alive. > >Although this kind of double-check requires building some additional application >logic, it will probably be worth it if you're going to trade any >significant flow >through your application or your application is going to be in service for more >than, say, a few months. > >Trust me, stuff happens. > >It always seems cheaper not to invest in this kind of additional >development cost. >Cheaper, at least, until you're on the losing side of a trading error >(and automated- >trading trading errors can blow up in a hurry). > >> Thanks! >> Dermot > > >Happy Trading! > > >K. Frank > > >> ----- Original Message ----- >> From: K. Frank <kfr...@gm...> >> To: Quickfix Developers List <qui...@li...> >> Date: 2017/4/17, Mon 21:33 >> Subject: Re: [Quickfix-developers] Network disconnect recovery testing >> >> Hi Dermot! >> >> On Mon, Apr 17, 2017 at 5:45 AM, <daw...@ya...> wrote: >>> ... >>> Hi Mike, >>> >>> Just getting back to orders that are sent (and don't go anywhere) during >>> network downtime - is there a recommended approach to identifying and >>> cleaning these up? I'm thinking to just run a process every minute or so >>> that checks "if currently logged on and order hasn't been acked in the last >>> minute then delete from database". What do you think? >> >> Some words of advice from the trenches ... >> >> Think through your recovery strategy carefully (as you seem to >> be doing). >> ... > >------------------------------------------------------------------------------ >Check out the vibrant tech community on one of the world's most >engaging tech sites, Slashdot.org! http://sdm.link/slashdot >_______________________________________________ >Quickfix-developers mailing list >Qui...@li... >https://lists.sourceforge.net/lists/listinfo/quickfix-developers > > > |