quickfix-users

[Quickfix-users] QuickFIX/J newbie needs tips on robustness

[Scott M. <dl....@gm...>]

I'm a FIX (and QuickFIX) newbie, using QuickFIX/J to write a simple
trading application.  The basic system design includes a database into
which pending orders are written, and my application, which
periodically sweeps the database, placing any pending orders.

Naturally, I want to ensure that orders aren't placed twice (because
they'll appear to the broker as two separate orders), so after sending
the order message to the broker, my application updates the database
to say that the order has been sent.  Later, when the order is
accepted or rejected by the broker, the database will be updated again
to say so.

One concern I have is this: suppose the power goes out (or pick your
favorite disaster) after the order message has been sent but before
the database has been updated to say it was sent.  Now, when my
application wakes up again, it will think the order was never sent,
and will (re-)send it to the broker.

Realizing this, I attacked the problem in two parts.  First, I changed
the code to do things the other way around: flag the order as sent,
then send the order message.  Second, when my application starts up,
it looks at all orders flagged as sent (but not yet acknowledged by
the broker) and resends them with PossResend=Y.

That seemed great -- except that it turns out that my particular
broker doesn't support orders with PossResend=Y.

*Sigh*.

So I changed it again: now, when my application starts up, it sends an
OrderStatusRequest for the order, which should result in an
ExecutionRpt giving the order's status; I figured that if my broker
indicated they'd never heard of the order, I should re-send it.

That seemed great -- except that it turns out that my broker rejects
OrderStatusRequest unless it includes their OrderID (not ClOrdID), and
you don't get their OrderID until you've received the acknowledgment
to your order request.  And the orders I'm worried about are precisely
the ones for which I don't have that information, because the power
went out at the wrong time.

I ran into the same problem (my broker demands an OrderID I don't yet
have in the case I care about) when I tried to use OrderCancelRequest,
and when I tried to use OrderCancelReplaceRequest.  I couldn't find
anything else promising in the FIX spec, and am now officially out of
ideas.

So, O wise and experienced FIX folks, how would you solve this
problem?  Is there a way to write my application so that it will
robustly recover from the state I'm concerned about?  My broker
supports FIX4.2, if it matters.  (Though not as completely as I would
like, as you can see. ;-)

-- 
"Life results from the non-random survival of randomly varying
replicators." -- Richard Dawkins

Re: [Quickfix-users] QuickFIX/J newbie needs tips on robustness

[Jerry W. <je...@we...>]

Your Brokers support is (In my experience) fairly average...
I really don't expect too much more from brokers.

I have a question to your problem:
1) Is it one?

The Fix Specs say that the ClOrdID is Unique per Order.
I would expect your broker to check this.  
If the ClOrdID already exists then He should reject the NewOrderSingle.

In this form, you have no problem with sending duplicates.

As long as your ClOrdID remains the same...

Of course if he rejects the NewOrderSingle, you still got the problem
of finding out what OrderId He gave to the thing....

But Someday some thing will happen to the order resulting in a message,
and the message will have the OrderID...  

Check if your broker accepts Deletes without OrderID, so that deletes are 
possible durring this unknown period.


2) Isn't this a rather theoretical case?Won't QuickFix Recovery get this for 
you?
Jerry Westrick.


On Wednesday 19 July 2006 06:39, Scott Maxwell wrote:
> QuickFIX Documentation:
> http://www.quickfixengine.org/quickfix/doc/html/index.html QuickFIX
> Support: http://www.quickfixengine.org/services.html
>
> I'm a FIX (and QuickFIX) newbie, using QuickFIX/J to write a simple
> trading application.  The basic system design includes a database into
> which pending orders are written, and my application, which
> periodically sweeps the database, placing any pending orders.
>
> Naturally, I want to ensure that orders aren't placed twice (because
> they'll appear to the broker as two separate orders), so after sending
> the order message to the broker, my application updates the database
> to say that the order has been sent.  Later, when the order is
> accepted or rejected by the broker, the database will be updated again
> to say so.
>
> One concern I have is this: suppose the power goes out (or pick your
> favorite disaster) after the order message has been sent but before
> the database has been updated to say it was sent.  Now, when my
> application wakes up again, it will think the order was never sent,
> and will (re-)send it to the broker.
>
> Realizing this, I attacked the problem in two parts.  First, I changed
> the code to do things the other way around: flag the order as sent,
> then send the order message.  Second, when my application starts up,
> it looks at all orders flagged as sent (but not yet acknowledged by
> the broker) and resends them with PossResend=Y.
>
> That seemed great -- except that it turns out that my particular
> broker doesn't support orders with PossResend=Y.
>
> *Sigh*.
>
> So I changed it again: now, when my application starts up, it sends an
> OrderStatusRequest for the order, which should result in an
> ExecutionRpt giving the order's status; I figured that if my broker
> indicated they'd never heard of the order, I should re-send it.
>
> That seemed great -- except that it turns out that my broker rejects
> OrderStatusRequest unless it includes their OrderID (not ClOrdID), and
> you don't get their OrderID until you've received the acknowledgment
> to your order request.  And the orders I'm worried about are precisely
> the ones for which I don't have that information, because the power
> went out at the wrong time.
>
> I ran into the same problem (my broker demands an OrderID I don't yet
> have in the case I care about) when I tried to use OrderCancelRequest,
> and when I tried to use OrderCancelReplaceRequest.  I couldn't find
> anything else promising in the FIX spec, and am now officially out of
> ideas.
>
> So, O wise and experienced FIX folks, how would you solve this
> problem?  Is there a way to write my application so that it will
> robustly recover from the state I'm concerned about?  My broker
> supports FIX4.2, if it matters.  (Though not as completely as I would
> like, as you can see. ;-)

Re: [Quickfix-users] QuickFIX/J newbie needs tips on robustness

[Scott M. <dl....@gm...>]

On 7/19/06, Jerry Westrick <je...@we...> wrote:

> The Fix Specs say that the ClOrdID is Unique per Order.
> I would expect your broker to check this.
> If the ClOrdID already exists then He should reject the NewOrderSingle.

Ah, clever idea.  I hadn't thought of just trying that.

Unfortunately, it didn't work out: they treat it as a new, separate
order.  So now, as far as they're concerned, I simply have two
identical orders with the same ClOrdID.

Good thing to try, though.


> 2) Isn't this a rather theoretical case?Won't QuickFix Recovery get this for
> you?

I am of course a newbie, so I can't speak too authoritatively.  Maybe
you're right.  It just seemed to me that since there's no way to both
(1) send the message (and guarantee it got to the broker) and (2)
update the database, all as part of the same guaranteed-atomic
operation, then you can always construct a scenario where you lose if
the power goes out at just the wrong moment.

You're right that it's a theoretical case, but you know how it is ...
if you don't deal with it, it'll come back and bite you at the worst
possible moment.

I think my strategy is going to be that I won't try to autonomously
recover -- there's just no way to do it, at least not that I can find,
in my particular circumstances.  Instead, we'll have to manually
recover (via phone calls to the broker or something) when the startup
code sees an order is marked as sent but has no broker-side order ID.

Thanks for the suggestions!

-- 
"Life results from the non-random survival of randomly varying
replicators." -- Richard Dawkins

Re: [Quickfix-users] QuickFIX/J newbie needs tips on robustness

[Jerry W. <je...@we...>]

If the brokeris using QuicklFix then I think the case would be theoretical.
But the level of Fix support offered by your broker leads me to disbelieve 
that they did the recovery correct either  :(

I assume you asked your broker how they want this handled, and your 
request fell on deaf ears...

If so you are forced to drop to manual processing as described.

what can I say...  Silly broker ;)

Jerry Westrick



On Thursday 20 July 2006 07:28, Scott Maxwell wrote:
> QuickFIX Documentation:
> http://www.quickfixengine.org/quickfix/doc/html/index.html QuickFIX
> Support: http://www.quickfixengine.org/services.html
>
> On 7/19/06, Jerry Westrick <je...@we...> wrote:
> > The Fix Specs say that the ClOrdID is Unique per Order.
> > I would expect your broker to check this.
> > If the ClOrdID already exists then He should reject the NewOrderSingle.
>
> Ah, clever idea.  I hadn't thought of just trying that.
>
> Unfortunately, it didn't work out: they treat it as a new, separate
> order.  So now, as far as they're concerned, I simply have two
> identical orders with the same ClOrdID.
>
> Good thing to try, though.
>
> > 2) Isn't this a rather theoretical case?Won't QuickFix Recovery get this
> > for you?
>
> I am of course a newbie, so I can't speak too authoritatively.  Maybe
> you're right.  It just seemed to me that since there's no way to both
> (1) send the message (and guarantee it got to the broker) and (2)
> update the database, all as part of the same guaranteed-atomic
> operation, then you can always construct a scenario where you lose if
> the power goes out at just the wrong moment.
>
> You're right that it's a theoretical case, but you know how it is ...
> if you don't deal with it, it'll come back and bite you at the worst
> possible moment.
>
> I think my strategy is going to be that I won't try to autonomously
> recover -- there's just no way to do it, at least not that I can find,
> in my particular circumstances.  Instead, we'll have to manually
> recover (via phone calls to the broker or something) when the startup
> code sees an order is marked as sent but has no broker-side order ID.
>
> Thanks for the suggestions!

Re: [Quickfix-users] QuickFIX/J newbie needs tips on robustness

[Joerg T. <Joe...@ma...>]

On 07/20/06 07:28, Scott Maxwell wrote:
> On 7/19/06, Jerry Westrick <je...@we...> wrote:
>=20
>> The Fix Specs say that the ClOrdID is Unique per Order.
>> I would expect your broker to check this.
>> If the ClOrdID already exists then He should reject the NewOrderSingle.
>=20
> Ah, clever idea.  I hadn't thought of just trying that.

Jerry is right here, the broker should support the OrderStatusRequest(ClO=
rdID) without the need to=20
     give the OrderID. Without this feature, you have to revert to manual=
 operation.

Otherwise the following scenario would work a bit better:

(1) Generate order with new ClOrdID in your database.
(2) Send order to QF; if sendToTarget() returns, the order is in the QF o=
utgoing queue
(3) Update database status: NEW --> SENT_TO_FIX

On recovery, you request the state of all known orders:

	OrderStatusRequest(ClOrdID)

If the crash happens between (1) and (2), the request would return "Unkno=
wn ClOrdID" since the order=20
did not even manage to go to QF.

If the crash happens between (2) and (3) and the order is still in the QF=
 outgoing queue, the=20
OrderStatusRequest is appended to the queue after the order itself. So yo=
u have to wait some time=20
until the order and the request is being processed.

The QF callback toApp() can be helpful here: Every time QF puts a message=
 on the wire, this callback=20
is triggered. If the order is sent the first time, the PossDup flag is fa=
lse. If the order is resent=20
from the queue after a crash or network issue, the PossDup flag is set to=
 true and the field=20
OrigSendingTime is populated. By comparing SendingTime and OrigSendingTim=
e you can identify a time=20
lag between the first try and the current one. For some orders this may b=
e too large, and you may=20
decide to cancel sending this order again by throwing a DoNotSend excepti=
on.

>> 2) Isn't this a rather theoretical case?Won't QuickFix Recovery get th=
is for
>> you?
>=20
> I am of course a newbie, so I can't speak too authoritatively.  Maybe
> you're right.  It just seemed to me that since there's no way to both
> (1) send the message (and guarantee it got to the broker) and (2)
> update the database, all as part of the same guaranteed-atomic
> operation, then you can always construct a scenario where you lose if
> the power goes out at just the wrong moment.

If the crash happens before (2), the OrderStatusRequest will finally upda=
te your database. After=20
(1), there is no way in FIX to guarantee that the message actually arrive=
d at the other sides=20
application layer. It may possibly stall in resend processing forever, ie=
 if your network is very=20
bad. This can be caught by time-outs and should be handled manually.

> You're right that it's a theoretical case, but you know how it is ...
> if you don't deal with it, it'll come back and bite you at the worst
> possible moment.

A good place to look for and discuss such issues are the forums at www.fi=
xprotocol.org. Here you=20
will get more feedback on questions of this more general nature. Except m=
entioning the toApp()=20
callback, my answer should apply to all compliant FIX engines.

Cheers, J=F6rg

--=20
Joerg Thoennes
                                http://macd.com
Tel.: +49 (0)241 44597-24      Macdonald Associates GmbH
Fax : +49 (0)241 44597-10      Lothringer Str. 52, D-52070 Aachen