Thread: [Quickfix-developers] System clock issues

SourceForge Headquarters 225 Broadway Suite 1600 San Diego, CA 92101 +1 (858) 422-6466

I recently had two different scenarios occur which impacted my quickfix implementation, and I thought I'd describe them here in case anybody else runs into a similar situation, as well as to inquire if anybody has any suggestions on how best to try to handle either situation the next time it occurs.

By way of background, we've created an algorithmic high-frequency trading system that utilizes the latest out-of-the-box quickfix implementation.  The system is written in .NET 2.0.  We have a FIX "router" which is a gateway to 2 direct exchanges and 2 brokers (one US, one LatAm).  The router serves about 20 client implementations here in the office.  All clients are running on Windows XP.

Scenario #1 ("My life in IT would be much easier without Users"):

One of my traders starts screaming that his trading application has gone nuts.  Investigation shows that his system clock is not set to the proper date.  Further digging reveals that the trader in question was "bored, and just playing with the system clock.  Why would that affect our trading system?"  Diagnosis:  Quickfix discovered that the date had changed on the system clock and reset the daily session, setting incoming and outgoing sequence numbers back to 1.  This caused an immediate disconnect from the FIX router and subsequent reconnects failed due to sequence number mismatches.

>From my perspective, quickfix behaved exactly as it should in this scenario.  Amusingly enough, the same scenario re-occurred with a different trader a few weeks later.  I can't resist sharing the explanation from the trader:  "I was online reserving a car for my vacation and when it asked me to enter a date for pickup, I thought that was where I should enter it".  Priceless.

Scenario #2:  ("#^$&# Windows...")

Today one of our traders suddenly experienced a disconnect from our FIX router in the middle of a large sequence of trades.  The loss of connection was not a welcome event at that moment.  After much digging, we finally discovered that Windows had decided on its own to initiate its weekly clock sync to internet time.  Because of hardware vagaries, this workstation's system time had drifted a little over two minutes since its last sync.  Upon resetting the system time, quickfix determined that too much time had elapsed during a heartbeat request (send it at 11:06:58 and two seconds later get a response but the system time is now 11:09:23) so it initiated a disconnect on timeout.  Quickfix then automatically (and successfully) tried to log on as determined in my config file settings, but the market had slipped on the trades in progress so the experience was considered a big problem.

In both cases I believe that quickfix behaved exactly as expected.  I suppose the reason I'm writing all this is to see if perhaps anybody has any suggestions on 1) how to perhaps handle either of these scenarios more gracefully within my code, or 2) how to perhaps try to prevent similar situations from occuring from an IT management perspective (workstation lockdown, etc.).  If anybody has any thoughts on the topic I'd love to hear them.

(And before you suggest it, I asked:  Getting rid of all my users is not an option....)

Many thanks,

   John

Thread: [Quickfix-developers] System clock issues

quickfix-developers