[Quickfixj-users] Server crash leaves QuickFIX/J engine (1.5.3) in corrupt state for FileStorePath=filestore

[<tom...@up...>]

Hello dear readers,

 

This is a situation that I've encountered twice now, at two different
clients of ours. Their application server with our FIX module running on it
based on the QuickFIX/J engine (1.5.3) crashes, and leaves the QuickFIX/J
engine in a corrupt state. The first time, the server crash was for a
virtualized server (some reason the hosting provider never told me) and the
second was due a UPS that did not signal the server to shut down properly
when the battery started running out. Our configuration has
FileStorePath=filestore in both instances. Our instances are buy side (i.e.
initiators).

 

On inspection of the QuickFIX/J state files, I'm seeing that they are
corrupted. I can fix this by removing all state files and restart our
module, then all is fine again. The first time happened about a year ago,
the second just earlier today. All state files were empty (.body, .header,
.senderseqnums, .targetseqnums), except for the .session file (see
attachment). You'll see it's filled with all ASCII NUL characters. A valid
one as I've noticed has something like <NUL><ETX><some number> as contents.

 

The exceptions I'm getting in my module's log file goes like:

 

quickfix.ConfigError: error during session initialization

             at
quickfix.mina.initiator.AbstractSocketInitiator.createSessions(AbstractSocke
tInitiator.java:169)

             at
quickfix.mina.initiator.AbstractSocketInitiator.createSessionInitiators(Abst
ractSocketInitiator.java:84)

             at quickfix.SocketInitiator.initialize(SocketInitiator.java:86)

             at quickfix.SocketInitiator.start(SocketInitiator.java:64)

             at <INTERNALS-STRIPPED>

             at <INTERNALS-STRIPPED>

             at <INTERNALS-STRIPPED>

             at java.lang.Thread.run(Unknown Source)

Caused by: java.lang.RuntimeException: java.io.IOException: invalid UTC
timestamp value: 

             at quickfix.FileStoreFactory.create(FileStoreFactory.java:80)

             at quickfix.Session.<init>(Session.java:467)

             at
quickfix.DefaultSessionFactory.create(DefaultSessionFactory.java:185)

             at
quickfix.mina.SessionConnector.createSession(SessionConnector.java:140)

             at
quickfix.mina.initiator.AbstractSocketInitiator.createSessions(AbstractSocke
tInitiator.java:163)

             ... 7 more

Caused by: java.io.IOException: invalid UTC timestamp value: 

             at
quickfix.FileStore.initializeSessionCreateTime(FileStore.java:137)

             at quickfix.FileStore.initializeCache(FileStore.java:123)

             at quickfix.FileStore.initialize(FileStore.java:116)

             at quickfix.FileStore.<init>(FileStore.java:101)

             at quickfix.FileStoreFactory.create(FileStoreFactory.java:78)

             ... 11 more

 

I remember this is the same exception I got the first time one of our
clients' server crashed.

 

Of course in an ideal world a server should never crash, but alas it does.
It would appear that the QuickFIX/J engine's state isn't properly flushed to
disk? Or maybe this happens because the engine's state cannot be written
atomically? I do not know, as I've never had the time to delve into
QuickFIX/J internals and see how this situation can arise.

 

Perhaps it has been reported before and fixed in a later version, I've not
had time to delve into that either. Perhaps this is something that simply
can't be fixed due to the way filesystems work? Or perhaps it can?

 

Besides this, I'm pretty happy with QuickFIX/J 1.5.3 and I wish to thank all
contributors to QuickFIX/J!

 

Kind regards,

--

Tom Tempelaere

Upsilon SA