[Quickfix-developers] More details -- RFC: transaction, speed upgrades

[John M. <jg...@jg...>]

******* please read IMPORTANT NOTE below if nothing else *******

I am beginning to formulate a plan.  Please let me know if I am going 
off the deep end:

FIX::MessageStore -- I plan to add a boolean isTransactional property.  
I will also add two virtual methods begin() and commit() -- no-ops by 
default.

FIX::MySQLStore / Factory -- the factory will pick up two additional 
settings: whether to use transactions and the database version number.  
If the version is > 4.1.1 then the prepared statements will be used.  
If we are to use transactions then the MessageStore is transactional.  
Auto-commit is still on (more on that below).

FIX::Responder -- Add pause() and resume() pure virtual methods.

FIX::Queue<T> -- add boolean signal property.  If property is marked 
false then push() does not signal.  If property is marked true then 
push() signals again and also signal() is called.

FIX::ThreadedSocketConnection -- Another FIX::Queue and thread are 
added for sending.  pause() is implemented to set send queue signal 
property to false.  resume() is implemented to set send queue signal 
property to true.  IMPORTANT NOTE: I'd like to change things so that 
ThreadedSocketConnection *always* uses a send queue and thread.  Anyone 
have a problem with that?  If the new thread encounters a write error 
then that will be communicated back to the send() method at next 
enqueue attempt as per my previous email.

FIX::SocketConnection -- pause() is implemented to create a std::queue 
and make send() enqueue instead of send.  resume() is implemented to 
send everything on the queue and destroy the queue.  Alternatively I 
can make the queue hang around all the time if that is what people want 
(slight performance increase).  I'm only doing this for people who want 
a single thread *and* transactions.

FIX::Session -- admin messages are not wrapped in transactions and 
Responder::pause() and resume() are not called.  (They are persisted 
since auto-commit is on in MySQL.)  Session::begin() will be 
implemented to call pause() on the responder and then begin() on the 
message store (by way of the session state).  Session::commit() will be 
implemented to call commit() on the message store followed by resume() 
on the responder.  fromApp() will be wrapped as follows:

if( session state -> message store -> is transactional )
	begin();
fromApp(); // incrNextSenderSeqNum() calls and set() calls will be in 
the transaction
m_state.incrNextTargetMsgSeqNum(); // removed from verify(), also 
called if admin type
if( session state -> message store -> is transactional )
	commit();

Note that begin and commit are called conditionally instead of just 
being no-ops in the non-transactional case for sake of clarity.  
Otherwise someone might think that a transaction always takes place.

Within fromApp() the user may want to use MySQL within the same 
transaction for different purposes.  If there is not currently a method 
for exposing that then I will add one.

That's it so far...  Comments?  Concerns?

-John

On Dec 17, 2003, at 5:05 PM, John Muehlhausen wrote:

>
> Hello,
>
> I am making some changes to QuickFIX and would like feedback on the 
> plans.  I'm not sure how best to get the changes into the source tree 
> but that is my goal since I think they'll be generally useful and I'd 
> like to not patch future versions...  :-)  I'm hoping to get some 
> consensus on the work to be done so that my chances of getting the 
> changes incorporated are greater... and to benefit from the combined 
> wisdom of the group.  I am relatively new to QuickFIX so please bear 
> with me!  I would need a volunteer to get the Java wrappers done since 
> we are C++ only...
>
> First: I'd like to modify MySQLStore such that it uses the "prepared 
> statements" in version 4.1.1+.  I'm hoping for a significant 
> performance improvement.
>
> Second: I'd like to provide the option to store the received messages 
> in the 'messages' table in case the developer needs to look at those 
> messages for some other reason.  This would be less error prone than 
> remembering to manually persist received messages.  For reasons that 
> will become clear soon, this would happen at dequeue from the 
> ThreadedSocket* objects.
>
> The third proposal needs some motivation:
>
> In some applications the validity of sending a FIX message depends on 
> the success of sending another FIX message.  It would therefore be 
> nice to have an atomic unit of work that involves the processing and 
> sending of an arbitrary number of messages.  One simple example: the 
> developer wants to send an execution report AND a drop copy somewhere 
> else.  If the execution report doesn't happen then the drop copy 
> should not either (and vice versa).
>
> Proposed solution (using MySQL 4.1.1+ with BDB or InnoDB tables):
>
> a) Add begin() and commit() methods to MySQLStore (and perhaps nil 
> versions in MessageStore) which returns handle(s) useful for 
> performing other MySQL statements in the same transaction.  The 
> generalized transaction looks like this:
>
> 	begin transaction
> 	receive message from queue, persisting changes to next incoming 
> seqnum (can't take place in socket reader thread-- does it now?)
> 	business logic resulting in message send(s) and potentially in other 
> persistence within the same transaction
> 	commit transaction
>
> The problem here is that we cannot actually send messages unless we 
> get past commit-- otherwise we might advertise something that didn't 
> actually happen!  More on this follows... Also, the user manually 
> calls begin() and commit() if the enclosed send(s) and other 
> persistence are not in response to a received message.
>
> b) static Utility::socket_send() is re-implemented to place bytes on a 
> send queue instead of actually sending on the socket-- and there is 
> YAWT (yet another worker thread) draining this queue onto the socket.  
> There are two reasons for this-- first, bandwidth utilization may 
> increase if business logic can run concurrently (our in-house engine 
> has this).  Second, this provides a mechanism for stalling the actual 
> send until post-commit (more in the next point).  If the worker thread 
> encounters a write error then the queue is marked bad/emptied and 
> socket_send() returns the error encountered by the write instead of 
> performing the next enqueue.  The enqueued messages are lost but will 
> be resent at next session startup (gap fill).  BTW, I am aware that 
> socket_send() is static-- the socket handle is used with a global map 
> to obtain the queue associated with the socket.  Hmmm... since there 
> is really no reason to do this with the non-threaded incarnation of 
> QuickFIX perhaps the right place for this is a re-implementation of 
> ThreadedSocketConnection::send().
>
> c) when beginning the transaction, a barrier is placed on the write 
> queues of each active session (whether connected or not).  The barrier 
> means "don't send past this point, wait here".  All such barriers are 
> removed just after the transaction is committed.  This delays the send 
> of each message generated from within the transaction until after the 
> message and seqnum increment have been persisted -- a guarantee that 
> they are available for servicing future resend requests.
>
> High performance, reliable routing-type applications stand to benefit 
> the most from these changes.  If a process/server dies in the middle 
> of a transaction then the incoming message being acted on is 
> effectively "un-received" and any messages produced are not sent.  Of 
> course changes would be made in such a way as to not impact current 
> QuickFIX users.
>
> It would be nice if "someone" wanted to do all of this for BDBStore 
> (imaginary Berkeley DB transactional implementation of MessageStore) 
> in the cases where MySQL is not practical-- like a desktop 
> application.
>
> Comments? Concerns?
>
> Thanks,
> John
>