[Quickfix-developers] One more detail -- RFC: transaction, speed upgrades

[John M. <jg...@jg...>]

Oops-- when I wrote down the details I forgot one important thing: 
pause() and resume() must happen for *all* Responders that are used 
within the transaction, not just for the one from the Session that 
began the transaction.

There are two options here:
a) pause all Responders in all Sessions when the transaction begins
b) record that a transaction has begun and pause each affected session 
at the first operation on it from within the transaction

After commit() all paused queues must be resumed.  In (a) that means 
all sessions.  In (b) that means only paused sessions.

While (b) is a bit more complex it will allow unaffected send queues to 
drain while a transaction is taking place-- a good thing for 
performance.  I think that is the way I want to go.

Also, we will research whether the session threads (not the queue 
threads) can each have their own MessageStore environment with regard 
to transactions.  This would allow for concurrent transactions and 
means that pause() and resume() must increment and decrement a counter 
instead of just toggling a boolean (and the "transaction in progress" 
indicator is also a counter).  Whether this is supported will probably 
be MessageStore dependent.

-John

On Dec 18, 2003, at 4:04 PM, John Muehlhausen wrote:

>
> ******* please read IMPORTANT NOTE below if nothing else *******
>
> I am beginning to formulate a plan.  Please let me know if I am going 
> off the deep end:
>
> FIX::MessageStore -- I plan to add a boolean isTransactional property. 
>  I will also add two virtual methods begin() and commit() -- no-ops by 
> default.
>
> FIX::MySQLStore / Factory -- the factory will pick up two additional 
> settings: whether to use transactions and the database version number. 
>  If the version is > 4.1.1 then the prepared statements will be used.  
> If we are to use transactions then the MessageStore is transactional.  
> Auto-commit is still on (more on that below).
>
> FIX::Responder -- Add pause() and resume() pure virtual methods.
>
> FIX::Queue<T> -- add boolean signal property.  If property is marked 
> false then push() does not signal.  If property is marked true then 
> push() signals again and also signal() is called.
>
> FIX::ThreadedSocketConnection -- Another FIX::Queue and thread are 
> added for sending.  pause() is implemented to set send queue signal 
> property to false.  resume() is implemented to set send queue signal 
> property to true.  IMPORTANT NOTE: I'd like to change things so that 
> ThreadedSocketConnection *always* uses a send queue and thread.  
> Anyone have a problem with that?  If the new thread encounters a write 
> error then that will be communicated back to the send() method at next 
> enqueue attempt as per my previous email.
>
> FIX::SocketConnection -- pause() is implemented to create a std::queue 
> and make send() enqueue instead of send.  resume() is implemented to 
> send everything on the queue and destroy the queue.  Alternatively I 
> can make the queue hang around all the time if that is what people 
> want (slight performance increase).  I'm only doing this for people 
> who want a single thread *and* transactions.
>
> FIX::Session -- admin messages are not wrapped in transactions and 
> Responder::pause() and resume() are not called.  (They are persisted 
> since auto-commit is on in MySQL.)  Session::begin() will be 
> implemented to call pause() on the responder and then begin() on the 
> message store (by way of the session state).  Session::commit() will 
> be implemented to call commit() on the message store followed by 
> resume() on the responder.  fromApp() will be wrapped as follows:
>
> if( session state -> message store -> is transactional )
> 	begin();
> fromApp(); // incrNextSenderSeqNum() calls and set() calls will be in 
> the transaction
> m_state.incrNextTargetMsgSeqNum(); // removed from verify(), also 
> called if admin type
> if( session state -> message store -> is transactional )
> 	commit();
>
> Note that begin and commit are called conditionally instead of just 
> being no-ops in the non-transactional case for sake of clarity.  
> Otherwise someone might think that a transaction always takes place.
>
> Within fromApp() the user may want to use MySQL within the same 
> transaction for different purposes.  If there is not currently a 
> method for exposing that then I will add one.
>
> That's it so far...  Comments?  Concerns?
>
> -John
>
> On Dec 17, 2003, at 5:05 PM, John Muehlhausen wrote:
>
>>
>> Hello,
>>
>> I am making some changes to QuickFIX and would like feedback on the 
>> plans.  I'm not sure how best to get the changes into the source tree 
>> but that is my goal since I think they'll be generally useful and I'd 
>> like to not patch future versions...  :-)  I'm hoping to get some 
>> consensus on the work to be done so that my chances of getting the 
>> changes incorporated are greater... and to benefit from the combined 
>> wisdom of the group.  I am relatively new to QuickFIX so please bear 
>> with me!  I would need a volunteer to get the Java wrappers done 
>> since we are C++ only...
>>
>> First: I'd like to modify MySQLStore such that it uses the "prepared 
>> statements" in version 4.1.1+.  I'm hoping for a significant 
>> performance improvement.
>>
>> Second: I'd like to provide the option to store the received messages 
>> in the 'messages' table in case the developer needs to look at those 
>> messages for some other reason.  This would be less error prone than 
>> remembering to manually persist received messages.  For reasons that 
>> will become clear soon, this would happen at dequeue from the 
>> ThreadedSocket* objects.
>>
>> The third proposal needs some motivation:
>>
>> In some applications the validity of sending a FIX message depends on 
>> the success of sending another FIX message.  It would therefore be 
>> nice to have an atomic unit of work that involves the processing and 
>> sending of an arbitrary number of messages.  One simple example: the 
>> developer wants to send an execution report AND a drop copy somewhere 
>> else.  If the execution report doesn't happen then the drop copy 
>> should not either (and vice versa).
>>
>> Proposed solution (using MySQL 4.1.1+ with BDB or InnoDB tables):
>>
>> a) Add begin() and commit() methods to MySQLStore (and perhaps nil 
>> versions in MessageStore) which returns handle(s) useful for 
>> performing other MySQL statements in the same transaction.  The 
>> generalized transaction looks like this:
>>
>> 	begin transaction
>> 	receive message from queue, persisting changes to next incoming 
>> seqnum (can't take place in socket reader thread-- does it now?)
>> 	business logic resulting in message send(s) and potentially in other 
>> persistence within the same transaction
>> 	commit transaction
>>
>> The problem here is that we cannot actually send messages unless we 
>> get past commit-- otherwise we might advertise something that didn't 
>> actually happen!  More on this follows... Also, the user manually 
>> calls begin() and commit() if the enclosed send(s) and other 
>> persistence are not in response to a received message.
>>
>> b) static Utility::socket_send() is re-implemented to place bytes on 
>> a send queue instead of actually sending on the socket-- and there is 
>> YAWT (yet another worker thread) draining this queue onto the socket. 
>>  There are two reasons for this-- first, bandwidth utilization may 
>> increase if business logic can run concurrently (our in-house engine 
>> has this).  Second, this provides a mechanism for stalling the actual 
>> send until post-commit (more in the next point).  If the worker 
>> thread encounters a write error then the queue is marked bad/emptied 
>> and socket_send() returns the error encountered by the write instead 
>> of performing the next enqueue.  The enqueued messages are lost but 
>> will be resent at next session startup (gap fill).  BTW, I am aware 
>> that socket_send() is static-- the socket handle is used with a 
>> global map to obtain the queue associated with the socket.  Hmmm... 
>> since there is really no reason to do this with the non-threaded 
>> incarnation of QuickFIX perhaps the right place for this is a 
>> re-implementation of ThreadedSocketConnection::send().
>>
>> c) when beginning the transaction, a barrier is placed on the write 
>> queues of each active session (whether connected or not).  The 
>> barrier means "don't send past this point, wait here".  All such 
>> barriers are removed just after the transaction is committed.  This 
>> delays the send of each message generated from within the transaction 
>> until after the message and seqnum increment have been persisted -- a 
>> guarantee that they are available for servicing future resend 
>> requests.
>>
>> High performance, reliable routing-type applications stand to benefit 
>> the most from these changes.  If a process/server dies in the middle 
>> of a transaction then the incoming message being acted on is 
>> effectively "un-received" and any messages produced are not sent.  Of 
>> course changes would be made in such a way as to not impact current 
>> QuickFIX users.
>>
>> It would be nice if "someone" wanted to do all of this for BDBStore 
>> (imaginary Berkeley DB transactional implementation of MessageStore) 
>> in the cases where MySQL is not practical-- like a desktop 
>> application.
>>
>> Comments? Concerns?
>>
>> Thanks,
>> John
>>
>