Re: [Quickfix-developers] FileStore & real time

[Oren M. <or...@qu...>]

Ok, let me explain why the file store implementation is the way it  
is, and then we can talk about potential improvements.

First off why is there a header and messages file?  Well the messages  
file is just a straight write of all messages without any delimiter.   
When a message comes in, QuickFIX knows it can just append the  
message to the end of the file.  It can then append the location of  
that message into the header file.  This allows us to have a dynamic  
message format the can be accessed randomly while allowing us to  
record messages of arbitrary size.

This gives us a few things.  One, we no longer need to cache or do a  
linear search through a file when a resend request comes in.  Caching  
this data takes a lot of memory, it also creates large startup times  
because the engine would have to read in and parse every message on  
startup.  These were shortcomings of early versions which caused  
serious issues for implementations that used a lot of messages.

The header and messages file can't really be integrated because they  
need to support messages of arbitrary lengths, and the header must be  
scanned in quickly on startup.  Since we do not know how many  
messages will be needed, we also do not know how much space to  
preallocate for the header portion, hence a separate file.

Combining sessions into a single file would improve scalability on  
systems that have limitations for file constructors, but would cause  
other problems, one of which would be worse performance.  First off,  
it is now impossible to clear out data for a single session since  
they are all integrated.  It also makes things more complicated if  
you change your configuration file to add or remove sessions.  No  
longer can you just delete the files for a session or have the  
session clear out its own files.  The performance is going to be  
worse because you are now synchronizing access to a single resource  
among all of your sessions.  We have users that make use of hundreds  
of high frequency sessions simultaneously.  The contention to access  
this single resource would make performance unbearable.  There is  
also no reason I know of that writing to multiple files would perform  
worse than writing to a single file in multiple locations.

Now the session/seqnums files might give us a little more to play  
with.  They could possibly be combined into the header file for  
instance.  The reason the seqnums file was separated is because it is  
a file that is most likely to be modified by a user, so we wanted to  
keep it separate to allow users to change it relatively safely.  In  
reality the data from those files could go into the header.

Also, with the current implementation, the session file is accessed  
very infrequently (once every 24 hours at most).  So there is really  
no good reason to hold on to that file descriptor during normal  
operations.  We could just open it whenever it needs to be accessed,  
and then close it.

--oren

On Jun 11, 2007, at 10:18 AM, que...@bn... wrote:

> QuickFIX Documentation: http://www.quickfixengine.org/quickfix/doc/ 
> html/index.html
> QuickFIX Support: http://www.quickfixengine.org/services.html
>
> Hi Djalma,
>
> Thanks a lot for your response.
> I know that databases are slower than files. We are currently using  
> Oracle
> 10G.
> We already have quit good results (around 7.5 ms to store  
> msg&seqnum), but
> we want to use files in order to have the best performances as  
> possible.
> (I am using one process per fix session so I do not care scalablity)
> That why I wonder if the file format can be changed in order to  
> obtain even
> better performances.
>
>
> Quentin.
>
>
>
>
> Internet
> drs...@gm...@lists.sourceforge.net - 06/11/2007 04:45 PM
>
>
> Sent by:    qui...@li...
>
>
>
> To:    quickfix-developers
>
> cc:
>
>
> Subject:    Re: [Quickfix-developers] FileStore & real time
>
> QuickFIX Documentation:
> http://www.quickfixengine.org/quickfix/doc/html/index.html
> QuickFIX Support: http://www.quickfixengine.org/services.html
>
> Hi Quentin,
>
> I confess that I also deslike the FileStore implementation.
> Frankly, my main problem with it is not exactly performance, but
> scalability, because my application runs out of file descriptors  
> very fast
> and unfortunately there is no FD_LIMIT in Windows. In practice,  
> this means
> that I cannot configure a large number of sessions in my  
> application and
> this is a risk for my project.
>
> Concerning performance, I am not sure that it is so bad, especially if
> compared with the databases stores. Normally, for the databases  
> like MySQL
> and PostgreSQL it is required only 4 tables for all sessions,  
> however you
> have to pay the overhead of the IPC (TCP) and SQL parsing. I rember  
> to have
> seen in this forum benchmarks that reported that FielStore were in  
> avarage
> 3x faster than the implemetations that use databases.
>
> Some months ago I started an attempt to understand the FileStore to  
> make
> some improvements and I came to the following conclusion regarding the
> created files:
>
> .session -> date and time of session creation (only 1 line)
> .seqnums -> contains the incoming and outgoing sequence number (only 1
> line)
> .body -> a stream with all sent messages
> .header -> works like an index with the offsets to recover the  
> message in
> .body for retransmission
>
> Well, why not 4 files for all sessions instead of 4 files for each  
> session?
> I think that it would solve the scalability problem, but  
> performance would
> probably be slower, especially for multi-threaded applications that  
> would
> need to rely in a single mutex to protect the integrity of the  
> files and it
> would be certainly a bottleneck.
>
> Actually, I prefered to invest my effort in Store implementations  
> to use
> embeded databases, like BerkeleyDB and SQLite (the development is  
> still
> ongoing).
>
> Djalma
>
> On 6/11/07, que...@bn... <
> que...@bn...> wrote:
>   QuickFIX Documentation:
>   http://www.quickfixengine.org/quickfix/doc/html/index.html
>   QuickFIX Support: http://www.quickfixengine.org/services.html
>
>
>
>   Hi,
>
>   I think that the way used by quickFix to store fix persistent  
> datas into
>   4
>   files is not the best solution to have the best performances  
> possible.
>
>   Indeed for each fix messages send to the counterparty, QuickFix  
> has to
>   save
>   synchronously datas into 3 files (seqnum+header+messages).
>
>   Can't we have a better file format that allows to save only into  
> one file
>   for each message ?
>
>   Please send me your thougths about this real time issue.
>
>
>   Thanks,
>
>   Quentin.
>
>
>
>   This message and any attachments (the "message") is
>   intended solely for the addressees and is confidential.
>   If you receive this message in error, please delete it and
>   immediately notify the sender. Any use not in accord with
>   its purpose, any dissemination or disclosure, either whole
>   or partial, is prohibited except formal approval. The internet
>   can not guarantee the integrity of this message.
>   BNP PARIBAS (and its subsidiaries) shall (will) not
>   therefore be liable for the message if modified.
>
>                   ---------------------------------------------
>
>   Ce message et toutes les pieces jointes (ci-apres le
>   "message") sont etablis a l'intention exclusive de ses
>   destinataires et sont confidentiels. Si vous recevez ce
>   message par erreur, merci de le detruire et d'en avertir
>   immediatement l'expediteur. Toute utilisation de ce
>   message non conforme a sa destination, toute diffusion
>   ou toute publication, totale ou partielle, est interdite, sauf
>   autorisation expresse. L'internet ne permettant pas
>   d'assurer l'integrite de ce message, BNP PARIBAS (et ses
>   filiales) decline(nt) toute responsabilite au titre de ce
>   message, dans l'hypothese ou il aurait ete modifie.
>
>
>    
> ---------------------------------------------------------------------- 
> ---
>   This SF.net email is sponsored by DB2 Express
>   Download DB2 Express C - the FREE version of DB2 express and take
>   control of your XML. No limits. Just data. Click to get it now.
>   http://sourceforge.net/powerbar/db2/
>   _______________________________________________
>   Quickfix-developers mailing list
>   Qui...@li...
>   https://lists.sourceforge.net/lists/listinfo/quickfix-developers
> ---------------------------------------------------------------------- 
> ---
> This SF.net email is sponsored by DB2 Express
> Download DB2 Express C - the FREE version of DB2 express and take
> control of your XML. No limits. Just data. Click to get it now.
> http://sourceforge.net/powerbar/db2/
> _______________________________________________
> Quickfix-developers mailing list
> Qui...@li...
> https://lists.sourceforge.net/lists/listinfo/quickfix-developers
>
> ---------------------------------------------------------------------- 
> ---
> This SF.net email is sponsored by DB2 Express
> Download DB2 Express C - the FREE version of DB2 express and take
> control of your XML. No limits. Just data. Click to get it now.
> http://sourceforge.net/powerbar/db2/
> _______________________________________________
> Quickfix-developers mailing list
> Qui...@li...
> https://lists.sourceforge.net/lists/listinfo/quickfix-developers
>