Thread: Re: [Quickfix-developers] FileStore & real time

SourceForge Headquarters 1320 Columbia Street Suite 310 San Diego, CA 92101 +1 (858) 422-6466

Hello,

Just my 2 cents about the problem.

We have a huge file fragmentation for these files (NTFS) and I'm pretty 
sure the disk IO performance is really bad.
Reading of the header and body files in case of crash is just awful.
My applications were disconnected several times by the heartbeat timeout 
during the recovering and everything was started over after the 
reconnection.

BTW, in the FileLog file I think std::flush is redundant - std::endl 
flushes the stream:
 void onIncoming( const std::string& value )
  { m_messages << value << std::endl << std::flush; }

Regards,
		Alexey Zubko

>
> ------------------------------------------------------------------------
>
> Subject:
> Re: [Quickfix-developers] FileStore & real time
> From:
> Oren Miller <or...@qu...>
> Date:
> Mon, 11 Jun 2007 11:28:38 -0500
> To:
> que...@bn...
>
> To:
> que...@bn...
> CC:
> qui...@li...
>
>
> Ok, let me explain why the file store implementation is the way it is, 
> and then we can talk about potential improvements.
>
> First off why is there a header and messages file?  Well the messages 
> file is just a straight write of all messages without any delimiter.  
> When a message comes in, QuickFIX knows it can just append the message 
> to the end of the file.  It can then append the location of that 
> message into the header file.  This allows us to have a dynamic 
> message format the can be accessed randomly while allowing us to 
> record messages of arbitrary size.
>
> This gives us a few things.  One, we no longer need to cache or do a 
> linear search through a file when a resend request comes in.  Caching 
> this data takes a lot of memory, it also creates large startup times 
> because the engine would have to read in and parse every message on 
> startup.  These were shortcomings of early versions which caused 
> serious issues for implementations that used a lot of messages.
>
> The header and messages file can't really be integrated because they 
> need to support messages of arbitrary lengths, and the header must be 
> scanned in quickly on startup.  Since we do not know how many messages 
> will be needed, we also do not know how much space to preallocate for 
> the header portion, hence a separate file.
>
> Combining sessions into a single file would improve scalability on 
> systems that have limitations for file constructors, but would cause 
> other problems, one of which would be worse performance.  First off, 
> it is now impossible to clear out data for a single session since they 
> are all integrated.  It also makes things more complicated if you 
> change your configuration file to add or remove sessions.  No longer 
> can you just delete the files for a session or have the session clear 
> out its own files.  The performance is going to be worse because you 
> are now synchronizing access to a single resource among all of your 
> sessions.  We have users that make use of hundreds of high frequency 
> sessions simultaneously.  The contention to access this single 
> resource would make performance unbearable.  There is also no reason I 
> know of that writing to multiple files would perform worse than 
> writing to a single file in multiple locations.
>
> Now the session/seqnums files might give us a little more to play 
> with.  They could possibly be combined into the header file for 
> instance.  The reason the seqnums file was separated is because it is 
> a file that is most likely to be modified by a user, so we wanted to 
> keep it separate to allow users to change it relatively safely.  In 
> reality the data from those files could go into the header.
>
> Also, with the current implementation, the session file is accessed 
> very infrequently (once every 24 hours at most).  So there is really 
> no good reason to hold on to that file descriptor during normal 
> operations.  We could just open it whenever it needs to be accessed, 
> and then close it.
>
> --oren
>
> On Jun 11, 2007, at 10:18 AM, que...@bn... wrote:
>
>> QuickFIX Documentation: 
>> http://www.quickfixengine.org/quickfix/doc/html/index.html
>> QuickFIX Support: http://www.quickfixengine.org/services.html
>>
>> Hi Djalma,
>>
>> Thanks a lot for your response.
>> I know that databases are slower than files. We are currently using 
>> Oracle
>> 10G.
>> We already have quit good results (around 7.5 ms to store 
>> msg&seqnum), but
>> we want to use files in order to have the best performances as possible.
>> (I am using one process per fix session so I do not care scalablity)
>> That why I wonder if the file format can be changed in order to 
>> obtain even
>> better performances.
>>
>>
>> Quentin.
>>
>>
>>
>>
>> Internet
>> drs...@gm...@lists.sourceforge.net - 06/11/2007 04:45 PM
>>
>>
>> Sent by:    qui...@li...
>>
>>
>>
>> To:    quickfix-developers
>>
>> cc:
>>
>>
>> Subject:    Re: [Quickfix-developers] FileStore & real time
>>
>> QuickFIX Documentation:
>> http://www.quickfixengine.org/quickfix/doc/html/index.html
>> QuickFIX Support: http://www.quickfixengine.org/services.html
>>
>> Hi Quentin,
>>
>> I confess that I also deslike the FileStore implementation.
>> Frankly, my main problem with it is not exactly performance, but
>> scalability, because my application runs out of file descriptors very 
>> fast
>> and unfortunately there is no FD_LIMIT in Windows. In practice, this 
>> means
>> that I cannot configure a large number of sessions in my application and
>> this is a risk for my project.
>>
>> Concerning performance, I am not sure that it is so bad, especially if
>> compared with the databases stores. Normally, for the databases like 
>> MySQL
>> and PostgreSQL it is required only 4 tables for all sessions, however 
>> you
>> have to pay the overhead of the IPC (TCP) and SQL parsing. I rember 
>> to have
>> seen in this forum benchmarks that reported that FielStore were in 
>> avarage
>> 3x faster than the implemetations that use databases.
>>
>> Some months ago I started an attempt to understand the FileStore to make
>> some improvements and I came to the following conclusion regarding the
>> created files:
>>
>> .session -> date and time of session creation (only 1 line)
>> .seqnums -> contains the incoming and outgoing sequence number (only 1
>> line)
>> .body -> a stream with all sent messages
>> .header -> works like an index with the offsets to recover the 
>> message in
>> .body for retransmission
>>
>> Well, why not 4 files for all sessions instead of 4 files for each 
>> session?
>> I think that it would solve the scalability problem, but performance 
>> would
>> probably be slower, especially for multi-threaded applications that 
>> would
>> need to rely in a single mutex to protect the integrity of the files 
>> and it
>> would be certainly a bottleneck.
>>
>> Actually, I prefered to invest my effort in Store implementations to use
>> embeded databases, like BerkeleyDB and SQLite (the development is still
>> ongoing).
>>
>> Djalma
>>
>> On 6/11/07, que...@bn... <
>> que...@bn...> wrote:
>>   QuickFIX Documentation:
>>   http://www.quickfixengine.org/quickfix/doc/html/index.html
>>   QuickFIX Support: http://www.quickfixengine.org/services.html
>>
>>
>>
>>   Hi,
>>
>>   I think that the way used by quickFix to store fix persistent datas 
>> into
>>   4
>>   files is not the best solution to have the best performances possible.
>>
>>   Indeed for each fix messages send to the counterparty, QuickFix has to
>>   save
>>   synchronously datas into 3 files (seqnum+header+messages).
>>
>>   Can't we have a better file format that allows to save only into 
>> one file
>>   for each message ?
>>
>>   Please send me your thougths about this real time issue.
>>
>>
>>   Thanks,
>>
>>   Quentin.
>>
>>
>>
>>   This message and any attachments (the "message") is
>>   intended solely for the addressees and is confidential.
>>   If you receive this message in error, please delete it and
>>   immediately notify the sender. Any use not in accord with
>>   its purpose, any dissemination or disclosure, either whole
>>   or partial, is prohibited except formal approval. The internet
>>   can not guarantee the integrity of this message.
>>   BNP PARIBAS (and its subsidiaries) shall (will) not
>>   therefore be liable for the message if modified.
>>
>>                   ---------------------------------------------
>>
>>   Ce message et toutes les pieces jointes (ci-apres le
>>   "message") sont etablis a l'intention exclusive de ses
>>   destinataires et sont confidentiels. Si vous recevez ce
>>   message par erreur, merci de le detruire et d'en avertir
>>   immediatement l'expediteur. Toute utilisation de ce
>>   message non conforme a sa destination, toute diffusion
>>   ou toute publication, totale ou partielle, est interdite, sauf
>>   autorisation expresse. L'internet ne permettant pas
>>   d'assurer l'integrite de ce message, BNP PARIBAS (et ses
>>   filiales) decline(nt) toute responsabilite au titre de ce
>>   message, dans l'hypothese ou il aurait ete modifie.
>>
>>
>>   
>> ------------------------------------------------------------------------- 
>>
>>   This SF.net email is sponsored by DB2 Express
>>   Download DB2 Express C - the FREE version of DB2 express and take
>>   control of your XML. No limits. Just data. Click to get it now.
>>   http://sourceforge.net/powerbar/db2/
>>   _______________________________________________
>>   Quickfix-developers mailing list
>>   Qui...@li...
>>   https://lists.sourceforge.net/lists/listinfo/quickfix-developers
>> ------------------------------------------------------------------------- 
>>
>> This SF.net email is sponsored by DB2 Express
>> Download DB2 Express C - the FREE version of DB2 express and take
>> control of your XML. No limits. Just data. Click to get it now.
>> http://sourceforge.net/powerbar/db2/
>> _______________________________________________
>> Quickfix-developers mailing list
>> Qui...@li...
>> https://lists.sourceforge.net/lists/listinfo/quickfix-developers
>>
>> ------------------------------------------------------------------------- 
>>
>> This SF.net email is sponsored by DB2 Express
>> Download DB2 Express C - the FREE version of DB2 express and take
>> control of your XML. No limits. Just data. Click to get it now.
>> http://sourceforge.net/powerbar/db2/
>> _______________________________________________
>> Quickfix-developers mailing list
>> Qui...@li...
>> https://lists.sourceforge.net/lists/listinfo/quickfix-developers
>>
>
>
>