Re: [Quickfix-developers] FileStore & real time

SourceForge Headquarters 1320 Columbia Street Suite 310 San Diego, CA 92101 +1 (858) 422-6466

Hi Oren,

Thank you for the explanation, but as I previously told I had already given
up to figure out improvements to FileStore, exactly because of some issues
you explained so well and now also because of others I wasn't yet aware of.
But, of course it does not mean that a genius somewhere can come with the
solution.

For higher scalability in Windows without the worst performance of regular
databases, my bet is BerkeleyDB and SQLite (or MemoryStore with
PersistMessages=N for configurations that do not require retransmission).

When development is finished, I will let you know about the test results.

Best regards,
Djalma

On 6/11/07, Oren Miller <or...@qu...> wrote:
>
> Ok, let me explain why the file store implementation is the way it
> is, and then we can talk about potential improvements.
>
> First off why is there a header and messages file?  Well the messages
> file is just a straight write of all messages without any delimiter.
> When a message comes in, QuickFIX knows it can just append the
> message to the end of the file.  It can then append the location of
> that message into the header file.  This allows us to have a dynamic
> message format the can be accessed randomly while allowing us to
> record messages of arbitrary size.
>
> This gives us a few things.  One, we no longer need to cache or do a
> linear search through a file when a resend request comes in.  Caching
> this data takes a lot of memory, it also creates large startup times
> because the engine would have to read in and parse every message on
> startup.  These were shortcomings of early versions which caused
> serious issues for implementations that used a lot of messages.
>
> The header and messages file can't really be integrated because they
> need to support messages of arbitrary lengths, and the header must be
> scanned in quickly on startup.  Since we do not know how many
> messages will be needed, we also do not know how much space to
> preallocate for the header portion, hence a separate file.
>
> Combining sessions into a single file would improve scalability on
> systems that have limitations for file constructors, but would cause
> other problems, one of which would be worse performance.  First off,
> it is now impossible to clear out data for a single session since
> they are all integrated.  It also makes things more complicated if
> you change your configuration file to add or remove sessions.  No
> longer can you just delete the files for a session or have the
> session clear out its own files.  The performance is going to be
> worse because you are now synchronizing access to a single resource
> among all of your sessions.  We have users that make use of hundreds
> of high frequency sessions simultaneously.  The contention to access
> this single resource would make performance unbearable.  There is
> also no reason I know of that writing to multiple files would perform
> worse than writing to a single file in multiple locations.
>
> Now the session/seqnums files might give us a little more to play
> with.  They could possibly be combined into the header file for
> instance.  The reason the seqnums file was separated is because it is
> a file that is most likely to be modified by a user, so we wanted to
> keep it separate to allow users to change it relatively safely.  In
> reality the data from those files could go into the header.
>
> Also, with the current implementation, the session file is accessed
> very infrequently (once every 24 hours at most).  So there is really
> no good reason to hold on to that file descriptor during normal
> operations.  We could just open it whenever it needs to be accessed,
> and then close it.
>
> --oren
>
> On Jun 11, 2007, at 10:18 AM, que...@bn... wrote:
>
> > QuickFIX Documentation: http://www.quickfixengine.org/quickfix/doc/
> > html/index.html
> > QuickFIX Support: http://www.quickfixengine.org/services.html
> >
> > Hi Djalma,
> >
> > Thanks a lot for your response.
> > I know that databases are slower than files. We are currently using
> > Oracle
> > 10G.
> > We already have quit good results (around 7.5 ms to store
> > msg&seqnum), but
> > we want to use files in order to have the best performances as
> > possible.
> > (I am using one process per fix session so I do not care scalablity)
> > That why I wonder if the file format can be changed in order to
> > obtain even
> > better performances.
> >
> >
> > Quentin.
> >
> >
> >
> >
> > Internet
> > drs...@gm...@lists.sourceforge.net - 06/11/2007 04:45 PM
> >
> >
> > Sent by:    qui...@li...
> >
> >
> >
> > To:    quickfix-developers
> >
> > cc:
> >
> >
> > Subject:    Re: [Quickfix-developers] FileStore & real time
> >
> > QuickFIX Documentation:
> > http://www.quickfixengine.org/quickfix/doc/html/index.html
> > QuickFIX Support: http://www.quickfixengine.org/services.html
> >
> > Hi Quentin,
> >
> > I confess that I also deslike the FileStore implementation.
> > Frankly, my main problem with it is not exactly performance, but
> > scalability, because my application runs out of file descriptors
> > very fast
> > and unfortunately there is no FD_LIMIT in Windows. In practice,
> > this means
> > that I cannot configure a large number of sessions in my
> > application and
> > this is a risk for my project.
> >
> > Concerning performance, I am not sure that it is so bad, especially if
> > compared with the databases stores. Normally, for the databases
> > like MySQL
> > and PostgreSQL it is required only 4 tables for all sessions,
> > however you
> > have to pay the overhead of the IPC (TCP) and SQL parsing. I rember
> > to have
> > seen in this forum benchmarks that reported that FielStore were in
> > avarage
> > 3x faster than the implemetations that use databases.
> >
> > Some months ago I started an attempt to understand the FileStore to
> > make
> > some improvements and I came to the following conclusion regarding the
> > created files:
> >
> > .session -> date and time of session creation (only 1 line)
> > .seqnums -> contains the incoming and outgoing sequence number (only 1
> > line)
> > .body -> a stream with all sent messages
> > .header -> works like an index with the offsets to recover the
> > message in
> > .body for retransmission
> >
> > Well, why not 4 files for all sessions instead of 4 files for each
> > session?
> > I think that it would solve the scalability problem, but
> > performance would
> > probably be slower, especially for multi-threaded applications that
> > would
> > need to rely in a single mutex to protect the integrity of the
> > files and it
> > would be certainly a bottleneck.
> >
> > Actually, I prefered to invest my effort in Store implementations
> > to use
> > embeded databases, like BerkeleyDB and SQLite (the development is
> > still
> > ongoing).
> >
> > Djalma
> >
> > On 6/11/07, que...@bn... <
> > que...@bn...> wrote:
> >   QuickFIX Documentation:
> >   http://www.quickfixengine.org/quickfix/doc/html/index.html
> >   QuickFIX Support: http://www.quickfixengine.org/services.html
> >
> >
> >
> >   Hi,
> >
> >   I think that the way used by quickFix to store fix persistent
> > datas into
> >   4
> >   files is not the best solution to have the best performances
> > possible.
> >
> >   Indeed for each fix messages send to the counterparty, QuickFix
> > has to
> >   save
> >   synchronously datas into 3 files (seqnum+header+messages).
> >
> >   Can't we have a better file format that allows to save only into
> > one file
> >   for each message ?
> >
> >   Please send me your thougths about this real time issue.
> >
> >
> >   Thanks,
> >
> >   Quentin.
> >
> >
> >
> >   This message and any attachments (the "message") is
> >   intended solely for the addressees and is confidential.
> >   If you receive this message in error, please delete it and
> >   immediately notify the sender. Any use not in accord with
> >   its purpose, any dissemination or disclosure, either whole
> >   or partial, is prohibited except formal approval. The internet
> >   can not guarantee the integrity of this message.
> >   BNP PARIBAS (and its subsidiaries) shall (will) not
> >   therefore be liable for the message if modified.
> >
> >                   ---------------------------------------------
> >
> >   Ce message et toutes les pieces jointes (ci-apres le
> >   "message") sont etablis a l'intention exclusive de ses
> >   destinataires et sont confidentiels. Si vous recevez ce
> >   message par erreur, merci de le detruire et d'en avertir
> >   immediatement l'expediteur. Toute utilisation de ce
> >   message non conforme a sa destination, toute diffusion
> >   ou toute publication, totale ou partielle, est interdite, sauf
> >   autorisation expresse. L'internet ne permettant pas
> >   d'assurer l'integrite de ce message, BNP PARIBAS (et ses
> >   filiales) decline(nt) toute responsabilite au titre de ce
> >   message, dans l'hypothese ou il aurait ete modifie.
> >
> >
> >
> > ----------------------------------------------------------------------
> > ---
> >   This SF.net email is sponsored by DB2 Express
> >   Download DB2 Express C - the FREE version of DB2 express and take
> >   control of your XML. No limits. Just data. Click to get it now.
> >   http://sourceforge.net/powerbar/db2/
> >   _______________________________________________
> >   Quickfix-developers mailing list
> >   Qui...@li...
> >   https://lists.sourceforge.net/lists/listinfo/quickfix-developers
> > ----------------------------------------------------------------------
> > ---
> > This SF.net email is sponsored by DB2 Express
> > Download DB2 Express C - the FREE version of DB2 express and take
> > control of your XML. No limits. Just data. Click to get it now.
> > http://sourceforge.net/powerbar/db2/
> > _______________________________________________
> > Quickfix-developers mailing list
> > Qui...@li...
> > https://lists.sourceforge.net/lists/listinfo/quickfix-developers
> >
> > ----------------------------------------------------------------------
> > ---
> > This SF.net email is sponsored by DB2 Express
> > Download DB2 Express C - the FREE version of DB2 express and take
> > control of your XML. No limits. Just data. Click to get it now.
> > http://sourceforge.net/powerbar/db2/
> > _______________________________________________
> > Quickfix-developers mailing list
> > Qui...@li...
> > https://lists.sourceforge.net/lists/listinfo/quickfix-developers
> >
>
>