Re: [Quickfix-developers] FileStore & real time

[Djalma R. d. S. F. <drs...@gm...>]

Hi Quentin,

I confess that I also deslike the FileStore implementation.
Frankly, my main problem with it is not exactly performance, but
scalability, because my application runs out of file descriptors very fast
and unfortunately there is no FD_LIMIT in Windows. In practice, this means
that I cannot configure a large number of sessions in my application and
this is a risk for my project.

Concerning performance, I am not sure that it is so bad, especially if
compared with the databases stores. Normally, for the databases like MySQL
and PostgreSQL it is required only 4 tables for all sessions, however you
have to pay the overhead of the IPC (TCP) and SQL parsing. I rember to have
seen in this forum benchmarks that reported that FielStore were in avarage
3x faster than the implemetations that use databases.

Some months ago I started an attempt to understand the FileStore to make
some improvements and I came to the following conclusion regarding the
created files:

.session -> date and time of session creation (only 1 line)
.seqnums -> contains the incoming and outgoing sequence number (only 1 line)
.body -> a stream with all sent messages
.header -> works like an index with the offsets to recover the message in
.body for retransmission

Well, why not 4 files for all sessions instead of 4 files for each session?
I think that it would solve the scalability problem, but performance would
probably be slower, especially for multi-threaded applications that would
need to rely in a single mutex to protect the integrity of the files and it
would be certainly a bottleneck.

Actually, I prefered to invest my effort in Store implementations to use
embeded databases, like BerkeleyDB and SQLite (the development is still
ongoing).

Djalma

On 6/11/07, que...@bn... <que...@bn...>
wrote:
>
> QuickFIX Documentation:
> http://www.quickfixengine.org/quickfix/doc/html/index.html
> QuickFIX Support: http://www.quickfixengine.org/services.html
>
>
>
> Hi,
>
> I think that the way used by quickFix to store fix persistent datas into 4
>
> files is not the best solution to have the best performances possible.
>
> Indeed for each fix messages send to the counterparty, QuickFix has to
> save
> synchronously datas into 3 files (seqnum+header+messages).
>
> Can't we have a better file format that allows to save only into one file
> for each message ?
>
> Please send me your thougths about this real time issue.
>
>
> Thanks,
>
> Quentin.
>
>
>
> This message and any attachments (the "message") is
> intended solely for the addressees and is confidential.
> If you receive this message in error, please delete it and
> immediately notify the sender. Any use not in accord with
> its purpose, any dissemination or disclosure, either whole
> or partial, is prohibited except formal approval. The internet
> can not guarantee the integrity of this message.
> BNP PARIBAS (and its subsidiaries) shall (will) not
> therefore be liable for the message if modified.
>
>                 ---------------------------------------------
>
> Ce message et toutes les pieces jointes (ci-apres le
> "message") sont etablis a l'intention exclusive de ses
> destinataires et sont confidentiels. Si vous recevez ce
> message par erreur, merci de le detruire et d'en avertir
> immediatement l'expediteur. Toute utilisation de ce
> message non conforme a sa destination, toute diffusion
> ou toute publication, totale ou partielle, est interdite, sauf
> autorisation expresse. L'internet ne permettant pas
> d'assurer l'integrite de ce message, BNP PARIBAS (et ses
> filiales) decline(nt) toute responsabilite au titre de ce
> message, dans l'hypothese ou il aurait ete modifie.
>
>
> -------------------------------------------------------------------------
> This SF.net email is sponsored by DB2 Express
> Download DB2 Express C - the FREE version of DB2 express and take
> control of your XML. No limits. Just data. Click to get it now.
> http://sourceforge.net/powerbar/db2/
> _______________________________________________
> Quickfix-developers mailing list
> Qui...@li...
> https://lists.sourceforge.net/lists/listinfo/quickfix-developers
>