Re: [Quickfix-developers] FileStore & real time

[<que...@bn...>]

Hi Djalma,

Thanks a lot for your response.
I know that databases are slower than files. We are currently using Oracle
10G.
We already have quit good results (around 7.5 ms to store msg&seqnum), but
we want to use files in order to have the best performances as possible.
(I am using one process per fix session so I do not care scalablity)
That why I wonder if the file format can be changed in order to obtain even
better performances.


Quentin.




Internet
drs...@gm...@lists.sourceforge.net - 06/11/2007 04:45 PM


Sent by:    qui...@li...



To:    quickfix-developers

cc:


Subject:    Re: [Quickfix-developers] FileStore & real time

QuickFIX Documentation:
http://www.quickfixengine.org/quickfix/doc/html/index.html
QuickFIX Support: http://www.quickfixengine.org/services.html

Hi Quentin,

I confess that I also deslike the FileStore implementation.
Frankly, my main problem with it is not exactly performance, but
scalability, because my application runs out of file descriptors very fast
and unfortunately there is no FD_LIMIT in Windows. In practice, this means
that I cannot configure a large number of sessions in my application and
this is a risk for my project.

Concerning performance, I am not sure that it is so bad, especially if
compared with the databases stores. Normally, for the databases like MySQL
and PostgreSQL it is required only 4 tables for all sessions, however you
have to pay the overhead of the IPC (TCP) and SQL parsing. I rember to have
seen in this forum benchmarks that reported that FielStore were in avarage
3x faster than the implemetations that use databases.

Some months ago I started an attempt to understand the FileStore to make
some improvements and I came to the following conclusion regarding the
created files:

.session -> date and time of session creation (only 1 line)
.seqnums -> contains the incoming and outgoing sequence number (only 1
line)
.body -> a stream with all sent messages
.header -> works like an index with the offsets to recover the message in
.body for retransmission

Well, why not 4 files for all sessions instead of 4 files for each session?
I think that it would solve the scalability problem, but performance would
probably be slower, especially for multi-threaded applications that would
need to rely in a single mutex to protect the integrity of the files and it
would be certainly a bottleneck.

Actually, I prefered to invest my effort in Store implementations to use
embeded databases, like BerkeleyDB and SQLite (the development is still
ongoing).

Djalma

On 6/11/07, que...@bn... <
que...@bn...> wrote:
  QuickFIX Documentation:
  http://www.quickfixengine.org/quickfix/doc/html/index.html
  QuickFIX Support: http://www.quickfixengine.org/services.html



  Hi,

  I think that the way used by quickFix to store fix persistent datas into
  4
  files is not the best solution to have the best performances possible.

  Indeed for each fix messages send to the counterparty, QuickFix has to
  save
  synchronously datas into 3 files (seqnum+header+messages).

  Can't we have a better file format that allows to save only into one file
  for each message ?

  Please send me your thougths about this real time issue.


  Thanks,

  Quentin.



  This message and any attachments (the "message") is
  intended solely for the addressees and is confidential.
  If you receive this message in error, please delete it and
  immediately notify the sender. Any use not in accord with
  its purpose, any dissemination or disclosure, either whole
  or partial, is prohibited except formal approval. The internet
  can not guarantee the integrity of this message.
  BNP PARIBAS (and its subsidiaries) shall (will) not
  therefore be liable for the message if modified.

                  ---------------------------------------------

  Ce message et toutes les pieces jointes (ci-apres le
  "message") sont etablis a l'intention exclusive de ses
  destinataires et sont confidentiels. Si vous recevez ce
  message par erreur, merci de le detruire et d'en avertir
  immediatement l'expediteur. Toute utilisation de ce
  message non conforme a sa destination, toute diffusion
  ou toute publication, totale ou partielle, est interdite, sauf
  autorisation expresse. L'internet ne permettant pas
  d'assurer l'integrite de ce message, BNP PARIBAS (et ses
  filiales) decline(nt) toute responsabilite au titre de ce
  message, dans l'hypothese ou il aurait ete modifie.


  -------------------------------------------------------------------------
  This SF.net email is sponsored by DB2 Express
  Download DB2 Express C - the FREE version of DB2 express and take
  control of your XML. No limits. Just data. Click to get it now.
  http://sourceforge.net/powerbar/db2/
  _______________________________________________
  Quickfix-developers mailing list
  Qui...@li...
  https://lists.sourceforge.net/lists/listinfo/quickfix-developers
-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
_______________________________________________
Quickfix-developers mailing list
Qui...@li...
https://lists.sourceforge.net/lists/listinfo/quickfix-developers