quickfix-developers — QuickFIX developers forum

[Quickfix-developers] FileLogPath/FileStorePath in Session section

[Joe A. <JA...@ru...>]

When I put the FileLogPath and FileStorePath entries into the respective
session sections, I get an error that the FileLogPath is missing.
Seeing as this creates multiple files per session anyway, why can't this
be configured to go to different directories?

[Quickfix-developers] Orders never sent vs session vs startTime

[John G. <joh...@wa...>]

Hi,

Thanks to the list and Oren, I know that if I start writing orders to the
session before StartTime they will never be sent because QF relies on
the counterparty to ask for missing messages. So I took extra care about
the startTime parameter to be sure it would be set long before we actually
connect to the QF powered process to send the orders.

What happened yesterday was the following (in UTC times)

- startTime in config file: 04:30
- actual launch of process: 04:00 (yes I told them it's useless)
- immediate "Created Session" trace from the app which is just an 
std::cout from onCreate().
- but the counterparty was down. So QF goes and tries the 4
SocketConnectHostN it has available in config file and of course fails on
all of them and goes on cycling.
- at 05:44 we start sending orders to QF
- at 07:01 the counterparty comes up and logon succeeds
- all the previously sent orders have never been heard of again

I thought I understood that as long as I was sending orders to QF after 
StartTime, they would be sent when logon suceeded. What do I have wrong ? 
Is this related to the fail over (SocketConnectHost/Port) mechanism or my 
(mis)understanding of how QF works ? 

QF version is 1.11.1

We had talked some while ago about seperating a StartTime for the session 
and a "ConnectionTime" parameter, is this still something considered 
possible ? 

Thanks for any help/hints.
Sincerely,
JG

Re: [Quickfix-developers] FileStorePath And FileLogPath with Date for Multiple Sessions

[Oren M. <or...@qu...>]

Currently there is no way to do this automatically.  You would need  
to create a end of day script to copy the log files into dated file  
names.

--oren

On Jun 18, 2007, at 2:59 PM, Joe Asta wrote:

> QuickFIX Documentation: http://www.quickfixengine.org/quickfix/doc/ 
> html/index.html
> QuickFIX Support: http://www.quickfixengine.org/services.html
>
> Is there a way to configure QuickFix to include a date in the path  
> where the logs and store are created?
>
> I would like to see a new directory created for each day that  
> contains the logs and store files for each of the sessions.
>
> It might look something like this:
>
> 20070601
> ->Session1
> -->logs
> --->[files for logs]
> -->store
> --->[files for store]
> ->Session2
> -->logs
> --->[files for logs]
> -->store
> --->[files for store]
> 20070602
> ->Session1
> -->logs
> --->[files for logs]
> -->store
> --->[files for store]
> ->Session2
> -->logs
> --->[files for logs]
> -->store
> --->[files for store]
> ---------------------------------------------------------------------- 
> ---
> This SF.net email is sponsored by DB2 Express
> Download DB2 Express C - the FREE version of DB2 express and take
> control of your XML. No limits. Just data. Click to get it now.
> http://sourceforge.net/powerbar/db2/ 
> _______________________________________________
> Quickfix-developers mailing list
> Qui...@li...
> https://lists.sourceforge.net/lists/listinfo/quickfix-developers

Re: [Quickfix-developers] Problem on execution report callback

[quickfixuser <fw...@ro...>]

never mind. I figure it out. Some vendor side customized fields caused
rejection and does not allow the execution report passed at all.
-- 
View this message in context: http://www.nabble.com/Problem-on-execution-report-callback-tf3942661.html#a11184155
Sent from the QuickFIX - Dev mailing list archive at Nabble.com.

[Quickfix-developers] Problem on execution report callback

[quickfixuser <fw...@ro...>]

Hi,

I used.NET API of quickfix to send and receive orders. One test I did is to
send an order to exchange with dupe ids and try to catch rejection. 

My problem is that I can see the order get rejected from fix log, with 35=8.
However, the callback is not triggered at all. I have no problems to receive
execution reports in normal conditons (correct order id).

public override void onMessage( QuickFix42.ExecutionReport message,
QuickFix.SessionID session )

Has anyone had any problem like this?

Thanks

quickfixuser

-- 
View this message in context: http://www.nabble.com/Problem-on-execution-report-callback-tf3942661.html#a11183551
Sent from the QuickFIX - Dev mailing list archive at Nabble.com.

[Quickfix-developers] FileStorePath And FileLogPath with Date for Multiple Sessions

[Joe A. <JA...@ru...>]

Is there a way to configure QuickFix to include a date in the path where
the logs and store are created?
=20
I would like to see a new directory created for each day that contains
the logs and store files for each of the sessions. =20
=20
It might look something like this:
=20
20070601
->Session1
-->logs
--->[files for logs]
-->store
--->[files for store]
->Session2
-->logs
--->[files for logs]
-->store
--->[files for store]
20070602
->Session1
-->logs
--->[files for logs]
-->store
--->[files for store]
->Session2
-->logs
--->[files for logs]
-->store
--->[files for store]

Re: [Quickfix-developers] Required tag missing (102) on OrderCancelReject

[mr19 <mar...@ya...>]

Ok, ignore my response.  I now understand what you are saying and see that
the exception is being thrown when I access the undefined 102 field from my
code.   Thanks again for all your help.



mr19 wrote:
> 
> FIX41.xml shows the required field for tag 102 as "N".  Not sure I
> understand why I would need to get a valid (INT) value in this field if it
> is not required.  If it is not required and not included in a message what
> is the reason for throwing the exception?
> 
> TIA
> Marc
> 
> 
> Eyvind Ness wrote:
>> 
>> QuickFIX Documentation:
>> http://www.quickfixengine.org/quickfix/doc/html/index.html
>> QuickFIX Support: http://www.quickfixengine.org/services.html
>> 
>> 
>> Check the QF XML dictionary, FIX41.xml. If necessary, change the Required
>> field for tag 102 from "Y" to "N", and restart the application. If it's
>> "N"
>> already, make sure you get a valid (INT) value in this field.
>> 
>> On 6/14/07, mr19 <mar...@ya...> wrote:
>>>
>>> QuickFIX Documentation:
>>> http://www.quickfixengine.org/quickfix/doc/html/index.html
>>> QuickFIX Support: http://www.quickfixengine.org/services.html
>>>
>>>
>>> Hi -
>>>
>>> My broker is sending me what seems to be a valid OrderCancelReject
>>> message
>>> and QuickFix is responding with a Reject message "Required tag missing
>>> (102)".  Looking at the FIX 4.1 dictionary at b2bits shows field 102
>>> (CxlRejectReason) as NOT being required.  How can I prevent my app from
>>> autoresponding with the Reject message in this situation?
>>>
>>> TIA
>>> Marc Rossi
>>> --
>>> View this message in context:
>>> http://www.nabble.com/Required-tag-missing-%28102%29-on-OrderCancelReject-tf3924120.html#a11128179
>>> Sent from the QuickFIX - Dev mailing list archive at Nabble.com.
>>>
>>>
>>> -------------------------------------------------------------------------
>>> This SF.net email is sponsored by DB2 Express
>>> Download DB2 Express C - the FREE version of DB2 express and take
>>> control of your XML. No limits. Just data. Click to get it now.
>>> http://sourceforge.net/powerbar/db2/
>>> _______________________________________________
>>> Quickfix-developers mailing list
>>> Qui...@li...
>>> https://lists.sourceforge.net/lists/listinfo/quickfix-developers
>>>
>> 
>> -------------------------------------------------------------------------
>> This SF.net email is sponsored by DB2 Express
>> Download DB2 Express C - the FREE version of DB2 express and take
>> control of your XML. No limits. Just data. Click to get it now.
>> http://sourceforge.net/powerbar/db2/
>> _______________________________________________
>> Quickfix-developers mailing list
>> Qui...@li...
>> https://lists.sourceforge.net/lists/listinfo/quickfix-developers
>> 
> 
> 

-- 
View this message in context: http://www.nabble.com/Required-tag-missing-%28102%29-on-OrderCancelReject-tf3924120.html#a11141396
Sent from the QuickFIX - Dev mailing list archive at Nabble.com.

Re: [Quickfix-developers] Tag missing messages

[mr19 <mar...@ya...>]

Ok, ignore my last response.  I didn't realize the exception was being thro=
wn
when I tried to access field 102 in my app.  Thanks again for all the help.



Steve Bate-3 wrote:
>=20
> QuickFIX Documentation:
> http://www.quickfixengine.org/quickfix/doc/html/index.html
> QuickFIX Support: http://www.quickfixengine.org/services.html
>=20
>> "Parties required" is not a required tag, hence the counterparty=20
>> need not send it, which source file would shed some light into=20
>> how the engine goes .   About parsing the schema, meaning does=20
>> it do a getChild() and moreover how it handles it.
>=20
> Shankar,
>=20
> I'm not sure I fully understand your question, but you can look at
> the DataDictionary class to see how the schema is parsed. There's
> also some validation-related code in that class. A component
> is treated like a macro in that the component contents are
> saved in the dictionary and not the component itself. The=20
> quickfix.Message class has the parsing code if you want to=20
> look at that.
>=20
> Steve Bate
> Smart Trade Technologies
> Phone: +33 4 42 90 03 97
> http://www.smart-trade.net/
>=20
>=20
> -------------------------------------------------------
> This SF.net email is sponsored by: Splunk Inc. Do you grep through log
> files
> for problems?  Stop!  Download the new AJAX search engine that makes
> searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
> http://sel.as-us.falkag.net/sel?cmd=3Dlnk&kid=103432&bid#0486&dat=121642
> _______________________________________________
> Quickfix-developers mailing list
> Qui...@li...
> https://lists.sourceforge.net/lists/listinfo/quickfix-developers
>=20
>=20

--=20
View this message in context: http://www.nabble.com/Tag-missing-messages-tf=
1079242.html#a11141394
Sent from the QuickFIX - Dev mailing list archive at Nabble.com.

Re: [Quickfix-developers] Required tag missing (102) on OrderCancelReject

[mr19 <mar...@ya...>]

FIX41.xml shows the required field for tag 102 as "N".  Not sure I understand
why I would need to get a valid (INT) value in this field if it is not
required.  If it is not required and not included in a message what is the
reason for throwing the exception?

TIA
Marc


Eyvind Ness wrote:
> 
> QuickFIX Documentation:
> http://www.quickfixengine.org/quickfix/doc/html/index.html
> QuickFIX Support: http://www.quickfixengine.org/services.html
> 
> 
> Check the QF XML dictionary, FIX41.xml. If necessary, change the Required
> field for tag 102 from "Y" to "N", and restart the application. If it's
> "N"
> already, make sure you get a valid (INT) value in this field.
> 
> On 6/14/07, mr19 <mar...@ya...> wrote:
>>
>> QuickFIX Documentation:
>> http://www.quickfixengine.org/quickfix/doc/html/index.html
>> QuickFIX Support: http://www.quickfixengine.org/services.html
>>
>>
>> Hi -
>>
>> My broker is sending me what seems to be a valid OrderCancelReject
>> message
>> and QuickFix is responding with a Reject message "Required tag missing
>> (102)".  Looking at the FIX 4.1 dictionary at b2bits shows field 102
>> (CxlRejectReason) as NOT being required.  How can I prevent my app from
>> autoresponding with the Reject message in this situation?
>>
>> TIA
>> Marc Rossi
>> --
>> View this message in context:
>> http://www.nabble.com/Required-tag-missing-%28102%29-on-OrderCancelReject-tf3924120.html#a11128179
>> Sent from the QuickFIX - Dev mailing list archive at Nabble.com.
>>
>>
>> -------------------------------------------------------------------------
>> This SF.net email is sponsored by DB2 Express
>> Download DB2 Express C - the FREE version of DB2 express and take
>> control of your XML. No limits. Just data. Click to get it now.
>> http://sourceforge.net/powerbar/db2/
>> _______________________________________________
>> Quickfix-developers mailing list
>> Qui...@li...
>> https://lists.sourceforge.net/lists/listinfo/quickfix-developers
>>
> 
> -------------------------------------------------------------------------
> This SF.net email is sponsored by DB2 Express
> Download DB2 Express C - the FREE version of DB2 express and take
> control of your XML. No limits. Just data. Click to get it now.
> http://sourceforge.net/powerbar/db2/
> _______________________________________________
> Quickfix-developers mailing list
> Qui...@li...
> https://lists.sourceforge.net/lists/listinfo/quickfix-developers
> 

-- 
View this message in context: http://www.nabble.com/Required-tag-missing-%28102%29-on-OrderCancelReject-tf3924120.html#a11141249
Sent from the QuickFIX - Dev mailing list archive at Nabble.com.

Re: [Quickfix-developers] FileStore & real time

[Djalma R. d. S. F. <drs...@gm...>]

Like BerkeleyDB It is also part of the Oracle embedded database family.
Probably with slightly different purposes.
I would like to read a paper with comparisons, but unfortunately the
links I found where broken.

On 6/14/07, que...@bn...
<que...@bn...> wrote:
>
> Hi
>
> We are also thinking of using Oracle TimeTen.
> It seems to be a solution event better than BerkeleyDB or SQLite.
>
> Does anyone has tried TimeTen ?
>
> Quentin.
>
>
>
>
> Internet
> drs...@gm...@lists.sourceforge.net - 06/11/2007 07:59 PM
>
>
> Sent by:    qui...@li...
>
>
>
> To:    quickfix-developers
>
> cc:
>
>
> Subject:    Re: [Quickfix-developers] FileStore & real time
>
> QuickFIX Documentation:
> http://www.quickfixengine.org/quickfix/doc/html/index.html
> QuickFIX Support: http://www.quickfixengine.org/services.html
>
> Hi Oren,
>
> Thank you for the explanation, but as I previously told I had already given
> up to figure out improvements to FileStore, exactly because of some issues
> you explained so well and now also because of others I wasn't yet aware of.
> But, of course it does not mean that a genius somewhere can come with the
> solution.
>
> For higher scalability in Windows without the worst performance of regular
> databases, my bet is BerkeleyDB and SQLite (or MemoryStore with
> PersistMessages=N for configurations that do not require retransmission).
>
> When development is finished, I will let you know about the test results.
>
> Best regards,
> Djalma
>
> On 6/11/07, Oren Miller < or...@qu...> wrote:
>   Ok, let me explain why the file store implementation is the way it
>   is, and then we can talk about potential improvements.
>
>   First off why is there a header and messages file?  Well the messages
>   file is just a straight write of all messages without any delimiter.
>   When a message comes in, QuickFIX knows it can just append the
>   message to the end of the file.  It can then append the location of
>   that message into the header file.  This allows us to have a dynamic
>   message format the can be accessed randomly while allowing us to
>   record messages of arbitrary size.
>
>   This gives us a few things.  One, we no longer need to cache or do a
>   linear search through a file when a resend request comes in.  Caching
>   this data takes a lot of memory, it also creates large startup times
>   because the engine would have to read in and parse every message on
>   startup.  These were shortcomings of early versions which caused
>   serious issues for implementations that used a lot of messages.
>
>   The header and messages file can't really be integrated because they
>   need to support messages of arbitrary lengths, and the header must be
>   scanned in quickly on startup.  Since we do not know how many
>   messages will be needed, we also do not know how much space to
>   preallocate for the header portion, hence a separate file.
>
>   Combining sessions into a single file would improve scalability on
>   systems that have limitations for file constructors, but would cause
>   other problems, one of which would be worse performance.  First off,
>   it is now impossible to clear out data for a single session since
>   they are all integrated.  It also makes things more complicated if
>   you change your configuration file to add or remove sessions.  No
>   longer can you just delete the files for a session or have the
>   session clear out its own files.  The performance is going to be
>   worse because you are now synchronizing access to a single resource
>   among all of your sessions.  We have users that make use of hundreds
>   of high frequency sessions simultaneously.  The contention to access
>   this single resource would make performance unbearable.  There is
>   also no reason I know of that writing to multiple files would perform
>   worse than writing to a single file in multiple locations.
>
>   Now the session/seqnums files might give us a little more to play
>   with.  They could possibly be combined into the header file for
>   instance.  The reason the seqnums file was separated is because it is
>   a file that is most likely to be modified by a user, so we wanted to
>   keep it separate to allow users to change it relatively safely.  In
>   reality the data from those files could go into the header.
>
>   Also, with the current implementation, the session file is accessed
>   very infrequently (once every 24 hours at most).  So there is really
>   no good reason to hold on to that file descriptor during normal
>   operations.  We could just open it whenever it needs to be accessed,
>   and then close it.
>
>   --oren
>
>   On Jun 11, 2007, at 10:18 AM, que...@bn... wrote:
>
>   > QuickFIX Documentation: http://www.quickfixengine.org/quickfix/doc/
>   > html/index.html
>   > QuickFIX Support: http://www.quickfixengine.org/services.html
>   >
>   > Hi Djalma,
>   >
>   > Thanks a lot for your response.
>   > I know that databases are slower than files. We are currently using
>   > Oracle
>   > 10G.
>   > We already have quit good results (around 7.5 ms to store
>   > msg&seqnum), but
>   > we want to use files in order to have the best performances as
>   > possible.
>   > (I am using one process per fix session so I do not care scalablity)
>   > That why I wonder if the file format can be changed in order to
>   > obtain even
>   > better performances.
>   >
>   >
>   > Quentin.
>   >
>   >
>   >
>   >
>   > Internet
>   > drs...@gm... @lists.sourceforge.net - 06/11/2007 04:45 PM
>   >
>   >
>   > Sent by:    qui...@li...
>   >
>   >
>   >
>   > To:    quickfix-developers
>   >
>   > cc:
>   >
>   >
>   > Subject:    Re: [Quickfix-developers] FileStore & real time
>   >
>   > QuickFIX Documentation:
>   > http://www.quickfixengine.org/quickfix/doc/html/index.html
>   > QuickFIX Support: http://www.quickfixengine.org/services.html
>   >
>   > Hi Quentin,
>   >
>   > I confess that I also deslike the FileStore implementation.
>   > Frankly, my main problem with it is not exactly performance, but
>   > scalability, because my application runs out of file descriptors
>   > very fast
>   > and unfortunately there is no FD_LIMIT in Windows. In practice,
>   > this means
>   > that I cannot configure a large number of sessions in my
>   > application and
>   > this is a risk for my project.
>   >
>   > Concerning performance, I am not sure that it is so bad, especially if
>   > compared with the databases stores. Normally, for the databases
>   > like MySQL
>   > and PostgreSQL it is required only 4 tables for all sessions,
>   > however you
>   > have to pay the overhead of the IPC (TCP) and SQL parsing. I rember
>   > to have
>   > seen in this forum benchmarks that reported that FielStore were in
>   > avarage
>   > 3x faster than the implemetations that use databases.
>   >
>   > Some months ago I started an attempt to understand the FileStore to
>   > make
>   > some improvements and I came to the following conclusion regarding the
>   > created files:
>   >
>   > .session -> date and time of session creation (only 1 line)
>   > .seqnums -> contains the incoming and outgoing sequence number (only 1
>   > line)
>   > .body -> a stream with all sent messages
>   > .header -> works like an index with the offsets to recover the
>   > message in
>   > .body for retransmission
>   >
>   > Well, why not 4 files for all sessions instead of 4 files for each
>   > session?
>   > I think that it would solve the scalability problem, but
>   > performance would
>   > probably be slower, especially for multi-threaded applications that
>   > would
>   > need to rely in a single mutex to protect the integrity of the
>   > files and it
>   > would be certainly a bottleneck.
>   >
>   > Actually, I prefered to invest my effort in Store implementations
>   > to use
>   > embeded databases, like BerkeleyDB and SQLite (the development is
>   > still
>   > ongoing).
>   >
>   > Djalma
>   >
>   > On 6/11/07, que...@bn... <
>   > que...@bn...> wrote:
>   >   QuickFIX Documentation:
>   >   http://www.quickfixengine.org/quickfix/doc/html/index.html
>   >   QuickFIX Support: http://www.quickfixengine.org/services.html
>   >
>   >
>   >
>   >   Hi,
>   >
>   >   I think that the way used by quickFix to store fix persistent
>   > datas into
>   >   4
>   >   files is not the best solution to have the best performances
>   > possible.
>   >
>   >   Indeed for each fix messages send to the counterparty, QuickFix
>   > has to
>   >   save
>   >   synchronously datas into 3 files (seqnum+header+messages).
>   >
>   >   Can't we have a better file format that allows to save only into
>   > one file
>   >   for each message ?
>   >
>   >   Please send me your thougths about this real time issue.
>   >
>   >
>   >   Thanks,
>   >
>   >   Quentin.
>   >
>   >
>   >
>   >   This message and any attachments (the "message") is
>   >   intended solely for the addressees and is confidential.
>   >   If you receive this message in error, please delete it and
>   >   immediately notify the sender. Any use not in accord with
>   >   its purpose, any dissemination or disclosure, either whole
>   >   or partial, is prohibited except formal approval. The internet
>   >   can not guarantee the integrity of this message.
>   >   BNP PARIBAS (and its subsidiaries) shall (will) not
>   >   therefore be liable for the message if modified.
>   >
>   >                   ---------------------------------------------
>   >
>   >   Ce message et toutes les pieces jointes (ci-apres le
>   >   "message") sont etablis a l'intention exclusive de ses
>   >   destinataires et sont confidentiels. Si vous recevez ce
>   >   message par erreur, merci de le detruire et d'en avertir
>   >   immediatement l'expediteur. Toute utilisation de ce
>   >   message non conforme a sa destination, toute diffusion
>   >   ou toute publication, totale ou partielle, est interdite, sauf
>   >   autorisation expresse. L'internet ne permettant pas
>   >   d'assurer l'integrite de ce message, BNP PARIBAS (et ses
>   >   filiales) decline(nt) toute responsabilite au titre de ce
>   >   message, dans l'hypothese ou il aurait ete modifie.
>   >
>   >
>   >
>   > ----------------------------------------------------------------------
>   > ---
>   >   This SF.net email is sponsored by DB2 Express
>   >   Download DB2 Express C - the FREE version of DB2 express and take
>   >   control of your XML. No limits. Just data. Click to get it now.
>   >   http://sourceforge.net/powerbar/db2/
>   >   _______________________________________________
>   >   Quickfix-developers mailing list
>   >   Qui...@li...
>   >   https://lists.sourceforge.net/lists/listinfo/quickfix-developers
>   > ----------------------------------------------------------------------
>   > ---
>   > This SF.net email is sponsored by DB2 Express
>   > Download DB2 Express C - the FREE version of DB2 express and take
>   > control of your XML. No limits. Just data. Click to get it now.
>   > http://sourceforge.net/powerbar/db2/
>   > _______________________________________________
>   > Quickfix-developers mailing list
>   > Qui...@li...
>   > https://lists.sourceforge.net/lists/listinfo/quickfix-developers
>   >
>   > ----------------------------------------------------------------------
>   > ---
>   > This SF.net email is sponsored by DB2 Express
>   > Download DB2 Express C - the FREE version of DB2 express and take
>   > control of your XML. No limits. Just data. Click to get it now.
>   > http://sourceforge.net/powerbar/db2/
>   > _______________________________________________
>   > Quickfix-developers mailing list
>   > Qui...@li...
>   > https://lists.sourceforge.net/lists/listinfo/quickfix-developers
>   >
>
> -------------------------------------------------------------------------
> This SF.net email is sponsored by DB2 Express
> Download DB2 Express C - the FREE version of DB2 express and take
> control of your XML. No limits. Just data. Click to get it now.
> http://sourceforge.net/powerbar/db2/
> _______________________________________________
> Quickfix-developers mailing list
> Qui...@li...
> https://lists.sourceforge.net/lists/listinfo/quickfix-developers
>

[Quickfix-developers] Dynamic Multisessions

[Gary G. <gar...@gm...>]

Hi,

I've created at QuickFix acceptor application written using .NET 2.0 which
at the moment accepts a single session connection. According to the Quickfix
documentation, all the sessions are held in the associated configuration
file. If you wanted to create sessions dynamically (when any initiator
requests a log on) how to you go about this?

Cheers

Gary

[Quickfix-developers] Quickfix 1.12.4 crashes with daily sessions depending on .session file value

[Abel M. <abe...@ho...>]

Hi everybody,

I found a weird behaviour in quickfix 1.12.4 (under RedHatEnterprise3). I 
have a session configurated in this way

[DEFAULT]
ConnectionType=acceptor
ReconnectInterval=20
SenderCompID=EXAMF
SocketAcceptPort=6500
SocketReuseAddress=Y
CheckLatency=N
MillisecondsInTimeStamp=Y

FileStorePath=./var/logs/stores
FileLogPath=./var/logs

[SESSION]
# inherit ConnectionType, ReconnectInterval and SenderCompID from default
TargetCompID=USER1
StartTime=06:30:00
EndTime=18:30:00
BeginString=FIX.4.2
DataDictionary=./etc/dictionaries/FIX42VT.xml

I found that sometimes, when my app start it crashs with a core creating the 
sessions

#4  0x0090f3f5 in __cxa_call_unexpected () from /usr/lib/libstdc++.so.5
#5  0x00604928 in Session (this=0x8cda540, application=@0x0, 
messageStoreFactory=@0x0, sessionID=@0x8cdde38, dataDictionary=Internal: 
global symbol `DataDictionary' found in DataDictionary.cpp psymtab but not 
in symtab.
DataDictionary may be an inlined function, or may be a template function
(if a template, try specifying an instantiation: DataDictionary<type>).
) at Mutex.h:76
#6  0x0062e067 in FIX::SessionFactory::create (this=0x11a7550, 
sessionID=@0x8ca69c8, settings=@0x8cc7574) at Field.h:308
#7  0x0063fe23 in FIX::Acceptor::initialize (this=0x8cc7870) at 
stl_tree.h:199
#8  0x0063f8a0 in FIX::Acceptor::Acceptor$base () at stl_function.h:197
#9  0x006550fe in FIX::ThreadedSocketAcceptor::ThreadedSocketAcceptor () at 
new:89

After some days fighting with this error, I discover that the creation of 
fix sessions crash depends on the log files of quickfix. Specifically, it 
depends on session file. If in this file I have the present date, or it 
isn't exist, it works fine. But, if it has some previous date, quickfix 
crashes with a core.
With a previous version (1.10.x) this doesn't happen never, it always works 
fine.

So, do someone know if this is a bug of 1.12.4 version? I saw that this core 
is an unexpected exception while creating session (in session constructor). 
In getValue function from IntField, the exception IncorrectDataFormat is 
thrown, but in Session constructor this exception is not waited, so it 
crashes.
Someone has some clue or has heard about it before?

Thanks,
Abel Monroy

_________________________________________________________________
Acepta el reto MSN Premium: Correos más divertidos con fotos y textos 
increíbles en MSN Premium. Descárgalo y pruébalo 2 meses gratis. 
http://join.msn.com?XAPID=1697&DI=1055&HL=Footer_mailsenviados_correosmasdivertidos

Re: [Quickfix-developers] Required tag missing (102) on OrderCancelReject

[Eyvind N. <eyv...@gm...>]

Check the QF XML dictionary, FIX41.xml. If necessary, change the Required
field for tag 102 from "Y" to "N", and restart the application. If it's "N"
already, make sure you get a valid (INT) value in this field.

On 6/14/07, mr19 <mar...@ya...> wrote:
>
> QuickFIX Documentation:
> http://www.quickfixengine.org/quickfix/doc/html/index.html
> QuickFIX Support: http://www.quickfixengine.org/services.html
>
>
> Hi -
>
> My broker is sending me what seems to be a valid OrderCancelReject message
> and QuickFix is responding with a Reject message "Required tag missing
> (102)".  Looking at the FIX 4.1 dictionary at b2bits shows field 102
> (CxlRejectReason) as NOT being required.  How can I prevent my app from
> autoresponding with the Reject message in this situation?
>
> TIA
> Marc Rossi
> --
> View this message in context:
> http://www.nabble.com/Required-tag-missing-%28102%29-on-OrderCancelReject-tf3924120.html#a11128179
> Sent from the QuickFIX - Dev mailing list archive at Nabble.com.
>
>
> -------------------------------------------------------------------------
> This SF.net email is sponsored by DB2 Express
> Download DB2 Express C - the FREE version of DB2 express and take
> control of your XML. No limits. Just data. Click to get it now.
> http://sourceforge.net/powerbar/db2/
> _______________________________________________
> Quickfix-developers mailing list
> Qui...@li...
> https://lists.sourceforge.net/lists/listinfo/quickfix-developers
>

[Quickfix-developers] Required tag missing (102) on OrderCancelReject

[mr19 <mar...@ya...>]

Hi -

My broker is sending me what seems to be a valid OrderCancelReject message
and QuickFix is responding with a Reject message "Required tag missing
(102)".  Looking at the FIX 4.1 dictionary at b2bits shows field 102
(CxlRejectReason) as NOT being required.  How can I prevent my app from
autoresponding with the Reject message in this situation?

TIA
Marc Rossi
-- 
View this message in context: http://www.nabble.com/Required-tag-missing-%28102%29-on-OrderCancelReject-tf3924120.html#a11128179
Sent from the QuickFIX - Dev mailing list archive at Nabble.com.

Re: [Quickfix-developers] FileStore & real time

[<que...@bn...>]

Hi

We are also thinking of using Oracle TimeTen.
It seems to be a solution event better than BerkeleyDB or SQLite.

Does anyone has tried TimeTen ?

Quentin.




Internet
drs...@gm...@lists.sourceforge.net - 06/11/2007 07:59 PM


Sent by:    qui...@li...



To:    quickfix-developers

cc:


Subject:    Re: [Quickfix-developers] FileStore & real time

QuickFIX Documentation:
http://www.quickfixengine.org/quickfix/doc/html/index.html
QuickFIX Support: http://www.quickfixengine.org/services.html

Hi Oren,

Thank you for the explanation, but as I previously told I had already given
up to figure out improvements to FileStore, exactly because of some issues
you explained so well and now also because of others I wasn't yet aware of.
But, of course it does not mean that a genius somewhere can come with the
solution.

For higher scalability in Windows without the worst performance of regular
databases, my bet is BerkeleyDB and SQLite (or MemoryStore with
PersistMessages=N for configurations that do not require retransmission).

When development is finished, I will let you know about the test results.

Best regards,
Djalma

On 6/11/07, Oren Miller < or...@qu...> wrote:
  Ok, let me explain why the file store implementation is the way it
  is, and then we can talk about potential improvements.

  First off why is there a header and messages file?  Well the messages
  file is just a straight write of all messages without any delimiter.
  When a message comes in, QuickFIX knows it can just append the
  message to the end of the file.  It can then append the location of
  that message into the header file.  This allows us to have a dynamic
  message format the can be accessed randomly while allowing us to
  record messages of arbitrary size.

  This gives us a few things.  One, we no longer need to cache or do a
  linear search through a file when a resend request comes in.  Caching
  this data takes a lot of memory, it also creates large startup times
  because the engine would have to read in and parse every message on
  startup.  These were shortcomings of early versions which caused
  serious issues for implementations that used a lot of messages.

  The header and messages file can't really be integrated because they
  need to support messages of arbitrary lengths, and the header must be
  scanned in quickly on startup.  Since we do not know how many
  messages will be needed, we also do not know how much space to
  preallocate for the header portion, hence a separate file.

  Combining sessions into a single file would improve scalability on
  systems that have limitations for file constructors, but would cause
  other problems, one of which would be worse performance.  First off,
  it is now impossible to clear out data for a single session since
  they are all integrated.  It also makes things more complicated if
  you change your configuration file to add or remove sessions.  No
  longer can you just delete the files for a session or have the
  session clear out its own files.  The performance is going to be
  worse because you are now synchronizing access to a single resource
  among all of your sessions.  We have users that make use of hundreds
  of high frequency sessions simultaneously.  The contention to access
  this single resource would make performance unbearable.  There is
  also no reason I know of that writing to multiple files would perform
  worse than writing to a single file in multiple locations.

  Now the session/seqnums files might give us a little more to play
  with.  They could possibly be combined into the header file for
  instance.  The reason the seqnums file was separated is because it is
  a file that is most likely to be modified by a user, so we wanted to
  keep it separate to allow users to change it relatively safely.  In
  reality the data from those files could go into the header.

  Also, with the current implementation, the session file is accessed
  very infrequently (once every 24 hours at most).  So there is really
  no good reason to hold on to that file descriptor during normal
  operations.  We could just open it whenever it needs to be accessed,
  and then close it.

  --oren

  On Jun 11, 2007, at 10:18 AM, que...@bn... wrote:

  > QuickFIX Documentation: http://www.quickfixengine.org/quickfix/doc/
  > html/index.html
  > QuickFIX Support: http://www.quickfixengine.org/services.html
  >
  > Hi Djalma,
  >
  > Thanks a lot for your response.
  > I know that databases are slower than files. We are currently using
  > Oracle
  > 10G.
  > We already have quit good results (around 7.5 ms to store
  > msg&seqnum), but
  > we want to use files in order to have the best performances as
  > possible.
  > (I am using one process per fix session so I do not care scalablity)
  > That why I wonder if the file format can be changed in order to
  > obtain even
  > better performances.
  >
  >
  > Quentin.
  >
  >
  >
  >
  > Internet
  > drs...@gm... @lists.sourceforge.net - 06/11/2007 04:45 PM
  >
  >
  > Sent by:    qui...@li...
  >
  >
  >
  > To:    quickfix-developers
  >
  > cc:
  >
  >
  > Subject:    Re: [Quickfix-developers] FileStore & real time
  >
  > QuickFIX Documentation:
  > http://www.quickfixengine.org/quickfix/doc/html/index.html
  > QuickFIX Support: http://www.quickfixengine.org/services.html
  >
  > Hi Quentin,
  >
  > I confess that I also deslike the FileStore implementation.
  > Frankly, my main problem with it is not exactly performance, but
  > scalability, because my application runs out of file descriptors
  > very fast
  > and unfortunately there is no FD_LIMIT in Windows. In practice,
  > this means
  > that I cannot configure a large number of sessions in my
  > application and
  > this is a risk for my project.
  >
  > Concerning performance, I am not sure that it is so bad, especially if
  > compared with the databases stores. Normally, for the databases
  > like MySQL
  > and PostgreSQL it is required only 4 tables for all sessions,
  > however you
  > have to pay the overhead of the IPC (TCP) and SQL parsing. I rember
  > to have
  > seen in this forum benchmarks that reported that FielStore were in
  > avarage
  > 3x faster than the implemetations that use databases.
  >
  > Some months ago I started an attempt to understand the FileStore to
  > make
  > some improvements and I came to the following conclusion regarding the
  > created files:
  >
  > .session -> date and time of session creation (only 1 line)
  > .seqnums -> contains the incoming and outgoing sequence number (only 1
  > line)
  > .body -> a stream with all sent messages
  > .header -> works like an index with the offsets to recover the
  > message in
  > .body for retransmission
  >
  > Well, why not 4 files for all sessions instead of 4 files for each
  > session?
  > I think that it would solve the scalability problem, but
  > performance would
  > probably be slower, especially for multi-threaded applications that
  > would
  > need to rely in a single mutex to protect the integrity of the
  > files and it
  > would be certainly a bottleneck.
  >
  > Actually, I prefered to invest my effort in Store implementations
  > to use
  > embeded databases, like BerkeleyDB and SQLite (the development is
  > still
  > ongoing).
  >
  > Djalma
  >
  > On 6/11/07, que...@bn... <
  > que...@bn...> wrote:
  >   QuickFIX Documentation:
  >   http://www.quickfixengine.org/quickfix/doc/html/index.html
  >   QuickFIX Support: http://www.quickfixengine.org/services.html
  >
  >
  >
  >   Hi,
  >
  >   I think that the way used by quickFix to store fix persistent
  > datas into
  >   4
  >   files is not the best solution to have the best performances
  > possible.
  >
  >   Indeed for each fix messages send to the counterparty, QuickFix
  > has to
  >   save
  >   synchronously datas into 3 files (seqnum+header+messages).
  >
  >   Can't we have a better file format that allows to save only into
  > one file
  >   for each message ?
  >
  >   Please send me your thougths about this real time issue.
  >
  >
  >   Thanks,
  >
  >   Quentin.
  >
  >
  >
  >   This message and any attachments (the "message") is
  >   intended solely for the addressees and is confidential.
  >   If you receive this message in error, please delete it and
  >   immediately notify the sender. Any use not in accord with
  >   its purpose, any dissemination or disclosure, either whole
  >   or partial, is prohibited except formal approval. The internet
  >   can not guarantee the integrity of this message.
  >   BNP PARIBAS (and its subsidiaries) shall (will) not
  >   therefore be liable for the message if modified.
  >
  >                   ---------------------------------------------
  >
  >   Ce message et toutes les pieces jointes (ci-apres le
  >   "message") sont etablis a l'intention exclusive de ses
  >   destinataires et sont confidentiels. Si vous recevez ce
  >   message par erreur, merci de le detruire et d'en avertir
  >   immediatement l'expediteur. Toute utilisation de ce
  >   message non conforme a sa destination, toute diffusion
  >   ou toute publication, totale ou partielle, est interdite, sauf
  >   autorisation expresse. L'internet ne permettant pas
  >   d'assurer l'integrite de ce message, BNP PARIBAS (et ses
  >   filiales) decline(nt) toute responsabilite au titre de ce
  >   message, dans l'hypothese ou il aurait ete modifie.
  >
  >
  >
  > ----------------------------------------------------------------------
  > ---
  >   This SF.net email is sponsored by DB2 Express
  >   Download DB2 Express C - the FREE version of DB2 express and take
  >   control of your XML. No limits. Just data. Click to get it now.
  >   http://sourceforge.net/powerbar/db2/
  >   _______________________________________________
  >   Quickfix-developers mailing list
  >   Qui...@li...
  >   https://lists.sourceforge.net/lists/listinfo/quickfix-developers
  > ----------------------------------------------------------------------
  > ---
  > This SF.net email is sponsored by DB2 Express
  > Download DB2 Express C - the FREE version of DB2 express and take
  > control of your XML. No limits. Just data. Click to get it now.
  > http://sourceforge.net/powerbar/db2/
  > _______________________________________________
  > Quickfix-developers mailing list
  > Qui...@li...
  > https://lists.sourceforge.net/lists/listinfo/quickfix-developers
  >
  > ----------------------------------------------------------------------
  > ---
  > This SF.net email is sponsored by DB2 Express
  > Download DB2 Express C - the FREE version of DB2 express and take
  > control of your XML. No limits. Just data. Click to get it now.
  > http://sourceforge.net/powerbar/db2/
  > _______________________________________________
  > Quickfix-developers mailing list
  > Qui...@li...
  > https://lists.sourceforge.net/lists/listinfo/quickfix-developers
  >

-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
_______________________________________________
Quickfix-developers mailing list
Qui...@li...
https://lists.sourceforge.net/lists/listinfo/quickfix-developers

Re: [Quickfix-developers] FileStore & real time

[<que...@bn...>]

Thanks for the explanation.
Actually, I was thinking that  if we add a delimiter into the messages file
we would not need the header file anymore.
But with this solution, access to the messages migth be too much slower ??

Quentin.




Internet
or...@qu... - 06/11/2007 06:28 PM



To:    Quentin Courtois

cc:    drsantosfilho, quickfix-developers


Subject:    Re: [Quickfix-developers] FileStore & real time

Ok, let me explain why the file store implementation is the way it
is, and then we can talk about potential improvements.

First off why is there a header and messages file?  Well the messages
file is just a straight write of all messages without any delimiter.
When a message comes in, QuickFIX knows it can just append the
message to the end of the file.  It can then append the location of
that message into the header file.  This allows us to have a dynamic
message format the can be accessed randomly while allowing us to
record messages of arbitrary size.

This gives us a few things.  One, we no longer need to cache or do a
linear search through a file when a resend request comes in.  Caching
this data takes a lot of memory, it also creates large startup times
because the engine would have to read in and parse every message on
startup.  These were shortcomings of early versions which caused
serious issues for implementations that used a lot of messages.

The header and messages file can't really be integrated because they
need to support messages of arbitrary lengths, and the header must be
scanned in quickly on startup.  Since we do not know how many
messages will be needed, we also do not know how much space to
preallocate for the header portion, hence a separate file.

Combining sessions into a single file would improve scalability on
systems that have limitations for file constructors, but would cause
other problems, one of which would be worse performance.  First off,
it is now impossible to clear out data for a single session since
they are all integrated.  It also makes things more complicated if
you change your configuration file to add or remove sessions.  No
longer can you just delete the files for a session or have the
session clear out its own files.  The performance is going to be
worse because you are now synchronizing access to a single resource
among all of your sessions.  We have users that make use of hundreds
of high frequency sessions simultaneously.  The contention to access
this single resource would make performance unbearable.  There is
also no reason I know of that writing to multiple files would perform
worse than writing to a single file in multiple locations.

Now the session/seqnums files might give us a little more to play
with.  They could possibly be combined into the header file for
instance.  The reason the seqnums file was separated is because it is
a file that is most likely to be modified by a user, so we wanted to
keep it separate to allow users to change it relatively safely.  In
reality the data from those files could go into the header.

Also, with the current implementation, the session file is accessed
very infrequently (once every 24 hours at most).  So there is really
no good reason to hold on to that file descriptor during normal
operations.  We could just open it whenever it needs to be accessed,
and then close it.

--oren

On Jun 11, 2007, at 10:18 AM, que...@bn... wrote:

> QuickFIX Documentation: http://www.quickfixengine.org/quickfix/doc/
> html/index.html
> QuickFIX Support: http://www.quickfixengine.org/services.html
>
> Hi Djalma,
>
> Thanks a lot for your response.
> I know that databases are slower than files. We are currently using
> Oracle
> 10G.
> We already have quit good results (around 7.5 ms to store
> msg&seqnum), but
> we want to use files in order to have the best performances as
> possible.
> (I am using one process per fix session so I do not care scalablity)
> That why I wonder if the file format can be changed in order to
> obtain even
> better performances.
>
>
> Quentin.
>
>
>
>
> Internet
> drs...@gm...@lists.sourceforge.net - 06/11/2007 04:45 PM
>
>
> Sent by:    qui...@li...
>
>
>
> To:    quickfix-developers
>
> cc:
>
>
> Subject:    Re: [Quickfix-developers] FileStore & real time
>
> QuickFIX Documentation:
> http://www.quickfixengine.org/quickfix/doc/html/index.html
> QuickFIX Support: http://www.quickfixengine.org/services.html
>
> Hi Quentin,
>
> I confess that I also deslike the FileStore implementation.
> Frankly, my main problem with it is not exactly performance, but
> scalability, because my application runs out of file descriptors
> very fast
> and unfortunately there is no FD_LIMIT in Windows. In practice,
> this means
> that I cannot configure a large number of sessions in my
> application and
> this is a risk for my project.
>
> Concerning performance, I am not sure that it is so bad, especially if
> compared with the databases stores. Normally, for the databases
> like MySQL
> and PostgreSQL it is required only 4 tables for all sessions,
> however you
> have to pay the overhead of the IPC (TCP) and SQL parsing. I rember
> to have
> seen in this forum benchmarks that reported that FielStore were in
> avarage
> 3x faster than the implemetations that use databases.
>
> Some months ago I started an attempt to understand the FileStore to
> make
> some improvements and I came to the following conclusion regarding the
> created files:
>
> .session -> date and time of session creation (only 1 line)
> .seqnums -> contains the incoming and outgoing sequence number (only 1
> line)
> .body -> a stream with all sent messages
> .header -> works like an index with the offsets to recover the
> message in
> .body for retransmission
>
> Well, why not 4 files for all sessions instead of 4 files for each
> session?
> I think that it would solve the scalability problem, but
> performance would
> probably be slower, especially for multi-threaded applications that
> would
> need to rely in a single mutex to protect the integrity of the
> files and it
> would be certainly a bottleneck.
>
> Actually, I prefered to invest my effort in Store implementations
> to use
> embeded databases, like BerkeleyDB and SQLite (the development is
> still
> ongoing).
>
> Djalma
>
> On 6/11/07, que...@bn... <
> que...@bn...> wrote:
>   QuickFIX Documentation:
>   http://www.quickfixengine.org/quickfix/doc/html/index.html
>   QuickFIX Support: http://www.quickfixengine.org/services.html
>
>
>
>   Hi,
>
>   I think that the way used by quickFix to store fix persistent
> datas into
>   4
>   files is not the best solution to have the best performances
> possible.
>
>   Indeed for each fix messages send to the counterparty, QuickFix
> has to
>   save
>   synchronously datas into 3 files (seqnum+header+messages).
>
>   Can't we have a better file format that allows to save only into
> one file
>   for each message ?
>
>   Please send me your thougths about this real time issue.
>
>
>   Thanks,
>
>   Quentin.
>
>
>
>   This message and any attachments (the "message") is
>   intended solely for the addressees and is confidential.
>   If you receive this message in error, please delete it and
>   immediately notify the sender. Any use not in accord with
>   its purpose, any dissemination or disclosure, either whole
>   or partial, is prohibited except formal approval. The internet
>   can not guarantee the integrity of this message.
>   BNP PARIBAS (and its subsidiaries) shall (will) not
>   therefore be liable for the message if modified.
>
>                   ---------------------------------------------
>
>   Ce message et toutes les pieces jointes (ci-apres le
>   "message") sont etablis a l'intention exclusive de ses
>   destinataires et sont confidentiels. Si vous recevez ce
>   message par erreur, merci de le detruire et d'en avertir
>   immediatement l'expediteur. Toute utilisation de ce
>   message non conforme a sa destination, toute diffusion
>   ou toute publication, totale ou partielle, est interdite, sauf
>   autorisation expresse. L'internet ne permettant pas
>   d'assurer l'integrite de ce message, BNP PARIBAS (et ses
>   filiales) decline(nt) toute responsabilite au titre de ce
>   message, dans l'hypothese ou il aurait ete modifie.
>
>
>
> ----------------------------------------------------------------------
> ---
>   This SF.net email is sponsored by DB2 Express
>   Download DB2 Express C - the FREE version of DB2 express and take
>   control of your XML. No limits. Just data. Click to get it now.
>   http://sourceforge.net/powerbar/db2/
>   _______________________________________________
>   Quickfix-developers mailing list
>   Qui...@li...
>   https://lists.sourceforge.net/lists/listinfo/quickfix-developers
> ----------------------------------------------------------------------
> ---
> This SF.net email is sponsored by DB2 Express
> Download DB2 Express C - the FREE version of DB2 express and take
> control of your XML. No limits. Just data. Click to get it now.
> http://sourceforge.net/powerbar/db2/
> _______________________________________________
> Quickfix-developers mailing list
> Qui...@li...
> https://lists.sourceforge.net/lists/listinfo/quickfix-developers
>
> ----------------------------------------------------------------------
> ---
> This SF.net email is sponsored by DB2 Express
> Download DB2 Express C - the FREE version of DB2 express and take
> control of your XML. No limits. Just data. Click to get it now.
> http://sourceforge.net/powerbar/db2/
> _______________________________________________
> Quickfix-developers mailing list
> Qui...@li...
> https://lists.sourceforge.net/lists/listinfo/quickfix-developers
>

[Quickfix-developers] Possible bug in quickfixj FieldMap.removeGroup(int)

[David B. <dbu...@gm...>]

Hi Folks,

wasn't sure what the 'official' channel was to submit a patch / find if this
was a real issue.

QuickFixJ 1.1.0, FieldMap.java line 641
FieldMap.removeGroup(int)
does:

getGroups(field).clear();

Then it removes the counter field itself from the message.

Problem is, the string calculation code just iterates over the groups key
set and adds the groups in the message for each counter.
Which in effect gives you a counter who's value = 0 (since the list of Group
objects is empty from the clear, but the Integer counter key remains)

simple fix is just replacing that line with:
groups.remove(new Integer(field));

Works for us, and I can submit a patch if there's a proper place to do it.
I suppose another question is, should the counter be removed (it appears
that was the intention by the original function, just doesn't work that way
when building the fix string)?

Was hoping to get it in the main line (assuming it's the right thing to do
that is) so we don't have to re-patch as future revs come out.

Thanks,
-David

Re: [Quickfix-developers] About UseDataDictionary, ValidateUserDefinedField and Reject

[Djalma R. d. S. F. <drs...@gm...>]

Hi,

I don't think that it is possible to completely turn off message
validation with QF.
What you can do is disable the validation of UserDefinedFields.
For UDF you must use a number above 5000, otherwhise QF will apply the
validation and reject the message if field is not correct according to
xml data dictionary.

UseDataDictionary is required to parse repeating groups.

You can also turn off ValidateFieldsOutOfOrder=N and ValidateFieldsHaveValues=N.

Djalma


On 6/8/07, quickfixuser <fw...@ro...> wrote:
> QuickFIX Documentation: http://www.quickfixengine.org/quickfix/doc/html/index.html
> QuickFIX Support: http://www.quickfixengine.org/services.html
>
>
> Hi,
>
> Since the party, which I try to connect to, does not allow any reject from
> my side. I tried to put ValidateUserDefinedField as N, but keep
> UseDataDictorary as Y after I modified the xml for the new fields. However,
> my quickfix still send out rejection due to "Tag not defined for this
> message type".
> A little bit confused about why we can set UseDataDictirony as Y and
> ValidateUserDefinedField as N? Also, if I want to disable rejection, what
> will be the best place to do it in quickfix?
>
> Thanks for the help
> --
> View this message in context: http://www.nabble.com/About-UseDataDictionary%2C-ValidateUserDefinedField-and-Reject-tf3890466.html#a11028586
> Sent from the QuickFIX - Dev mailing list archive at Nabble.com.
>
>
> -------------------------------------------------------------------------
> This SF.net email is sponsored by DB2 Express
> Download DB2 Express C - the FREE version of DB2 express and take
> control of your XML. No limits. Just data. Click to get it now.
> http://sourceforge.net/powerbar/db2/
> _______________________________________________
> Quickfix-developers mailing list
> Qui...@li...
> https://lists.sourceforge.net/lists/listinfo/quickfix-developers
>

Re: [Quickfix-developers] FileStore & real time

[Alexey Z. <ale...@gm...>]

Oren,

I didn't mean that the performance is bad. I just horrified that the 
files at the end of a day usually have more than 5K elements. And the 
messages logs even more. I use quickfix 1.12.4.1897 version and I see 
the messages are resending and the disconnection happening on a missing 
HeartBeat.

Regards,
		Alexey Zubko



Oren Miller wrote:
>> Just my 2 cents about the problem.
>>
>> We have a huge file fragmentation for these files (NTFS) and I'm pretty
>> sure the disk IO performance is really bad.
>> Reading of the header and body files in case of crash is just awful.
>> My applications were disconnected several times by the heartbeat timeout
>> during the recovering and everything was started over after the
>> reconnection.
>
> I'm not sure how this is possible.  The body file is not read on 
> startup.  This is part of why the header file exists.  I think what 
> you are seeing is the time it takes to send the resend requests over 
> the socket.  This is a problem that has been resolved by processing 
> heartbeats during resend requests.  It has nothing to do with store 
> performance.
>
> --oren
>

Re: [Quickfix-developers] FileStore & real time

[Djalma R. d. S. F. <drs...@gm...>]

Hi Oren,

Thank you for the explanation, but as I previously told I had already given
up to figure out improvements to FileStore, exactly because of some issues
you explained so well and now also because of others I wasn't yet aware of.
But, of course it does not mean that a genius somewhere can come with the
solution.

For higher scalability in Windows without the worst performance of regular
databases, my bet is BerkeleyDB and SQLite (or MemoryStore with
PersistMessages=N for configurations that do not require retransmission).

When development is finished, I will let you know about the test results.

Best regards,
Djalma

On 6/11/07, Oren Miller <or...@qu...> wrote:
>
> Ok, let me explain why the file store implementation is the way it
> is, and then we can talk about potential improvements.
>
> First off why is there a header and messages file?  Well the messages
> file is just a straight write of all messages without any delimiter.
> When a message comes in, QuickFIX knows it can just append the
> message to the end of the file.  It can then append the location of
> that message into the header file.  This allows us to have a dynamic
> message format the can be accessed randomly while allowing us to
> record messages of arbitrary size.
>
> This gives us a few things.  One, we no longer need to cache or do a
> linear search through a file when a resend request comes in.  Caching
> this data takes a lot of memory, it also creates large startup times
> because the engine would have to read in and parse every message on
> startup.  These were shortcomings of early versions which caused
> serious issues for implementations that used a lot of messages.
>
> The header and messages file can't really be integrated because they
> need to support messages of arbitrary lengths, and the header must be
> scanned in quickly on startup.  Since we do not know how many
> messages will be needed, we also do not know how much space to
> preallocate for the header portion, hence a separate file.
>
> Combining sessions into a single file would improve scalability on
> systems that have limitations for file constructors, but would cause
> other problems, one of which would be worse performance.  First off,
> it is now impossible to clear out data for a single session since
> they are all integrated.  It also makes things more complicated if
> you change your configuration file to add or remove sessions.  No
> longer can you just delete the files for a session or have the
> session clear out its own files.  The performance is going to be
> worse because you are now synchronizing access to a single resource
> among all of your sessions.  We have users that make use of hundreds
> of high frequency sessions simultaneously.  The contention to access
> this single resource would make performance unbearable.  There is
> also no reason I know of that writing to multiple files would perform
> worse than writing to a single file in multiple locations.
>
> Now the session/seqnums files might give us a little more to play
> with.  They could possibly be combined into the header file for
> instance.  The reason the seqnums file was separated is because it is
> a file that is most likely to be modified by a user, so we wanted to
> keep it separate to allow users to change it relatively safely.  In
> reality the data from those files could go into the header.
>
> Also, with the current implementation, the session file is accessed
> very infrequently (once every 24 hours at most).  So there is really
> no good reason to hold on to that file descriptor during normal
> operations.  We could just open it whenever it needs to be accessed,
> and then close it.
>
> --oren
>
> On Jun 11, 2007, at 10:18 AM, que...@bn... wrote:
>
> > QuickFIX Documentation: http://www.quickfixengine.org/quickfix/doc/
> > html/index.html
> > QuickFIX Support: http://www.quickfixengine.org/services.html
> >
> > Hi Djalma,
> >
> > Thanks a lot for your response.
> > I know that databases are slower than files. We are currently using
> > Oracle
> > 10G.
> > We already have quit good results (around 7.5 ms to store
> > msg&seqnum), but
> > we want to use files in order to have the best performances as
> > possible.
> > (I am using one process per fix session so I do not care scalablity)
> > That why I wonder if the file format can be changed in order to
> > obtain even
> > better performances.
> >
> >
> > Quentin.
> >
> >
> >
> >
> > Internet
> > drs...@gm...@lists.sourceforge.net - 06/11/2007 04:45 PM
> >
> >
> > Sent by:    qui...@li...
> >
> >
> >
> > To:    quickfix-developers
> >
> > cc:
> >
> >
> > Subject:    Re: [Quickfix-developers] FileStore & real time
> >
> > QuickFIX Documentation:
> > http://www.quickfixengine.org/quickfix/doc/html/index.html
> > QuickFIX Support: http://www.quickfixengine.org/services.html
> >
> > Hi Quentin,
> >
> > I confess that I also deslike the FileStore implementation.
> > Frankly, my main problem with it is not exactly performance, but
> > scalability, because my application runs out of file descriptors
> > very fast
> > and unfortunately there is no FD_LIMIT in Windows. In practice,
> > this means
> > that I cannot configure a large number of sessions in my
> > application and
> > this is a risk for my project.
> >
> > Concerning performance, I am not sure that it is so bad, especially if
> > compared with the databases stores. Normally, for the databases
> > like MySQL
> > and PostgreSQL it is required only 4 tables for all sessions,
> > however you
> > have to pay the overhead of the IPC (TCP) and SQL parsing. I rember
> > to have
> > seen in this forum benchmarks that reported that FielStore were in
> > avarage
> > 3x faster than the implemetations that use databases.
> >
> > Some months ago I started an attempt to understand the FileStore to
> > make
> > some improvements and I came to the following conclusion regarding the
> > created files:
> >
> > .session -> date and time of session creation (only 1 line)
> > .seqnums -> contains the incoming and outgoing sequence number (only 1
> > line)
> > .body -> a stream with all sent messages
> > .header -> works like an index with the offsets to recover the
> > message in
> > .body for retransmission
> >
> > Well, why not 4 files for all sessions instead of 4 files for each
> > session?
> > I think that it would solve the scalability problem, but
> > performance would
> > probably be slower, especially for multi-threaded applications that
> > would
> > need to rely in a single mutex to protect the integrity of the
> > files and it
> > would be certainly a bottleneck.
> >
> > Actually, I prefered to invest my effort in Store implementations
> > to use
> > embeded databases, like BerkeleyDB and SQLite (the development is
> > still
> > ongoing).
> >
> > Djalma
> >
> > On 6/11/07, que...@bn... <
> > que...@bn...> wrote:
> >   QuickFIX Documentation:
> >   http://www.quickfixengine.org/quickfix/doc/html/index.html
> >   QuickFIX Support: http://www.quickfixengine.org/services.html
> >
> >
> >
> >   Hi,
> >
> >   I think that the way used by quickFix to store fix persistent
> > datas into
> >   4
> >   files is not the best solution to have the best performances
> > possible.
> >
> >   Indeed for each fix messages send to the counterparty, QuickFix
> > has to
> >   save
> >   synchronously datas into 3 files (seqnum+header+messages).
> >
> >   Can't we have a better file format that allows to save only into
> > one file
> >   for each message ?
> >
> >   Please send me your thougths about this real time issue.
> >
> >
> >   Thanks,
> >
> >   Quentin.
> >
> >
> >
> >   This message and any attachments (the "message") is
> >   intended solely for the addressees and is confidential.
> >   If you receive this message in error, please delete it and
> >   immediately notify the sender. Any use not in accord with
> >   its purpose, any dissemination or disclosure, either whole
> >   or partial, is prohibited except formal approval. The internet
> >   can not guarantee the integrity of this message.
> >   BNP PARIBAS (and its subsidiaries) shall (will) not
> >   therefore be liable for the message if modified.
> >
> >                   ---------------------------------------------
> >
> >   Ce message et toutes les pieces jointes (ci-apres le
> >   "message") sont etablis a l'intention exclusive de ses
> >   destinataires et sont confidentiels. Si vous recevez ce
> >   message par erreur, merci de le detruire et d'en avertir
> >   immediatement l'expediteur. Toute utilisation de ce
> >   message non conforme a sa destination, toute diffusion
> >   ou toute publication, totale ou partielle, est interdite, sauf
> >   autorisation expresse. L'internet ne permettant pas
> >   d'assurer l'integrite de ce message, BNP PARIBAS (et ses
> >   filiales) decline(nt) toute responsabilite au titre de ce
> >   message, dans l'hypothese ou il aurait ete modifie.
> >
> >
> >
> > ----------------------------------------------------------------------
> > ---
> >   This SF.net email is sponsored by DB2 Express
> >   Download DB2 Express C - the FREE version of DB2 express and take
> >   control of your XML. No limits. Just data. Click to get it now.
> >   http://sourceforge.net/powerbar/db2/
> >   _______________________________________________
> >   Quickfix-developers mailing list
> >   Qui...@li...
> >   https://lists.sourceforge.net/lists/listinfo/quickfix-developers
> > ----------------------------------------------------------------------
> > ---
> > This SF.net email is sponsored by DB2 Express
> > Download DB2 Express C - the FREE version of DB2 express and take
> > control of your XML. No limits. Just data. Click to get it now.
> > http://sourceforge.net/powerbar/db2/
> > _______________________________________________
> > Quickfix-developers mailing list
> > Qui...@li...
> > https://lists.sourceforge.net/lists/listinfo/quickfix-developers
> >
> > ----------------------------------------------------------------------
> > ---
> > This SF.net email is sponsored by DB2 Express
> > Download DB2 Express C - the FREE version of DB2 express and take
> > control of your XML. No limits. Just data. Click to get it now.
> > http://sourceforge.net/powerbar/db2/
> > _______________________________________________
> > Quickfix-developers mailing list
> > Qui...@li...
> > https://lists.sourceforge.net/lists/listinfo/quickfix-developers
> >
>
>

Re: [Quickfix-developers] FileStore & real time

[Oren M. <or...@qu...>]

> Just my 2 cents about the problem.
>
> We have a huge file fragmentation for these files (NTFS) and I'm  
> pretty
> sure the disk IO performance is really bad.
> Reading of the header and body files in case of crash is just awful.
> My applications were disconnected several times by the heartbeat  
> timeout
> during the recovering and everything was started over after the
> reconnection.

I'm not sure how this is possible.  The body file is not read on  
startup.  This is part of why the header file exists.  I think what  
you are seeing is the time it takes to send the resend requests over  
the socket.  This is a problem that has been resolved by processing  
heartbeats during resend requests.  It has nothing to do with store  
performance.

--oren

Re: [Quickfix-developers] FileStore & real time

[Alexey Z. <ale...@gm...>]

Hello,

Just my 2 cents about the problem.

We have a huge file fragmentation for these files (NTFS) and I'm pretty 
sure the disk IO performance is really bad.
Reading of the header and body files in case of crash is just awful.
My applications were disconnected several times by the heartbeat timeout 
during the recovering and everything was started over after the 
reconnection.

BTW, in the FileLog file I think std::flush is redundant - std::endl 
flushes the stream:
 void onIncoming( const std::string& value )
  { m_messages << value << std::endl << std::flush; }


Regards,
		Alexey Zubko



>
> ------------------------------------------------------------------------
>
> Subject:
> Re: [Quickfix-developers] FileStore & real time
> From:
> Oren Miller <or...@qu...>
> Date:
> Mon, 11 Jun 2007 11:28:38 -0500
> To:
> que...@bn...
>
> To:
> que...@bn...
> CC:
> qui...@li...
>
>
> Ok, let me explain why the file store implementation is the way it is, 
> and then we can talk about potential improvements.
>
> First off why is there a header and messages file?  Well the messages 
> file is just a straight write of all messages without any delimiter.  
> When a message comes in, QuickFIX knows it can just append the message 
> to the end of the file.  It can then append the location of that 
> message into the header file.  This allows us to have a dynamic 
> message format the can be accessed randomly while allowing us to 
> record messages of arbitrary size.
>
> This gives us a few things.  One, we no longer need to cache or do a 
> linear search through a file when a resend request comes in.  Caching 
> this data takes a lot of memory, it also creates large startup times 
> because the engine would have to read in and parse every message on 
> startup.  These were shortcomings of early versions which caused 
> serious issues for implementations that used a lot of messages.
>
> The header and messages file can't really be integrated because they 
> need to support messages of arbitrary lengths, and the header must be 
> scanned in quickly on startup.  Since we do not know how many messages 
> will be needed, we also do not know how much space to preallocate for 
> the header portion, hence a separate file.
>
> Combining sessions into a single file would improve scalability on 
> systems that have limitations for file constructors, but would cause 
> other problems, one of which would be worse performance.  First off, 
> it is now impossible to clear out data for a single session since they 
> are all integrated.  It also makes things more complicated if you 
> change your configuration file to add or remove sessions.  No longer 
> can you just delete the files for a session or have the session clear 
> out its own files.  The performance is going to be worse because you 
> are now synchronizing access to a single resource among all of your 
> sessions.  We have users that make use of hundreds of high frequency 
> sessions simultaneously.  The contention to access this single 
> resource would make performance unbearable.  There is also no reason I 
> know of that writing to multiple files would perform worse than 
> writing to a single file in multiple locations.
>
> Now the session/seqnums files might give us a little more to play 
> with.  They could possibly be combined into the header file for 
> instance.  The reason the seqnums file was separated is because it is 
> a file that is most likely to be modified by a user, so we wanted to 
> keep it separate to allow users to change it relatively safely.  In 
> reality the data from those files could go into the header.
>
> Also, with the current implementation, the session file is accessed 
> very infrequently (once every 24 hours at most).  So there is really 
> no good reason to hold on to that file descriptor during normal 
> operations.  We could just open it whenever it needs to be accessed, 
> and then close it.
>
> --oren
>
> On Jun 11, 2007, at 10:18 AM, que...@bn... wrote:
>
>> QuickFIX Documentation: 
>> http://www.quickfixengine.org/quickfix/doc/html/index.html
>> QuickFIX Support: http://www.quickfixengine.org/services.html
>>
>> Hi Djalma,
>>
>> Thanks a lot for your response.
>> I know that databases are slower than files. We are currently using 
>> Oracle
>> 10G.
>> We already have quit good results (around 7.5 ms to store 
>> msg&seqnum), but
>> we want to use files in order to have the best performances as possible.
>> (I am using one process per fix session so I do not care scalablity)
>> That why I wonder if the file format can be changed in order to 
>> obtain even
>> better performances.
>>
>>
>> Quentin.
>>
>>
>>
>>
>> Internet
>> drs...@gm...@lists.sourceforge.net - 06/11/2007 04:45 PM
>>
>>
>> Sent by:    qui...@li...
>>
>>
>>
>> To:    quickfix-developers
>>
>> cc:
>>
>>
>> Subject:    Re: [Quickfix-developers] FileStore & real time
>>
>> QuickFIX Documentation:
>> http://www.quickfixengine.org/quickfix/doc/html/index.html
>> QuickFIX Support: http://www.quickfixengine.org/services.html
>>
>> Hi Quentin,
>>
>> I confess that I also deslike the FileStore implementation.
>> Frankly, my main problem with it is not exactly performance, but
>> scalability, because my application runs out of file descriptors very 
>> fast
>> and unfortunately there is no FD_LIMIT in Windows. In practice, this 
>> means
>> that I cannot configure a large number of sessions in my application and
>> this is a risk for my project.
>>
>> Concerning performance, I am not sure that it is so bad, especially if
>> compared with the databases stores. Normally, for the databases like 
>> MySQL
>> and PostgreSQL it is required only 4 tables for all sessions, however 
>> you
>> have to pay the overhead of the IPC (TCP) and SQL parsing. I rember 
>> to have
>> seen in this forum benchmarks that reported that FielStore were in 
>> avarage
>> 3x faster than the implemetations that use databases.
>>
>> Some months ago I started an attempt to understand the FileStore to make
>> some improvements and I came to the following conclusion regarding the
>> created files:
>>
>> .session -> date and time of session creation (only 1 line)
>> .seqnums -> contains the incoming and outgoing sequence number (only 1
>> line)
>> .body -> a stream with all sent messages
>> .header -> works like an index with the offsets to recover the 
>> message in
>> .body for retransmission
>>
>> Well, why not 4 files for all sessions instead of 4 files for each 
>> session?
>> I think that it would solve the scalability problem, but performance 
>> would
>> probably be slower, especially for multi-threaded applications that 
>> would
>> need to rely in a single mutex to protect the integrity of the files 
>> and it
>> would be certainly a bottleneck.
>>
>> Actually, I prefered to invest my effort in Store implementations to use
>> embeded databases, like BerkeleyDB and SQLite (the development is still
>> ongoing).
>>
>> Djalma
>>
>> On 6/11/07, que...@bn... <
>> que...@bn...> wrote:
>>   QuickFIX Documentation:
>>   http://www.quickfixengine.org/quickfix/doc/html/index.html
>>   QuickFIX Support: http://www.quickfixengine.org/services.html
>>
>>
>>
>>   Hi,
>>
>>   I think that the way used by quickFix to store fix persistent datas 
>> into
>>   4
>>   files is not the best solution to have the best performances possible.
>>
>>   Indeed for each fix messages send to the counterparty, QuickFix has to
>>   save
>>   synchronously datas into 3 files (seqnum+header+messages).
>>
>>   Can't we have a better file format that allows to save only into 
>> one file
>>   for each message ?
>>
>>   Please send me your thougths about this real time issue.
>>
>>
>>   Thanks,
>>
>>   Quentin.
>>
>>
>>
>>   This message and any attachments (the "message") is
>>   intended solely for the addressees and is confidential.
>>   If you receive this message in error, please delete it and
>>   immediately notify the sender. Any use not in accord with
>>   its purpose, any dissemination or disclosure, either whole
>>   or partial, is prohibited except formal approval. The internet
>>   can not guarantee the integrity of this message.
>>   BNP PARIBAS (and its subsidiaries) shall (will) not
>>   therefore be liable for the message if modified.
>>
>>                   ---------------------------------------------
>>
>>   Ce message et toutes les pieces jointes (ci-apres le
>>   "message") sont etablis a l'intention exclusive de ses
>>   destinataires et sont confidentiels. Si vous recevez ce
>>   message par erreur, merci de le detruire et d'en avertir
>>   immediatement l'expediteur. Toute utilisation de ce
>>   message non conforme a sa destination, toute diffusion
>>   ou toute publication, totale ou partielle, est interdite, sauf
>>   autorisation expresse. L'internet ne permettant pas
>>   d'assurer l'integrite de ce message, BNP PARIBAS (et ses
>>   filiales) decline(nt) toute responsabilite au titre de ce
>>   message, dans l'hypothese ou il aurait ete modifie.
>>
>>
>>   
>> ------------------------------------------------------------------------- 
>>
>>   This SF.net email is sponsored by DB2 Express
>>   Download DB2 Express C - the FREE version of DB2 express and take
>>   control of your XML. No limits. Just data. Click to get it now.
>>   http://sourceforge.net/powerbar/db2/
>>   _______________________________________________
>>   Quickfix-developers mailing list
>>   Qui...@li...
>>   https://lists.sourceforge.net/lists/listinfo/quickfix-developers
>> ------------------------------------------------------------------------- 
>>
>> This SF.net email is sponsored by DB2 Express
>> Download DB2 Express C - the FREE version of DB2 express and take
>> control of your XML. No limits. Just data. Click to get it now.
>> http://sourceforge.net/powerbar/db2/
>> _______________________________________________
>> Quickfix-developers mailing list
>> Qui...@li...
>> https://lists.sourceforge.net/lists/listinfo/quickfix-developers
>>
>> ------------------------------------------------------------------------- 
>>
>> This SF.net email is sponsored by DB2 Express
>> Download DB2 Express C - the FREE version of DB2 express and take
>> control of your XML. No limits. Just data. Click to get it now.
>> http://sourceforge.net/powerbar/db2/
>> _______________________________________________
>> Quickfix-developers mailing list
>> Qui...@li...
>> https://lists.sourceforge.net/lists/listinfo/quickfix-developers
>>
>
>
>

Re: [Quickfix-developers] FileStore & real time

[Oren M. <or...@qu...>]

Ok, let me explain why the file store implementation is the way it  
is, and then we can talk about potential improvements.

First off why is there a header and messages file?  Well the messages  
file is just a straight write of all messages without any delimiter.   
When a message comes in, QuickFIX knows it can just append the  
message to the end of the file.  It can then append the location of  
that message into the header file.  This allows us to have a dynamic  
message format the can be accessed randomly while allowing us to  
record messages of arbitrary size.

This gives us a few things.  One, we no longer need to cache or do a  
linear search through a file when a resend request comes in.  Caching  
this data takes a lot of memory, it also creates large startup times  
because the engine would have to read in and parse every message on  
startup.  These were shortcomings of early versions which caused  
serious issues for implementations that used a lot of messages.

The header and messages file can't really be integrated because they  
need to support messages of arbitrary lengths, and the header must be  
scanned in quickly on startup.  Since we do not know how many  
messages will be needed, we also do not know how much space to  
preallocate for the header portion, hence a separate file.

Combining sessions into a single file would improve scalability on  
systems that have limitations for file constructors, but would cause  
other problems, one of which would be worse performance.  First off,  
it is now impossible to clear out data for a single session since  
they are all integrated.  It also makes things more complicated if  
you change your configuration file to add or remove sessions.  No  
longer can you just delete the files for a session or have the  
session clear out its own files.  The performance is going to be  
worse because you are now synchronizing access to a single resource  
among all of your sessions.  We have users that make use of hundreds  
of high frequency sessions simultaneously.  The contention to access  
this single resource would make performance unbearable.  There is  
also no reason I know of that writing to multiple files would perform  
worse than writing to a single file in multiple locations.

Now the session/seqnums files might give us a little more to play  
with.  They could possibly be combined into the header file for  
instance.  The reason the seqnums file was separated is because it is  
a file that is most likely to be modified by a user, so we wanted to  
keep it separate to allow users to change it relatively safely.  In  
reality the data from those files could go into the header.

Also, with the current implementation, the session file is accessed  
very infrequently (once every 24 hours at most).  So there is really  
no good reason to hold on to that file descriptor during normal  
operations.  We could just open it whenever it needs to be accessed,  
and then close it.

--oren

On Jun 11, 2007, at 10:18 AM, que...@bn... wrote:

> QuickFIX Documentation: http://www.quickfixengine.org/quickfix/doc/ 
> html/index.html
> QuickFIX Support: http://www.quickfixengine.org/services.html
>
> Hi Djalma,
>
> Thanks a lot for your response.
> I know that databases are slower than files. We are currently using  
> Oracle
> 10G.
> We already have quit good results (around 7.5 ms to store  
> msg&seqnum), but
> we want to use files in order to have the best performances as  
> possible.
> (I am using one process per fix session so I do not care scalablity)
> That why I wonder if the file format can be changed in order to  
> obtain even
> better performances.
>
>
> Quentin.
>
>
>
>
> Internet
> drs...@gm...@lists.sourceforge.net - 06/11/2007 04:45 PM
>
>
> Sent by:    qui...@li...
>
>
>
> To:    quickfix-developers
>
> cc:
>
>
> Subject:    Re: [Quickfix-developers] FileStore & real time
>
> QuickFIX Documentation:
> http://www.quickfixengine.org/quickfix/doc/html/index.html
> QuickFIX Support: http://www.quickfixengine.org/services.html
>
> Hi Quentin,
>
> I confess that I also deslike the FileStore implementation.
> Frankly, my main problem with it is not exactly performance, but
> scalability, because my application runs out of file descriptors  
> very fast
> and unfortunately there is no FD_LIMIT in Windows. In practice,  
> this means
> that I cannot configure a large number of sessions in my  
> application and
> this is a risk for my project.
>
> Concerning performance, I am not sure that it is so bad, especially if
> compared with the databases stores. Normally, for the databases  
> like MySQL
> and PostgreSQL it is required only 4 tables for all sessions,  
> however you
> have to pay the overhead of the IPC (TCP) and SQL parsing. I rember  
> to have
> seen in this forum benchmarks that reported that FielStore were in  
> avarage
> 3x faster than the implemetations that use databases.
>
> Some months ago I started an attempt to understand the FileStore to  
> make
> some improvements and I came to the following conclusion regarding the
> created files:
>
> .session -> date and time of session creation (only 1 line)
> .seqnums -> contains the incoming and outgoing sequence number (only 1
> line)
> .body -> a stream with all sent messages
> .header -> works like an index with the offsets to recover the  
> message in
> .body for retransmission
>
> Well, why not 4 files for all sessions instead of 4 files for each  
> session?
> I think that it would solve the scalability problem, but  
> performance would
> probably be slower, especially for multi-threaded applications that  
> would
> need to rely in a single mutex to protect the integrity of the  
> files and it
> would be certainly a bottleneck.
>
> Actually, I prefered to invest my effort in Store implementations  
> to use
> embeded databases, like BerkeleyDB and SQLite (the development is  
> still
> ongoing).
>
> Djalma
>
> On 6/11/07, que...@bn... <
> que...@bn...> wrote:
>   QuickFIX Documentation:
>   http://www.quickfixengine.org/quickfix/doc/html/index.html
>   QuickFIX Support: http://www.quickfixengine.org/services.html
>
>
>
>   Hi,
>
>   I think that the way used by quickFix to store fix persistent  
> datas into
>   4
>   files is not the best solution to have the best performances  
> possible.
>
>   Indeed for each fix messages send to the counterparty, QuickFix  
> has to
>   save
>   synchronously datas into 3 files (seqnum+header+messages).
>
>   Can't we have a better file format that allows to save only into  
> one file
>   for each message ?
>
>   Please send me your thougths about this real time issue.
>
>
>   Thanks,
>
>   Quentin.
>
>
>
>   This message and any attachments (the "message") is
>   intended solely for the addressees and is confidential.
>   If you receive this message in error, please delete it and
>   immediately notify the sender. Any use not in accord with
>   its purpose, any dissemination or disclosure, either whole
>   or partial, is prohibited except formal approval. The internet
>   can not guarantee the integrity of this message.
>   BNP PARIBAS (and its subsidiaries) shall (will) not
>   therefore be liable for the message if modified.
>
>                   ---------------------------------------------
>
>   Ce message et toutes les pieces jointes (ci-apres le
>   "message") sont etablis a l'intention exclusive de ses
>   destinataires et sont confidentiels. Si vous recevez ce
>   message par erreur, merci de le detruire et d'en avertir
>   immediatement l'expediteur. Toute utilisation de ce
>   message non conforme a sa destination, toute diffusion
>   ou toute publication, totale ou partielle, est interdite, sauf
>   autorisation expresse. L'internet ne permettant pas
>   d'assurer l'integrite de ce message, BNP PARIBAS (et ses
>   filiales) decline(nt) toute responsabilite au titre de ce
>   message, dans l'hypothese ou il aurait ete modifie.
>
>
>    
> ---------------------------------------------------------------------- 
> ---
>   This SF.net email is sponsored by DB2 Express
>   Download DB2 Express C - the FREE version of DB2 express and take
>   control of your XML. No limits. Just data. Click to get it now.
>   http://sourceforge.net/powerbar/db2/
>   _______________________________________________
>   Quickfix-developers mailing list
>   Qui...@li...
>   https://lists.sourceforge.net/lists/listinfo/quickfix-developers
> ---------------------------------------------------------------------- 
> ---
> This SF.net email is sponsored by DB2 Express
> Download DB2 Express C - the FREE version of DB2 express and take
> control of your XML. No limits. Just data. Click to get it now.
> http://sourceforge.net/powerbar/db2/
> _______________________________________________
> Quickfix-developers mailing list
> Qui...@li...
> https://lists.sourceforge.net/lists/listinfo/quickfix-developers
>
> ---------------------------------------------------------------------- 
> ---
> This SF.net email is sponsored by DB2 Express
> Download DB2 Express C - the FREE version of DB2 express and take
> control of your XML. No limits. Just data. Click to get it now.
> http://sourceforge.net/powerbar/db2/
> _______________________________________________
> Quickfix-developers mailing list
> Qui...@li...
> https://lists.sourceforge.net/lists/listinfo/quickfix-developers
>

Re: [Quickfix-developers] FileStore & real time

[<que...@bn...>]

Hi Djalma,

Thanks a lot for your response.
I know that databases are slower than files. We are currently using Oracle
10G.
We already have quit good results (around 7.5 ms to store msg&seqnum), but
we want to use files in order to have the best performances as possible.
(I am using one process per fix session so I do not care scalablity)
That why I wonder if the file format can be changed in order to obtain even
better performances.


Quentin.




Internet
drs...@gm...@lists.sourceforge.net - 06/11/2007 04:45 PM


Sent by:    qui...@li...



To:    quickfix-developers

cc:


Subject:    Re: [Quickfix-developers] FileStore & real time

QuickFIX Documentation:
http://www.quickfixengine.org/quickfix/doc/html/index.html
QuickFIX Support: http://www.quickfixengine.org/services.html

Hi Quentin,

I confess that I also deslike the FileStore implementation.
Frankly, my main problem with it is not exactly performance, but
scalability, because my application runs out of file descriptors very fast
and unfortunately there is no FD_LIMIT in Windows. In practice, this means
that I cannot configure a large number of sessions in my application and
this is a risk for my project.

Concerning performance, I am not sure that it is so bad, especially if
compared with the databases stores. Normally, for the databases like MySQL
and PostgreSQL it is required only 4 tables for all sessions, however you
have to pay the overhead of the IPC (TCP) and SQL parsing. I rember to have
seen in this forum benchmarks that reported that FielStore were in avarage
3x faster than the implemetations that use databases.

Some months ago I started an attempt to understand the FileStore to make
some improvements and I came to the following conclusion regarding the
created files:

.session -> date and time of session creation (only 1 line)
.seqnums -> contains the incoming and outgoing sequence number (only 1
line)
.body -> a stream with all sent messages
.header -> works like an index with the offsets to recover the message in
.body for retransmission

Well, why not 4 files for all sessions instead of 4 files for each session?
I think that it would solve the scalability problem, but performance would
probably be slower, especially for multi-threaded applications that would
need to rely in a single mutex to protect the integrity of the files and it
would be certainly a bottleneck.

Actually, I prefered to invest my effort in Store implementations to use
embeded databases, like BerkeleyDB and SQLite (the development is still
ongoing).

Djalma

On 6/11/07, que...@bn... <
que...@bn...> wrote:
  QuickFIX Documentation:
  http://www.quickfixengine.org/quickfix/doc/html/index.html
  QuickFIX Support: http://www.quickfixengine.org/services.html



  Hi,

  I think that the way used by quickFix to store fix persistent datas into
  4
  files is not the best solution to have the best performances possible.

  Indeed for each fix messages send to the counterparty, QuickFix has to
  save
  synchronously datas into 3 files (seqnum+header+messages).

  Can't we have a better file format that allows to save only into one file
  for each message ?

  Please send me your thougths about this real time issue.


  Thanks,

  Quentin.



  This message and any attachments (the "message") is
  intended solely for the addressees and is confidential.
  If you receive this message in error, please delete it and
  immediately notify the sender. Any use not in accord with
  its purpose, any dissemination or disclosure, either whole
  or partial, is prohibited except formal approval. The internet
  can not guarantee the integrity of this message.
  BNP PARIBAS (and its subsidiaries) shall (will) not
  therefore be liable for the message if modified.

                  ---------------------------------------------

  Ce message et toutes les pieces jointes (ci-apres le
  "message") sont etablis a l'intention exclusive de ses
  destinataires et sont confidentiels. Si vous recevez ce
  message par erreur, merci de le detruire et d'en avertir
  immediatement l'expediteur. Toute utilisation de ce
  message non conforme a sa destination, toute diffusion
  ou toute publication, totale ou partielle, est interdite, sauf
  autorisation expresse. L'internet ne permettant pas
  d'assurer l'integrite de ce message, BNP PARIBAS (et ses
  filiales) decline(nt) toute responsabilite au titre de ce
  message, dans l'hypothese ou il aurait ete modifie.


  -------------------------------------------------------------------------
  This SF.net email is sponsored by DB2 Express
  Download DB2 Express C - the FREE version of DB2 express and take
  control of your XML. No limits. Just data. Click to get it now.
  http://sourceforge.net/powerbar/db2/
  _______________________________________________
  Quickfix-developers mailing list
  Qui...@li...
  https://lists.sourceforge.net/lists/listinfo/quickfix-developers
-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
_______________________________________________
Quickfix-developers mailing list
Qui...@li...
https://lists.sourceforge.net/lists/listinfo/quickfix-developers

Re: [Quickfix-developers] FileStore & real time

[Djalma R. d. S. F. <drs...@gm...>]

Hi Quentin,

I confess that I also deslike the FileStore implementation.
Frankly, my main problem with it is not exactly performance, but
scalability, because my application runs out of file descriptors very fast
and unfortunately there is no FD_LIMIT in Windows. In practice, this means
that I cannot configure a large number of sessions in my application and
this is a risk for my project.

Concerning performance, I am not sure that it is so bad, especially if
compared with the databases stores. Normally, for the databases like MySQL
and PostgreSQL it is required only 4 tables for all sessions, however you
have to pay the overhead of the IPC (TCP) and SQL parsing. I rember to have
seen in this forum benchmarks that reported that FielStore were in avarage
3x faster than the implemetations that use databases.

Some months ago I started an attempt to understand the FileStore to make
some improvements and I came to the following conclusion regarding the
created files:

.session -> date and time of session creation (only 1 line)
.seqnums -> contains the incoming and outgoing sequence number (only 1 line)
.body -> a stream with all sent messages
.header -> works like an index with the offsets to recover the message in
.body for retransmission

Well, why not 4 files for all sessions instead of 4 files for each session?
I think that it would solve the scalability problem, but performance would
probably be slower, especially for multi-threaded applications that would
need to rely in a single mutex to protect the integrity of the files and it
would be certainly a bottleneck.

Actually, I prefered to invest my effort in Store implementations to use
embeded databases, like BerkeleyDB and SQLite (the development is still
ongoing).

Djalma

On 6/11/07, que...@bn... <que...@bn...>
wrote:
>
> QuickFIX Documentation:
> http://www.quickfixengine.org/quickfix/doc/html/index.html
> QuickFIX Support: http://www.quickfixengine.org/services.html
>
>
>
> Hi,
>
> I think that the way used by quickFix to store fix persistent datas into 4
>
> files is not the best solution to have the best performances possible.
>
> Indeed for each fix messages send to the counterparty, QuickFix has to
> save
> synchronously datas into 3 files (seqnum+header+messages).
>
> Can't we have a better file format that allows to save only into one file
> for each message ?
>
> Please send me your thougths about this real time issue.
>
>
> Thanks,
>
> Quentin.
>
>
>
> This message and any attachments (the "message") is
> intended solely for the addressees and is confidential.
> If you receive this message in error, please delete it and
> immediately notify the sender. Any use not in accord with
> its purpose, any dissemination or disclosure, either whole
> or partial, is prohibited except formal approval. The internet
> can not guarantee the integrity of this message.
> BNP PARIBAS (and its subsidiaries) shall (will) not
> therefore be liable for the message if modified.
>
>                 ---------------------------------------------
>
> Ce message et toutes les pieces jointes (ci-apres le
> "message") sont etablis a l'intention exclusive de ses
> destinataires et sont confidentiels. Si vous recevez ce
> message par erreur, merci de le detruire et d'en avertir
> immediatement l'expediteur. Toute utilisation de ce
> message non conforme a sa destination, toute diffusion
> ou toute publication, totale ou partielle, est interdite, sauf
> autorisation expresse. L'internet ne permettant pas
> d'assurer l'integrite de ce message, BNP PARIBAS (et ses
> filiales) decline(nt) toute responsabilite au titre de ce
> message, dans l'hypothese ou il aurait ete modifie.
>
>
> -------------------------------------------------------------------------
> This SF.net email is sponsored by DB2 Express
> Download DB2 Express C - the FREE version of DB2 express and take
> control of your XML. No limits. Just data. Click to get it now.
> http://sourceforge.net/powerbar/db2/
> _______________________________________________
> Quickfix-developers mailing list
> Qui...@li...
> https://lists.sourceforge.net/lists/listinfo/quickfix-developers
>