Thread: [Quickfix-developers] FileLog, FileStore flush
Brought to you by:
orenmnero
Menu
▾
▴
quickfix-developers
|
From: Alexey Z. <ale...@gm...> - 2005-10-07 14:43:11
|
Hello, I have a question regarding flushing of files in the FileLog and FileStore classes (C++). We have many QF applications running on one machine (Win2k). The CPU usage is almost nothing, but I believe the disk IO is really heavy (we save everything). I want to try to eliminate these flush calls from the classes. Is it potential danger? Does anyone have such experience? Is it better or faster to use a SQL server to store logs? Thank you in advance. -- Regards, Alexey Zubko |
|
From: Caleb E. <cal...@gm...> - 2005-10-07 15:10:54
|
On 10/7/05, Alexey Zubko <ale...@gm...> wrote: > > I want to try to eliminate these flush calls from the classes. > Is it potential danger? Does anyone have such experience? > Is it better or faster to use a SQL server to store logs? It is definitely a potential danger. Any data that hasn't been flushed to disk (notably the .seqnums and also perhaps the .outgoing file) won't be written to disk if your process dies unexpectedly or the machine reboots. When you recover, your application may end up re-processing messages it has already received. OIr perhaps even worse, unable to resend messages that th= e counterparty has not received. This is a classic safety vs. speed tradeoff. The right answer depends on th= e requirements of your own system, and the default implementation of these classes in QuickFIX takes the safest possible (but also slowest) approach t= o guarantee recoverability. -- Caleb Epstein caleb dot epstein at gmail dot com |
|
From: Alexey Z. <ale...@gm...> - 2005-10-07 15:32:36
|
Caleb, Thank you for the reply. Unfortunately speeding is the highest priority for us and hardware upgrade may not to help much. I added disabling of the flushing for the FileLog class because these files are the biggest. If your are interesting in these changes I'll send them. -- Regards, Alexey Zubko Caleb Epstein wrote: > On 10/7/05, *Alexey Zubko* <ale...@gm... > <mailto:ale...@gm...>> wrote: > > I want to try to eliminate these flush calls from the classes. > Is it potential danger? Does anyone have such experience? > Is it better or faster to use a SQL server to store logs? > > > It is definitely a potential danger. Any data that hasn't been > flushed to disk (notably the .seqnums and also perhaps the .outgoing > file) won't be written to disk if your process dies unexpectedly or > the machine reboots. When you recover, your application may end up > re-processing messages it has already received. OIr perhaps even > worse, unable to resend messages that the counterparty has not received. > > This is a classic safety vs. speed tradeoff. The right answer depends > on the requirements of your own system, and the default implementation > of these classes in QuickFIX takes the safest possible (but also > slowest) approach to guarantee recoverability. > > -- > Caleb Epstein > caleb dot epstein at gmail dot com |
|
From: Caleb E. <cal...@gm...> - 2005-10-07 17:46:11
|
On 10/7/05, Alexey Zubko <ale...@gm...> wrote: > Unfortunately speeding is the highest priority for us and hardware upgrad= e > may not to help much. > > I added disabling of the flushing for the FileLog class because these > files are the biggest. > If your are interesting in these changes I'll send them. If you're concerned about speed and the size of these files, don't use any log at all (you can use the ScreenLog or derive your own LogFactory to do this). Only the MessageStore is required for operation. The incoming/outgoing logs are only for informational purposes and aren't used by QuickFIX once they are written. -- Caleb Epstein caleb dot epstein at gmail dot com |
|
From: Alexey Z. <ale...@gm...> - 2005-10-07 18:21:05
|
Caleb, Incoming and outgoing logs are important to control the main application. Plus if something unusual happened how it's possible to analyze the situation without these logs? I was thinking of using a database to store logs because it might be faster and the disk is not as fragmented as when you write into a file. It would be nice to hear some feedbacks from guys using databases. Are there any problems? We have some problems of using MS SQL (not with QF) - particularly it's fast growing log file. I'm not sure that MySQL will be much better. Caleb Epstein wrote: > On 10/7/05, *Alexey Zubko* <ale...@gm... > <mailto:ale...@gm...>> wrote: > > Unfortunately speeding is the highest priority for us and hardware > upgrade may not to help much. > > I added disabling of the flushing for the FileLog class because > these files are the biggest. > If your are interesting in these changes I'll send them. > > > If you're concerned about speed and the size of these files, don't use > any log at all (you can use the ScreenLog or derive your own > LogFactory to do this). Only the MessageStore is required for > operation. The incoming/outgoing logs are only for informational > purposes and aren't used by QuickFIX once they are written. > > -- > Caleb Epstein > caleb dot epstein at gmail dot com -- Regards, Alexey Zubko |
|
From: Oren M. <or...@qu...> - 2005-10-07 19:05:41
|
You are not going to get better performace out of a database than the =
flat file logs. With a database you still need to do disk writes (and =
the database will definately err on the side of caution and make sure =
the buffer gets flushed with each transaction). You also have socket =
communication plus managing indexes on top of it. Assuming your =
database is on another machine, there is also network communication =
overhead.
The performance test runner (pt.exe in the bin directory), indicates I =
can store about 125,000 basic NewOrderSingle messages into a file on my =
laptop. Surely not a great harddrive. This seems to scale up pretty =
well (I got the pretty eqivalent numbers storing 5,000, 50,000, and =
500,000 messages into a single file). I doubt you will get nearly as =
good performance out of a relational database on any hardware.
Caleb has contributed an implementation for BerkelyDB (which has =
unfortunately not been integrated into a release yet), but even that =
will be slightly slower than the file logger.
If you are going to rely on the logs to analyze problems that result in =
a crash of the application, you will probably find that disabling the =
flush will essentially make the logs useless as it will be very likely =
that you are missing the most recent data, particularly the message that =
caused the crash.
You mentioned that the I/O is really heavy, but you have not mentioned =
how this is affecting your latency. Have you measured this?
--oren
----- Original Message -----=20
From: Alexey Zubko=20
To: Caleb Epstein=20
Cc: qui...@li...=20
Sent: Friday, October 07, 2005 1:20 PM
Subject: Re: [Quickfix-developers] FileLog, FileStore flush
Caleb,
Incoming and outgoing logs are important to control the main =
application.
Plus if something unusual happened how it's possible to analyze the =
situation without these logs?
I was thinking of using a database to store logs because it might be =
faster and the disk is not as fragmented as when you write into a file.
It would be nice to hear some feedbacks from guys using databases.
Are there any problems?
We have some problems of using MS SQL (not with QF) - particularly =
it's fast growing log file.
I'm not sure that MySQL will be much better.
Caleb Epstein wrote:=20
On 10/7/05, Alexey Zubko <ale...@gm...> wrote:
Unfortunately speeding is the highest priority for us and hardware =
upgrade may not to help much.
I added disabling of the flushing for the FileLog class because =
these files are the biggest.
If your are interesting in these changes I'll send them.
If you're concerned about speed and the size of these files, don't =
use any log at all (you can use the ScreenLog or derive your own =
LogFactory to do this). Only the MessageStore is required for =
operation. The incoming/outgoing logs are only for informational =
purposes and aren't used by QuickFIX once they are written.
--=20
Caleb Epstein
caleb dot epstein at gmail dot com=20
--=20
Regards,
Alexey Zubko
|
|
From: Alexey Z. <ale...@gm...> - 2005-10-07 19:53:41
|
Oren, Unfortunately I have no numbers and in our current computer configuration it's not possible to measure correctly. We just noted that file access to the computer became slower when we started to use more QF applications. I wrote a small application saving messages (FileStore::set() calls) with and without flushing. There was no big difference when you run one process, but when I ran 10 ones the situation has changed. And as bigger the output files were the bigger were the difference. I don't know a solution for a hardware crash. If you got it I don't believe you can restart you application soon. At least my computer is protected by UPS, so I have a chance to stop my application before the reboot. Ok. I'm going to try this change for non-critical logs tomorrow. Will see if it really helps. Thanks for your reply. Oren Miller wrote: > You are not going to get better performace out of a database than the > flat file logs. With a database you still need to do disk writes (and > the database will definately err on the side of caution and make sure > the buffer gets flushed with each transaction). You also have socket > communication plus managing indexes on top of it. Assuming your > database is on another machine, there is also network communication > overhead. > > The performance test runner (pt.exe in the bin directory), indicates I > can store about 125,000 basic NewOrderSingle messages into a file on > my laptop. Surely not a great harddrive. This seems to scale up > pretty well (I got the pretty eqivalent numbers storing 5,000, 50,000, > and 500,000 messages into a single file). I doubt you will get nearly > as good performance out of a relational database on any hardware. > > Caleb has contributed an implementation for BerkelyDB (which has > unfortunately not been integrated into a release yet), but even that > will be slightly slower than the file logger. > > If you are going to rely on the logs to analyze problems that result > in a crash of the application, you will probably find that disabling > the flush will essentially make the logs useless as it will be very > likely that you are missing the most recent data, particularly > the message that caused the crash. > > You mentioned that the I/O is really heavy, but you have not mentioned > how this is affecting your latency. Have you measured this? > > --oren > > ----- Original Message ----- > > *From:* Alexey Zubko <mailto:ale...@gm...> > *To:* Caleb Epstein <mailto:cal...@gm...> > *Cc:* qui...@li... > <mailto:qui...@li...> > *Sent:* Friday, October 07, 2005 1:20 PM > *Subject:* Re: [Quickfix-developers] FileLog, FileStore flush > > Caleb, > > Incoming and outgoing logs are important to control the main > application. > Plus if something unusual happened how it's possible to analyze > the situation without these logs? > I was thinking of using a database to store logs because it might > be faster and the disk is not as fragmented as when you write into > a file. > > It would be nice to hear some feedbacks from guys using databases. > Are there any problems? > We have some problems of using MS SQL (not with QF) - particularly > it's fast growing log file. > I'm not sure that MySQL will be much better. > > > > Caleb Epstein wrote: > >> On 10/7/05, *Alexey Zubko* <ale...@gm... >> <mailto:ale...@gm...>> wrote: >> >> Unfortunately speeding is the highest priority for us and >> hardware upgrade may not to help much. >> >> I added disabling of the flushing for the FileLog class >> because these files are the biggest. >> If your are interesting in these changes I'll send them. >> >> >> If you're concerned about speed and the size of these files, >> don't use any log at all (you can use the ScreenLog or derive >> your own LogFactory to do this). Only the MessageStore is >> required for operation. The incoming/outgoing logs are only for >> informational purposes and aren't used by QuickFIX once they are >> written. >> >> -- >> Caleb Epstein >> caleb dot epstein at gmail dot com > > >-- >Regards, > Alexey Zubko > > -- Regards, Alexey Zubko |
