Re: [Quickfix-developers] FileLog, FileStore flush

[Alexey Z. <ale...@gm...>]

Oren,

Unfortunately I have no numbers and in our current computer 
configuration it's not possible to measure correctly.
We just noted that file access to the computer became slower when we 
started to use more QF applications.

I wrote a small application saving messages (FileStore::set() calls) 
with and without flushing.
There was no big difference when you run one process, but when I ran 10 
ones the situation has changed.
And as bigger the output files were the bigger were the difference.

I don't know a solution for a hardware crash. If you got it I don't 
believe you can restart you application soon.
At least my computer is protected by UPS, so I have a chance to stop my 
application before the reboot.

Ok. I'm going to try this change for non-critical logs tomorrow. Will 
see if it really helps.


Thanks for your reply.


Oren Miller wrote:

> You are not going to get better performace out of a database than the 
> flat file logs.  With a database you still need to do disk writes (and 
> the database will definately err on the side of caution and make sure 
> the buffer gets flushed with each transaction).  You also have socket 
> communication plus managing indexes on top of it.  Assuming your 
> database is on another machine, there is also network communication 
> overhead.
>  
> The performance test runner (pt.exe in the bin directory), indicates I 
> can store about 125,000 basic NewOrderSingle messages into a file on 
> my laptop.  Surely not a great harddrive.  This seems to scale up 
> pretty well (I got the pretty eqivalent numbers storing 5,000, 50,000, 
> and 500,000 messages into a single file).  I doubt you will get nearly 
> as good performance out of a relational database on any hardware.
>  
> Caleb has contributed an implementation for BerkelyDB (which has 
> unfortunately not been integrated into a release yet), but even that 
> will be slightly slower than the file logger.
>  
> If you are going to rely on the logs to analyze problems that result 
> in a crash of the application, you will probably find that disabling 
> the flush will essentially make the logs useless as it will be very 
> likely that you are missing the most recent data, particularly 
> the message that caused the crash.
>  
> You mentioned that the I/O is really heavy, but you have not mentioned 
> how this is affecting your latency.  Have you measured this?
>  
> --oren
>  
> ----- Original Message -----
>
>     *From:* Alexey Zubko <mailto:ale...@gm...>
>     *To:* Caleb Epstein <mailto:cal...@gm...>
>     *Cc:* qui...@li...
>     <mailto:qui...@li...>
>     *Sent:* Friday, October 07, 2005 1:20 PM
>     *Subject:* Re: [Quickfix-developers] FileLog, FileStore flush
>
>     Caleb,
>
>     Incoming and outgoing logs are important to control the main
>     application.
>     Plus if something unusual happened how it's possible to analyze
>     the situation without these logs?
>     I was thinking of using a database to store logs because it might
>     be faster and the disk is not as fragmented as when you write into
>     a file.
>
>     It would be nice to hear some feedbacks from guys using databases.
>     Are there any problems?
>     We have some problems of using MS SQL (not with QF) - particularly
>     it's fast growing log file.
>     I'm not sure that MySQL will be much better.
>
>
>
>     Caleb Epstein wrote:
>
>>     On 10/7/05, *Alexey Zubko* <ale...@gm...
>>     <mailto:ale...@gm...>> wrote:
>>
>>         Unfortunately speeding is the highest priority for us and
>>         hardware upgrade may not to help much.
>>
>>         I added disabling of the flushing for the FileLog class
>>         because these files are the biggest.
>>         If your are interesting in these changes I'll send them.
>>
>>
>>     If you're concerned about speed and the size of these files,
>>     don't use any log at all (you can use the ScreenLog or derive
>>     your own LogFactory to do this).   Only the MessageStore is
>>     required for operation.  The incoming/outgoing logs are only for
>>     informational purposes and aren't used by QuickFIX once they are
>>     written.
>>
>>     -- 
>>     Caleb Epstein
>>     caleb dot epstein at gmail dot com 
>
>
>-- 
>Regards,
>		Alexey Zubko
>    
>

-- 
Regards,
		Alexey Zubko