Re: [Quickfix-developers] FileLog, FileStore flush

[Oren M. <or...@qu...>]

You are not going to get better performace out of a database than the =
flat file logs.  With a database you still need to do disk writes (and =
the database will definately err on the side of caution and make sure =
the buffer gets flushed with each transaction).  You also have socket =
communication plus managing indexes on top of it.  Assuming your =
database is on another machine, there is also network communication =
overhead.

The performance test runner (pt.exe in the bin directory), indicates I =
can store about 125,000 basic NewOrderSingle messages into a file on my =
laptop.  Surely not a great harddrive.  This seems to scale up pretty =
well (I got the pretty eqivalent numbers storing 5,000, 50,000, and =
500,000 messages into a single file).  I doubt you will get nearly as =
good performance out of a relational database on any hardware.

Caleb has contributed an implementation for BerkelyDB (which has =
unfortunately not been integrated into a release yet), but even that =
will be slightly slower than the file logger.

If you are going to rely on the logs to analyze problems that result in =
a crash of the application, you will probably find that disabling the =
flush will essentially make the logs useless as it will be very likely =
that you are missing the most recent data, particularly the message that =
caused the crash.

You mentioned that the I/O is really heavy, but you have not mentioned =
how this is affecting your latency.  Have you measured this?

--oren

----- Original Message -----=20
  From: Alexey Zubko=20
  To: Caleb Epstein=20
  Cc: qui...@li...=20
  Sent: Friday, October 07, 2005 1:20 PM
  Subject: Re: [Quickfix-developers] FileLog, FileStore flush


  Caleb,

  Incoming and outgoing logs are important to control the main =
application.
  Plus if something unusual happened how it's possible to analyze the =
situation without these logs?
  I was thinking of using a database to store logs because it might be =
faster and the disk is not as fragmented as when you write into a file.

  It would be nice to hear some feedbacks from guys using databases.
  Are there any problems?
  We have some problems of using MS SQL (not with QF) - particularly =
it's fast growing log file.
  I'm not sure that MySQL will be much better.



  Caleb Epstein wrote:=20
    On 10/7/05, Alexey Zubko <ale...@gm...> wrote:

      Unfortunately speeding is the highest priority for us and hardware =
upgrade may not to help much.

      I added disabling of the flushing for the FileLog class because =
these files are the biggest.
      If your are interesting in these changes I'll send them.

    If you're concerned about speed and the size of these files, don't =
use any log at all (you can use the ScreenLog or derive your own =
LogFactory to do this).   Only the MessageStore is required for =
operation.  The incoming/outgoing logs are only for informational =
purposes and aren't used by QuickFIX once they are written.


    --=20
    Caleb Epstein
    caleb dot epstein at gmail dot com=20


--=20
Regards,
		Alexey Zubko