Re: [Quickfix-developers] FileStore issues

[Nikhil B. <ass...@ya...>]

Hi John,
 
I was looking at the Berkeley DB documentation from Sleepy Cat.  Looks great.  What I couldn't figure out is where they persist the database on secondary storage.  They don't seem to use an underlying database, so where is the data persisted?  Do they use a file or have their own database implementation that uses low-level disk IO?
 
Nikhil

John Muehlhausen <jg...@jg...> wrote:
Kenny,

I think someone should implement a Berkeley DB store and use the transaction support. Want to volunteer?

www.sleepycat.com

-John

-----Original Message-----
From: qui...@li... 
To: qui...@li... 
Sent: Mon Jan 12 22:17:38 2004
Subject: [Quickfix-developers] FileStore issues

Hi,
I've noticed a few potential issues with the FileStore::set method.
Namely, that in the event of a write error, the files may be left in a bad
state leading to corruption.

The the method looks like:
if ( fseek( m_msgFile, 0, SEEK_END ) ) throw IOException();
if ( fseek( m_headerFile, 0, SEEK_END ) ) throw IOException();

int offset = ftell( m_msgFile );
if ( offset < 0 ) throw IOException();
int size = msg.size();

if ( fprintf( m_headerFile, "%d,%d,%d ", msgSeqNum, offset, size ) < 0 )
throw IOException();
m_offsets[ msgSeqNum ] = std::make_pair( offset, size );
fwrite( msg.c_str(), sizeof( char ), msg.size(), m_msgFile );
if ( ferror( m_msgFile ) ) throw IOException();
if ( fflush( m_msgFile ) == EOF ) throw IOException();
if ( fflush( m_headerFile ) == EOF ) throw IOException();
return true;

So, first off, a file may not be more then 2GB, otherwise the ftell may return
a negative number. No sweat, I'm not dealing with such large things yet.

The real problems come in the last part where the actual writing is done.
If the fprintf fails by running out of space and doing a partial write, nothing
is cleaned up. Since a seek to end of file is always done, any future writes
will also be affected by the superfluous characters.
Same for the fwrite - no cleanup is done on error, thereby leaving the file
in a un-usable state. Not only this, but also note that the header info is not
rolled-back in the event of failure to persist the message file - leading to
corrupt state on recovery.

Having said this, I'm not sure I know the correct fix within the restrictions
of ANSI C.
If this were a POSIX world, I'd use little guard objects to roll back
(ftruncate) the header and message files in the event of an error.

Is there any interest in a FileStore-compatible POSIX-only MessageStore?

thanks,
-Kenny

__________________________________
Do you Yahoo!?
Yahoo! Hotjobs: Enter the "Signing Bonus" Sweepstakes
http://hotjobs.sweepstakes.yahoo.com/signingbonus


-------------------------------------------------------
The SF.Net email is sponsored by EclipseCon 2004
Premiere Conference on Open Tools Development and Integration
See the breadth of Eclipse activity. February 3-5 in Anaheim, CA.
http://www.eclipsecon.org/osdn
_______________________________________________
Quickfix-developers mailing list
Qui...@li...
https://lists.sourceforge.net/lists/listinfo/quickfix-developers



-------------------------------------------------------
The SF.Net email is sponsored by EclipseCon 2004
Premiere Conference on Open Tools Development and Integration
See the breadth of Eclipse activity. February 3-5 in Anaheim, CA.
http://www.eclipsecon.org/osdn
_______________________________________________
Quickfix-developers mailing list
Qui...@li...
https://lists.sourceforge.net/lists/listinfo/quickfix-developers

---------------------------------
Do you Yahoo!?
Yahoo! Hotjobs: Enter the "Signing Bonus" Sweepstakes