Re: [Quickfixj-users] What are the dangers of FileStoreSync=N

[Colin D. <co...@ma...>]

Unfortunately, the answer is that, if you turn sync off, you have to 
come up with a way of making sure you don't process a message twice. On 
startup, your FIX engines will sync. If you have already processed a 
message but the file store was not yet updated when a system failure 
occurred, your engine will request the same message again. You will 
process this message twice.

We don't use file stores, but we do use async message processing, which 
causes the same sorts of issues. When we process a message, we make a 
record of that message being processed in the database. Before we 
process a message, we make sure we haven't already processed it. 
Obviously, the potential for a similar problem exists - if we've 
processed a message but not marked it so in the database. We try to 
address this by processing and marking in the same database transaction. 
If you keep drilling down, though, you cannot perfectly eliminate either 
failing to deliver a message or processing a message twice - you have to 
choose one or the other as your bias. If you're redelivering messages, 
you have the option of setting the posdup flag, but, that doesn't help 
if you're required to perform some business logic yourself.

The other option is to not worry about it, which only works if you don't 
care if you process all messages (like market data or price requests).

On 2/8/19 6:53 AM, Kuhn, Simon wrote:
> QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
> QuickFIX/J Support: http://www.quickfixj.org/support/
>
>
>
> Hi
>
> Who has experience with FileStoreSync?
>
> We have learnt that FileStoreSync=Y is a good idea to avoid 
> inconsistencies after system crashes, network issues or stuff like 
> this. We have no issues with having the file-store synchronized on 
> local SSD disk, on Storage Area Network (SAN) disk however it becomes 
> a big performance issue for us. We have to use SAN disk to meet 
> Business Continuity Management (BCM) requirements. With the 
> requirements SAN disk and FileStoreSync=Y we are not able to meet the 
> performance goals (700 messages / second).
>
> It looks like we have two options:
>
> 1) FileStoreSync=N and to know how to deal with inconsistencies after 
> system issues
>
> 2) Implementing an own file-store in the hope that it is even possible 
> to achieve the performance goals
>
> We would really like to go with option 1 (FileStoreSync=N) but we are 
> not sure what problems we could face in case of system issues. What 
> are your experiences, could we end up in a state where we couldn't 
> proceed with doing business with our customers without manual 
> interventions (correcting generated QuickFixJ files) after a system 
> crash, network issues or stuff like that? What can happen with 
> FileStoreSync=N, what are the dangers?
>
> Just in case, has someone experience with implementing a file-store 
> with performance enhancements (option 2)?
>
> Greetings to the QuickFixJ community!
>
> Simon
>
> The content of this e-mail is intended only for the confidential use 
> of the person addressed.
> If you are not the intended recipient, please notify the sender and 
> delete this e-mail immediately.
> Thank you.
>
>
>
> _______________________________________________
> Quickfixj-users mailing list
> Qui...@li...
> https://lists.sourceforge.net/lists/listinfo/quickfixj-users

-- 
Colin DuPlantis
Chief Architect, Marketcetera
Download, Run, Trade
888.868.4884
https://www.marketcetera.com