quickfix-developers

[Quickfix-developers] QuickFIX store and log DB down

[bennyzlm <ben...@ho...>]

Hi,

Currently I am focusing on recovery if some component in the system down.
More specificly, if I use MySQL to be the store and log destination for
quickfix, what will the quickfix engine response to the messaged received
during MySQL is down? I did some experiments and it seems quickfix throws
some SQL exception but do not response any reject message. Is there any way
in quickfix to response the other side that it is down, or reject to any
action taken by the other side? Thanks for any help!

Regards,
Benny
-- 
View this message in context: http://www.nabble.com/QuickFIX-store-and-log-DB-down-tf3950636.html#a11208236
Sent from the QuickFIX - Dev mailing list archive at Nabble.com.

Re: [Quickfix-developers] QuickFIX store and log DB down

[Djalma R. d. S. F. <drs...@gm...>]

Hi Benny,

Following is my opinion:

I don't believe that the logging unavailability should be a reason to
reject a message, although this is a problem that I would like to be
able to detect and correct it at my side asap.

Anyway, some weeks ago when I was debugging a specific application
error, I also found that the errors in underlying database are simply
hidden from the application, like for example undesired schema
modifications, connection failures etc.

I thought about an alert mechanism for monitoring, some kind of
callback function or an external file to log errors in logging class
(MySQLLog e.g.). This way, it would become easier to detect errors and
have a recovery mechanism in the application for logging and store.

Following was the issue I opened with a feature request.
http://www.quickfixengine.org/bugtracker/bug.php?op=show&bugid=255

I figure out that the problem is worse if the Store is down,
SendToTarget will return false, although the message might have been
sent.  But, with no warranty at all, since retransmission won't be
possible in this situation.

I belive that currently QF has a very optimistic aproach for the
problems we're concerned about.

Djalma

On 6/20/07, bennyzlm <ben...@ho...> wrote:
> QuickFIX Documentation: http://www.quickfixengine.org/quickfix/doc/html/index.html
> QuickFIX Support: http://www.quickfixengine.org/services.html
>
>
> Hi,
>
> Currently I am focusing on recovery if some component in the system down.
> More specificly, if I use MySQL to be the store and log destination for
> quickfix, what will the quickfix engine response to the messaged received
> during MySQL is down? I did some experiments and it seems quickfix throws
> some SQL exception but do not response any reject message. Is there any way
> in quickfix to response the other side that it is down, or reject to any
> action taken by the other side? Thanks for any help!
>
> Regards,
> Benny
> --
> View this message in context: http://www.nabble.com/QuickFIX-store-and-log-DB-down-tf3950636.html#a11208236
> Sent from the QuickFIX - Dev mailing list archive at Nabble.com.
>
>
> -------------------------------------------------------------------------
> This SF.net email is sponsored by DB2 Express
> Download DB2 Express C - the FREE version of DB2 express and take
> control of your XML. No limits. Just data. Click to get it now.
> http://sourceforge.net/powerbar/db2/
> _______________________________________________
> Quickfix-developers mailing list
> Qui...@li...
> https://lists.sourceforge.net/lists/listinfo/quickfix-developers
>

Re: [Quickfix-developers] QuickFIX store and log DB down

[Oren M. <or...@qu...>]

The repository changes this.  Now quickfix return false on send if  
the store is unavailable.

--oren

On Jun 20, 2007, at 1:51 PM, Djalma Rosa dos Santos Filho wrote:

> QuickFIX Documentation: http://www.quickfixengine.org/quickfix/doc/ 
> html/index.html
> QuickFIX Support: http://www.quickfixengine.org/services.html
>
> Hi Benny,
>
> Following is my opinion:
>
> I don't believe that the logging unavailability should be a reason to
> reject a message, although this is a problem that I would like to be
> able to detect and correct it at my side asap.
>
> Anyway, some weeks ago when I was debugging a specific application
> error, I also found that the errors in underlying database are simply
> hidden from the application, like for example undesired schema
> modifications, connection failures etc.
>
> I thought about an alert mechanism for monitoring, some kind of
> callback function or an external file to log errors in logging class
> (MySQLLog e.g.). This way, it would become easier to detect errors and
> have a recovery mechanism in the application for logging and store.
>
> Following was the issue I opened with a feature request.
> http://www.quickfixengine.org/bugtracker/bug.php?op=show&bugid=255
>
> I figure out that the problem is worse if the Store is down,
> SendToTarget will return false, although the message might have been
> sent.  But, with no warranty at all, since retransmission won't be
> possible in this situation.
>
> I belive that currently QF has a very optimistic aproach for the
> problems we're concerned about.
>
> Djalma
>
> On 6/20/07, bennyzlm <ben...@ho...> wrote:
>> QuickFIX Documentation: http://www.quickfixengine.org/quickfix/doc/ 
>> html/index.html
>> QuickFIX Support: http://www.quickfixengine.org/services.html
>>
>>
>> Hi,
>>
>> Currently I am focusing on recovery if some component in the  
>> system down.
>> More specificly, if I use MySQL to be the store and log  
>> destination for
>> quickfix, what will the quickfix engine response to the messaged  
>> received
>> during MySQL is down? I did some experiments and it seems quickfix  
>> throws
>> some SQL exception but do not response any reject message. Is  
>> there any way
>> in quickfix to response the other side that it is down, or reject  
>> to any
>> action taken by the other side? Thanks for any help!
>>
>> Regards,
>> Benny
>> --
>> View this message in context: http://www.nabble.com/QuickFIX-store- 
>> and-log-DB-down-tf3950636.html#a11208236
>> Sent from the QuickFIX - Dev mailing list archive at Nabble.com.
>>
>>
>> --------------------------------------------------------------------- 
>> ----
>> This SF.net email is sponsored by DB2 Express
>> Download DB2 Express C - the FREE version of DB2 express and take
>> control of your XML. No limits. Just data. Click to get it now.
>> http://sourceforge.net/powerbar/db2/
>> _______________________________________________
>> Quickfix-developers mailing list
>> Qui...@li...
>> https://lists.sourceforge.net/lists/listinfo/quickfix-developers
>>
>
> ---------------------------------------------------------------------- 
> ---
> This SF.net email is sponsored by DB2 Express
> Download DB2 Express C - the FREE version of DB2 express and take
> control of your XML. No limits. Just data. Click to get it now.
> http://sourceforge.net/powerbar/db2/
> _______________________________________________
> Quickfix-developers mailing list
> Qui...@li...
> https://lists.sourceforge.net/lists/listinfo/quickfix-developers
>

Re: [Quickfix-developers] QuickFIX store and log DB down

[bennyzlm <ben...@ho...>]

Thanks Djalma and Oren for help! I still have some concern:
If the store is down (not able to connect or schema change, etc), quickfix
will return false on send.
1. Will the message be sent to the other side? or no guarantee?
2. What will the QF engine response if the other side send message to QF
during the period store down?

>From my personal opinion, I don't want quickfix becomes responseless if the
store/log is down. For the first point, the application behind the engine
will know if something is wrong in quickfix and can decide what is the next
step to do. Thus would simply do not send out the message be more feasible
for the application to "rescue" (no matter stop the engine, give out warn,
something like "plug and play" to another store/log destination)? For the
second point, I will further check if the other things in QF works correctly
if the store/log down.

Regards,
Benny


Oren Miller wrote:
> 
> QuickFIX Documentation:
> http://www.quickfixengine.org/quickfix/doc/html/index.html
> QuickFIX Support: http://www.quickfixengine.org/services.html
> 
> The repository changes this.  Now quickfix return false on send if  
> the store is unavailable.
> 
> --oren
> 
> On Jun 20, 2007, at 1:51 PM, Djalma Rosa dos Santos Filho wrote:
> 
>> QuickFIX Documentation: http://www.quickfixengine.org/quickfix/doc/ 
>> html/index.html
>> QuickFIX Support: http://www.quickfixengine.org/services.html
>>
>> Hi Benny,
>>
>> Following is my opinion:
>>
>> I don't believe that the logging unavailability should be a reason to
>> reject a message, although this is a problem that I would like to be
>> able to detect and correct it at my side asap.
>>
>> Anyway, some weeks ago when I was debugging a specific application
>> error, I also found that the errors in underlying database are simply
>> hidden from the application, like for example undesired schema
>> modifications, connection failures etc.
>>
>> I thought about an alert mechanism for monitoring, some kind of
>> callback function or an external file to log errors in logging class
>> (MySQLLog e.g.). This way, it would become easier to detect errors and
>> have a recovery mechanism in the application for logging and store.
>>
>> Following was the issue I opened with a feature request.
>> http://www.quickfixengine.org/bugtracker/bug.php?op=show&bugid=255
>>
>> I figure out that the problem is worse if the Store is down,
>> SendToTarget will return false, although the message might have been
>> sent.  But, with no warranty at all, since retransmission won't be
>> possible in this situation.
>>
>> I belive that currently QF has a very optimistic aproach for the
>> problems we're concerned about.
>>
>> Djalma
>>
>> On 6/20/07, bennyzlm <ben...@ho...> wrote:
>>> QuickFIX Documentation: http://www.quickfixengine.org/quickfix/doc/ 
>>> html/index.html
>>> QuickFIX Support: http://www.quickfixengine.org/services.html
>>>
>>>
>>> Hi,
>>>
>>> Currently I am focusing on recovery if some component in the  
>>> system down.
>>> More specificly, if I use MySQL to be the store and log  
>>> destination for
>>> quickfix, what will the quickfix engine response to the messaged  
>>> received
>>> during MySQL is down? I did some experiments and it seems quickfix  
>>> throws
>>> some SQL exception but do not response any reject message. Is  
>>> there any way
>>> in quickfix to response the other side that it is down, or reject  
>>> to any
>>> action taken by the other side? Thanks for any help!
>>>
>>> Regards,
>>> Benny
>>> --
>>> View this message in context: http://www.nabble.com/QuickFIX-store- 
>>> and-log-DB-down-tf3950636.html#a11208236
>>> Sent from the QuickFIX - Dev mailing list archive at Nabble.com.
>>>
>>>
>>> --------------------------------------------------------------------- 
>>> ----
>>> This SF.net email is sponsored by DB2 Express
>>> Download DB2 Express C - the FREE version of DB2 express and take
>>> control of your XML. No limits. Just data. Click to get it now.
>>> http://sourceforge.net/powerbar/db2/
>>> _______________________________________________
>>> Quickfix-developers mailing list
>>> Qui...@li...
>>> https://lists.sourceforge.net/lists/listinfo/quickfix-developers
>>>
>>
>> ---------------------------------------------------------------------- 
>> ---
>> This SF.net email is sponsored by DB2 Express
>> Download DB2 Express C - the FREE version of DB2 express and take
>> control of your XML. No limits. Just data. Click to get it now.
>> http://sourceforge.net/powerbar/db2/
>> _______________________________________________
>> Quickfix-developers mailing list
>> Qui...@li...
>> https://lists.sourceforge.net/lists/listinfo/quickfix-developers
>>
> 
> 
> -------------------------------------------------------------------------
> This SF.net email is sponsored by DB2 Express
> Download DB2 Express C - the FREE version of DB2 express and take
> control of your XML. No limits. Just data. Click to get it now.
> http://sourceforge.net/powerbar/db2/
> _______________________________________________
> Quickfix-developers mailing list
> Qui...@li...
> https://lists.sourceforge.net/lists/listinfo/quickfix-developers
> 
> 

-- 
View this message in context: http://www.nabble.com/QuickFIX-store-and-log-DB-down-tf3950636.html#a11226570
Sent from the QuickFIX - Dev mailing list archive at Nabble.com.

Re: [Quickfix-developers] QuickFIX store and log DB down

[Djalma R. d. S. F. <drs...@gm...>]

Hi,

My opinion is that logging is not critical, its purpose is tracking and audit.
For me, it is not a really a problem If the engine is not able to log
for a short period of time, but the application can still process the
received message and sending is working. I just would like to  have
the information somewhere saying that I lost some logging during this
period of time because the database is out (that is why I would like
to have an alert mechanism provided by the storage classes).

> 2. What will the QF engine response if the other side send message to QF
> during the period store down?

-> The store is used for retransmission purposes, thus the received
messages are not persisted in the store, only the ones that are sent.

Thanks Oren, I will check the changes in repository.

Regards,
Djalma

On 6/21/07, bennyzlm <ben...@ho...> wrote:
> QuickFIX Documentation: http://www.quickfixengine.org/quickfix/doc/html/index.html
> QuickFIX Support: http://www.quickfixengine.org/services.html
>
>
> Thanks Djalma and Oren for help! I still have some concern:
> If the store is down (not able to connect or schema change, etc), quickfix
> will return false on send.
> 1. Will the message be sent to the other side? or no guarantee?
> 2. What will the QF engine response if the other side send message to QF
> during the period store down?
>
> >From my personal opinion, I don't want quickfix becomes responseless if the
> store/log is down. For the first point, the application behind the engine
> will know if something is wrong in quickfix and can decide what is the next
> step to do. Thus would simply do not send out the message be more feasible
> for the application to "rescue" (no matter stop the engine, give out warn,
> something like "plug and play" to another store/log destination)? For the
> second point, I will further check if the other things in QF works correctly
> if the store/log down.
>
> Regards,
> Benny
>
>
> Oren Miller wrote:
> >
> > QuickFIX Documentation:
> > http://www.quickfixengine.org/quickfix/doc/html/index.html
> > QuickFIX Support: http://www.quickfixengine.org/services.html
> >
> > The repository changes this.  Now quickfix return false on send if
> > the store is unavailable.
> >
> > --oren
> >
> > On Jun 20, 2007, at 1:51 PM, Djalma Rosa dos Santos Filho wrote:
> >
> >> QuickFIX Documentation: http://www.quickfixengine.org/quickfix/doc/
> >> html/index.html
> >> QuickFIX Support: http://www.quickfixengine.org/services.html
> >>
> >> Hi Benny,
> >>
> >> Following is my opinion:
> >>
> >> I don't believe that the logging unavailability should be a reason to
> >> reject a message, although this is a problem that I would like to be
> >> able to detect and correct it at my side asap.
> >>
> >> Anyway, some weeks ago when I was debugging a specific application
> >> error, I also found that the errors in underlying database are simply
> >> hidden from the application, like for example undesired schema
> >> modifications, connection failures etc.
> >>
> >> I thought about an alert mechanism for monitoring, some kind of
> >> callback function or an external file to log errors in logging class
> >> (MySQLLog e.g.). This way, it would become easier to detect errors and
> >> have a recovery mechanism in the application for logging and store.
> >>
> >> Following was the issue I opened with a feature request.
> >> http://www.quickfixengine.org/bugtracker/bug.php?op=show&bugid=255
> >>
> >> I figure out that the problem is worse if the Store is down,
> >> SendToTarget will return false, although the message might have been
> >> sent.  But, with no warranty at all, since retransmission won't be
> >> possible in this situation.
> >>
> >> I belive that currently QF has a very optimistic aproach for the
> >> problems we're concerned about.
> >>
> >> Djalma
> >>
> >> On 6/20/07, bennyzlm <ben...@ho...> wrote:
> >>> QuickFIX Documentation: http://www.quickfixengine.org/quickfix/doc/
> >>> html/index.html
> >>> QuickFIX Support: http://www.quickfixengine.org/services.html
> >>>
> >>>
> >>> Hi,
> >>>
> >>> Currently I am focusing on recovery if some component in the
> >>> system down.
> >>> More specificly, if I use MySQL to be the store and log
> >>> destination for
> >>> quickfix, what will the quickfix engine response to the messaged
> >>> received
> >>> during MySQL is down? I did some experiments and it seems quickfix
> >>> throws
> >>> some SQL exception but do not response any reject message. Is
> >>> there any way
> >>> in quickfix to response the other side that it is down, or reject
> >>> to any
> >>> action taken by the other side? Thanks for any help!
> >>>
> >>> Regards,
> >>> Benny
> >>> --
> >>> View this message in context: http://www.nabble.com/QuickFIX-store-
> >>> and-log-DB-down-tf3950636.html#a11208236
> >>> Sent from the QuickFIX - Dev mailing list archive at Nabble.com.
> >>>
> >>>
> >>> ---------------------------------------------------------------------
> >>> ----
> >>> This SF.net email is sponsored by DB2 Express
> >>> Download DB2 Express C - the FREE version of DB2 express and take
> >>> control of your XML. No limits. Just data. Click to get it now.
> >>> http://sourceforge.net/powerbar/db2/
> >>> _______________________________________________
> >>> Quickfix-developers mailing list
> >>> Qui...@li...
> >>> https://lists.sourceforge.net/lists/listinfo/quickfix-developers
> >>>
> >>
> >> ----------------------------------------------------------------------
> >> ---
> >> This SF.net email is sponsored by DB2 Express
> >> Download DB2 Express C - the FREE version of DB2 express and take
> >> control of your XML. No limits. Just data. Click to get it now.
> >> http://sourceforge.net/powerbar/db2/
> >> _______________________________________________
> >> Quickfix-developers mailing list
> >> Qui...@li...
> >> https://lists.sourceforge.net/lists/listinfo/quickfix-developers
> >>
> >
> >
> > -------------------------------------------------------------------------
> > This SF.net email is sponsored by DB2 Express
> > Download DB2 Express C - the FREE version of DB2 express and take
> > control of your XML. No limits. Just data. Click to get it now.
> > http://sourceforge.net/powerbar/db2/
> > _______________________________________________
> > Quickfix-developers mailing list
> > Qui...@li...
> > https://lists.sourceforge.net/lists/listinfo/quickfix-developers
> >
> >
>
> --
> View this message in context: http://www.nabble.com/QuickFIX-store-and-log-DB-down-tf3950636.html#a11226570
> Sent from the QuickFIX - Dev mailing list archive at Nabble.com.
>
>
> -------------------------------------------------------------------------
> This SF.net email is sponsored by DB2 Express
> Download DB2 Express C - the FREE version of DB2 express and take
> control of your XML. No limits. Just data. Click to get it now.
> http://sourceforge.net/powerbar/db2/
> _______________________________________________
> Quickfix-developers mailing list
> Qui...@li...
> https://lists.sourceforge.net/lists/listinfo/quickfix-developers
>