quickfix-developers — QuickFIX developers forum

Re: [Quickfix-developers] QuickFix on VS2015

[<daw...@ya...>]

Hi Frank,

This is gold dust.. thanks so much! I had a horrible feeling I'd need to roll back the VS version but your fix worked a charm and saved me a lot of time and stress.

Thanks Again!
Dermot


----- Original Message -----
>From: K. Frank <kfr...@gm...>
>To: "qui...@li..." <qui...@li...> 
>Date: 2017/7/2, Sun 01:10
>Subject: Re: [Quickfix-developers] QuickFix on VS2015
> 
>QuickFIX Documentation: http://www.quickfixengine.org/quickfix/doc/html/
>
>Hello Dermot!
>
>I think you may be running into a known (sem-) bug.  See comments,
>in line, below.
>
>On Sat, Jul 1, 2017 at 3:33 AM,  <dra...@ac...> wrote:
>>
>> Hi All, Can someone please tell me the position on QuickFix on VS2015. Is it supported?
>
>I cannot speak as to whether QuickFIX is supported on VS2015.
>
>> I upgraded the VS2012 solution file to VS2015 and got it to compile by changing one line of code in Utility.cpp, line 399:-
>
>I think that there is an outright bug in the windows version of QuickFIX,
>that, for whatever reason, was tolerated by some versions of VS, perhaps
>VS2012.  Since I believe that it's a bug, then I wouldn't be surprised to
>find out that it is no longer tolerated by newer versions of VS, for example,
>VS2015.
>
>> OLD: result = _beginthreadex( NULL, 0, &func, var, 0, &id );
>> NEW: result = _beginthreadex( NULL, 0, (unsigned(__stdcall*)(void*)) &func, var, 0, &id );
>
>Speaking from memory, "func" in an actual pointer-to-function (not the name
>of a function which would then "decay" to pointer-to-function).  So &func is a
>pointer-to-pointer-to-function and is therefore the wrong type for the third
>argument of _beginthreadex.  (The third argument should be pointer-to-function.)
>
>If I am right that your cast is a microsoftian version of
>pointer-to-function, then
>your cast is completely bogus:  You pass in the address of a location in memory
>that holds the address of a function, but you lie to the compiler, and
>tell it that
>you are passing in the address of a function.
>
>> Then compiled and ran the tradeclient example. Creating the initiator object took about 30 seconds (it was pretty instantaneous on VS2010) and then it became unstable after the call to initiator.start() - it crashed on the subsequent call to std::cout in application.run().
>
>I'm not at all surprised that you get a crash.  I imagine that your
>spawned thread
>tries to run "random data" on the stack or heap somewhere as if it
>were a function,
>so you get "undefined behavior" / crash /.corruption.
>
>> Any ideas?
>
>I believe that the correct fix would be to remove the "&" from "&func", that is,
>instead of using your cast, replace
>
>   result = _beginthreadex( NULL, 0, &func, var, 0, &id );
>
>with
>
>   result = _beginthreadex( NULL, 0, func, var, 0, &id );
>
>This worked for me some years ago with an older version of QuickFIX using
>mingw_w64 instead of VS.
>
>There are two old threads about this issue:
>
>  https://sourceforge.net/p/quickfix/mailman/quickfix-developers/thread/CANL%3DBToQ%2BwdfkL-XwBO-pX8Wb%3DBFpmxxPD80kWNH50d7tF0SOg%40mail.gmail.com/#msg30386701
>
>  https://sourceforge.net/p/quickfix/mailman/message/35810081/
>
>The short story is that I think "&func" is a bug, my fix worked for me
>using mingw_w64,
>my fix seems to have worked for someone using "visual studio community
>2015", the
>bug version ("&func") worked for Mike Gatny on a 32-bit VS version
>back in 2013, the
>bug is not present in the linux version of QuickFIX, and I've never
>understood why the
>"&func" version ever compiled on windows (although I don't doubt that it did).
>
>Unless I'm completely confused about what seems to be straightforward
>c++, this really
>is a bug in the windows version of QuickFIX and it ought to be fixed
>in the QuickFIX source.
>
>> Thanks,
>> Dermot
>
>
>Happy Hacking!
>
>
>K. Frank
>
>------------------------------------------------------------------------------
>Check out the vibrant tech community on one of the world's most
>engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>_______________________________________________
>Quickfix-developers mailing list
>Qui...@li...
>https://lists.sourceforge.net/lists/listinfo/quickfix-developers
>
>
>

Re: [Quickfix-developers] QuickFix on VS2015

[K. F. <kfr...@gm...>]

Hello Dermot!

I think you may be running into a known (sem-) bug.  See comments,
in line, below.

On Sat, Jul 1, 2017 at 3:33 AM,  <dra...@ac...> wrote:
>
> Hi All, Can someone please tell me the position on QuickFix on VS2015. Is it supported?

I cannot speak as to whether QuickFIX is supported on VS2015.

> I upgraded the VS2012 solution file to VS2015 and got it to compile by changing one line of code in Utility.cpp, line 399:-

I think that there is an outright bug in the windows version of QuickFIX,
that, for whatever reason, was tolerated by some versions of VS, perhaps
VS2012.  Since I believe that it's a bug, then I wouldn't be surprised to
find out that it is no longer tolerated by newer versions of VS, for example,
VS2015.

> OLD: result = _beginthreadex( NULL, 0, &func, var, 0, &id );
> NEW: result = _beginthreadex( NULL, 0, (unsigned(__stdcall*)(void*)) &func, var, 0, &id );

Speaking from memory, "func" in an actual pointer-to-function (not the name
of a function which would then "decay" to pointer-to-function).  So &func is a
pointer-to-pointer-to-function and is therefore the wrong type for the third
argument of _beginthreadex.  (The third argument should be pointer-to-function.)

If I am right that your cast is a microsoftian version of
pointer-to-function, then
your cast is completely bogus:  You pass in the address of a location in memory
that holds the address of a function, but you lie to the compiler, and
tell it that
you are passing in the address of a function.

> Then compiled and ran the tradeclient example. Creating the initiator object took about 30 seconds (it was pretty instantaneous on VS2010) and then it became unstable after the call to initiator.start() - it crashed on the subsequent call to std::cout in application.run().

I'm not at all surprised that you get a crash.  I imagine that your
spawned thread
tries to run "random data" on the stack or heap somewhere as if it
were a function,
so you get "undefined behavior" / crash /.corruption.

> Any ideas?

I believe that the correct fix would be to remove the "&" from "&func", that is,
instead of using your cast, replace

   result = _beginthreadex( NULL, 0, &func, var, 0, &id );

with

   result = _beginthreadex( NULL, 0, func, var, 0, &id );

This worked for me some years ago with an older version of QuickFIX using
mingw_w64 instead of VS.

There are two old threads about this issue:

   https://sourceforge.net/p/quickfix/mailman/quickfix-developers/thread/CANL%3DBToQ%2BwdfkL-XwBO-pX8Wb%3DBFpmxxPD80kWNH50d7tF0SOg%40mail.gmail.com/#msg30386701

   https://sourceforge.net/p/quickfix/mailman/message/35810081/

The short story is that I think "&func" is a bug, my fix worked for me
using mingw_w64,
my fix seems to have worked for someone using "visual studio community
2015", the
bug version ("&func") worked for Mike Gatny on a 32-bit VS version
back in 2013, the
bug is not present in the linux version of QuickFIX, and I've never
understood why the
"&func" version ever compiled on windows (although I don't doubt that it did).

Unless I'm completely confused about what seems to be straightforward
c++, this really
is a bug in the windows version of QuickFIX and it ought to be fixed
in the QuickFIX source.

> Thanks,
> Dermot


Happy Hacking!


K. Frank

[Quickfix-developers] QuickFix on VS2015

[<daw...@ya...>]

Hi All, Can someone please tell me the position on QuickFix on VS2015. Is it supported?

I upgraded the VS2012 solution file to VS2015 and got it to compile by changing one line of code in Utility.cpp, line 399:-

OLD: result = _beginthreadex( NULL, 0, &func, var, 0, &id );
NEW: result = _beginthreadex( NULL, 0, (unsigned(__stdcall*)(void*)) &func, var, 0, &id );

Then compiled and ran the tradeclient example. Creating the initiator object took about 30 seconds (it was pretty instantaneous on VS2010) and then it became unstable after the call to initiator.start() - it crashed on the subsequent call to std::cout in application.run().

Any ideas?

Thanks,
Dermot

Re: [Quickfix-developers] Fw: Disconnecting Due To First message not login

[<daw...@ya...>]

Ha.. Sure understood Chris. Thanks!


----- Original Message -----
>From: Christoph John <chr...@ma...>
>To: daw...@ya...; "qui...@li..." <qui...@li...> 
>Date: 2017/6/23, Fri 21:15
>Subject: Re: [Quickfix-developers] Fw: Disconnecting Due To First message not login
> 
>
>Hi Dermot,
>
>I did not look at the spreadsheet but only read what you were writing in the mail. :)
>
>
>If the broker still things the connection is fine back in row 43 then wouldn't they reject a new connection request (row 40)?Between row 43 and 40 are 13 seconds. I assume the broker considered the connection broken somewhere during that time frame. But only guessing, though. ;)
>
>Cheers,
>Chris.
>
>
>
>
>On 23/06/17 10:44, daw...@ya...wrote:
>
>Hi Chris,
>>
>>
>>Thanks a lot for getting back. So I just wanna look at your response in relation to the log spreadsheet (attaching again).
>>
>>
>>[sends a test request] - You're taking about row 43 here?  
>>
>>
>>[other side thinks the connection is stillup]  - That would make sense because there is no mention of a disconnect in the broker's logs (Yellow). 
>>
>>
>>[while your side considers the connection broken and sends a new Logonwhich the other side of course ignores since the session is still working.] - the thing here is that the broker doesn't seem to be ignoring it because they send a login response (row 36). And this new connection seems to be working fine until right after heartbeat exchange (rows 13,14). If the broker still things the connection is fine back in row 43 then wouldn't they reject a new connection request (row 40)? 
>>
>>
>>Help much appreciated!
>>
>>
>>Thanks,
>>Dermot
>>
>>
>>
>>
>>
>>
>>
>>
>>----- Original Message -----
>>>From:Christoph John via Quickfix-developers <qui...@li...>
>>>To: "qui...@li..." <qui...@li...> 
>>>Date:2017/6/20, Tue 19:03
>>>Subject:Re: [Quickfix-developers] Fw: Disconnecting Due To First message not login
>>> 
>>>QuickFIX Documentation: http://www.quickfixengine.org/quickfix/doc/html/
>>>
>>>Looks to me like a timeout on the network level. Then the other side thinks the connection is still 
>>>up and sends a test request while your side considers the connection broken and sends a new Logon 
>>>which the other side of course ignores since the session is still working.
>>>
>>>Chris.
>>>
>>>
>>>On 14/06/17 09:21, daw...@ya...wrote:
>>>> QuickFIX Documentation: http://www.quickfixengine.org/quickfix/doc/html/
>>>>
>>>>
>>>> Hi Guys, Below is one I posted a month ago. Understand you're probably very busy but if you could 
>>>> take a look I'd appreciate it. You may have seen something similar before.
>>>>
>>>> Thanks!
>>>> Dermot
>>>>
>>>>    ----- Forwarded Message -----
>>>>    *From:* "daw...@ya..." <daw...@ya...>
>>>>    *To:* "qui...@li..." <qui...@li...>
>>>>    *Date:* 2017/5/11, Thu 04:17
>>>>    *Subject:* [Quickfix-developers] Disconnecting Due To First message not login
>>>>
>>>>    QuickFIX Documentation: http://www.quickfixengine.org/quickfix/doc/html/
>>>>
>>>>
>>>>    Hi Guys,
>>>>
>>>>    I've got a strange one here I'm hoping you might be able to help with. I've been doing some
>>>>    testing with my broker, fail scenarios, specifically a "dirty disconnect" [disabling the LAN
>>>>    connection] and then letting QuickFix handle the reconnect. It reconnects correctly, exchanges
>>>>    heartbeats and then disconnects due to test request timeout. The broker tells me it's a
>>>>    problem on my side because "first message is not LOGIN". He also says I'm connecting twice
>>>>    (which I can't see is possible).
>>>>
>>>>    I've put together the attached file which shows what's happening. Blue rows are my outgoing
>>>>    FIX messages, orange rows are the broker's incoming FIX messages, white are quickfix events
>>>>    and the yellow rows are from the broker's log which he has sent me and I've spliced in there.
>>>>    Could you please take a look? Start from the bottom - this is where I disconnected,
>>>>
>>>>    [Disconnecting Connection ID - 0 Due To First message not login] seems like a strange log
>>>>    entry given the session has already exchanged sequence requests and heartbeats. One thing that
>>>>    may be related.. the broker sends a TEST REQUEST at 14:58:30.341 which my side doesn't seem to
>>>>    reply to. Please let me know if you have any ideas.
>>>>
>>>>    Thanks!
>>>>    Dermot
>>>>
>>>>   
------------------------------------------------------------------------------
>>>>    Check out the vibrant tech community on one of the world's most
>>>>    engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>>>>    _______________________________________________
>>>>    Quickfix-developers mailing list
>>>>    Qui...@li...<mailto:Qui...@li...>
>>>>    https://lists.sourceforge.net/lists/listinfo/quickfix-developers
>>>>
>>>>
>>>>
>>>
>>>-- 
>>>Christoph John
>>>Development & Support
>>>Direct: +49 241 557080-28
>>>Mailto:Chr...@ma...
>>>    
>>>
>>>
>>>http://www.macd.com<http://www.macd.com/>
>>>----------------------------------------------------------------------------------------------------
>>>    
>>>----------------------------------------------------------------------------------------------------
>>>MACD GmbH
>>>Oppenhoffallee 103
>>>D-52066 Aachen
>>>Tel: +49 241 557080-0 | Fax: +49 241 557080-10
>>>    Amtsgericht Aachen: HRB 8151
>>>Ust.-Id: DE 813021663
>>>
>>>Geschäftsführer: George Macdonald
>>>----------------------------------------------------------------------------------------------------
>>>    
>>>----------------------------------------------------------------------------------------------------
>>>
>>>take care of the environment - print only if necessary
>>>
>>>------------------------------------------------------------------------------
>>>Check out the vibrant tech community on one of the world's most
>>>engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>>>_______________________________________________
>>>Quickfix-developers mailing list
>>>Qui...@li...
>>>https://lists.sourceforge.net/lists/listinfo/quickfix-developers
>>>
>>>
>>>
>
>-- 
> 
>Christoph John
>Development & Support
>Direct: +49 241 557080-28
>Mailto:Chr...@ma...
> 
>
>
>http://www.macd.com
> 
>
>________________________________
> 
>________________________________
> 
>MACD GmbH
>Oppenhoffallee 103
>D-52066 Aachen
>Tel: +49 241 557080-0 | Fax: +49 241 557080-10
>  Aachen District Court: HRB 8151 
>VAT-ID: DE 813 021 663 
>
>Managing Director: George Macdonald
> 
>
>________________________________
> 
>________________________________
> 
>take care of the environment - print only if necessary
>
>
>

Re: [Quickfix-developers] Fw: Disconnecting Due To First message not login

[Christoph J. <chr...@ma...>]

Hi Dermot,

I did not look at the spreadsheet but only read what you were writing in the mail. :)

> If the broker still things the connection is fine back in row 43 then wouldn't they reject a new 
> connection request (row 40)?
Between row 43 and 40 are 13 seconds. I assume the broker considered the connection broken somewhere 
during that time frame. But only guessing, though. ;)

Cheers,
Chris.



On 23/06/17 10:44, daw...@ya... wrote:
> Hi Chris,
>
> Thanks a lot for getting back. So I just wanna look at your response in relation to the log 
> spreadsheet (attaching again).
>
> [sends a test request] - You're taking about row 43 here?
>
> [other side thinks the connection is stillup]  - That would make sense because there is no mention 
> of a disconnect in the broker's logs (Yellow).
>
> [while your side considers the connection broken and sends a new Logonwhich the other side of 
> course ignores since the session is still working.] - the thing here is that the broker doesn't 
> seem to be ignoring it because they send a login response (row 36). And this new connection seems 
> to be working fine until right after heartbeat exchange (rows 13,14). If the broker still things 
> the connection is fine back in row 43 then wouldn't they reject a new connection request (row 40)?
>
> Help much appreciated!
>
> Thanks,
> Dermot
>
>
>
>
>     ----- Original Message -----
>     *From:* Christoph John via Quickfix-developers <qui...@li...>
>     *To:* "qui...@li..." <qui...@li...>
>     *Date:* 2017/6/20, Tue 19:03
>     *Subject:* Re: [Quickfix-developers] Fw: Disconnecting Due To First message not login
>
>     QuickFIX Documentation: http://www.quickfixengine.org/quickfix/doc/html/
>
>     Looks to me like a timeout on the network level. Then the other side thinks the connection is
>     still
>     up and sends a test request while your side considers the connection broken and sends a new Logon
>     which the other side of course ignores since the session is still working.
>
>     Chris.
>
>
>     On 14/06/17 09:21, daw...@ya... <mailto:daw...@ya...>wrote:
>     > QuickFIX Documentation: http://www.quickfixengine.org/quickfix/doc/html/
>     >
>     >
>     > Hi Guys, Below is one I posted a month ago. Understand you're probably very busy but if you
>     could
>     > take a look I'd appreciate it. You may have seen something similar before.
>     >
>     > Thanks!
>     > Dermot
>     >
>     >    ----- Forwarded Message -----
>     >    *From:* "daw...@ya... <mailto:daw...@ya...>"
>     <daw...@ya... <mailto:daw...@ya...>>
>     >    *To:* "qui...@li...
>     <mailto:qui...@li...>" <qui...@li...
>     <mailto:qui...@li...>>
>     >    *Date:* 2017/5/11, Thu 04:17
>     >    *Subject:* [Quickfix-developers] Disconnecting Due To First message not login
>     >
>     >    QuickFIX Documentation: http://www.quickfixengine.org/quickfix/doc/html/
>     >
>     >
>     >    Hi Guys,
>     >
>     >    I've got a strange one here I'm hoping you might be able to help with. I've been doing some
>     >    testing with my broker, fail scenarios, specifically a "dirty disconnect" [disabling the LAN
>     >    connection] and then letting QuickFix handle the reconnect. It reconnects correctly,
>     exchanges
>     >    heartbeats and then disconnects due to test request timeout. The broker tells me it's a
>     >    problem on my side because "first message is not LOGIN". He also says I'm connecting twice
>     >    (which I can't see is possible).
>     >
>     >    I've put together the attached file which shows what's happening. Blue rows are my outgoing
>     >    FIX messages, orange rows are the broker's incoming FIX messages, white are quickfix events
>     >    and the yellow rows are from the broker's log which he has sent me and I've spliced in there.
>     >    Could you please take a look? Start from the bottom - this is where I disconnected,
>     >
>     >    [Disconnecting Connection ID - 0 Due To First message not login] seems like a strange log
>     >    entry given the session has already exchanged sequence requests and heartbeats. One thing
>     that
>     >    may be related.. the broker sends a TEST REQUEST at 14:58:30.341 which my side doesn't
>     seem to
>     >    reply to. Please let me know if you have any ideas.
>     >
>     >    Thanks!
>     >    Dermot
>     >
>     > ------------------------------------------------------------------------------
>     >    Check out the vibrant tech community on one of the world's most
>     >    engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>     > _______________________________________________
>     >    Quickfix-developers mailing list
>     > Qui...@li...
>     <mailto:Qui...@li...><mailto:Qui...@li...
>     <mailto:Qui...@li...>>
>     > https://lists.sourceforge.net/lists/listinfo/quickfix-developers
>     >
>     >
>     >
>
>     -- 
>     Christoph John
>     Development & Support
>     Direct: +49 241 557080-28
>     Mailto:Chr...@ma... <mailto:Chr...@ma...>
>
>
>
>     http://www.macd.com <http://www.macd.com/><http://www.macd.com/>
>     ----------------------------------------------------------------------------------------------------
>
>     ----------------------------------------------------------------------------------------------------
>     MACD GmbH
>     Oppenhoffallee 103
>     D-52066 Aachen
>     Tel: +49 241 557080-0 | Fax: +49 241 557080-10
>         Amtsgericht Aachen: HRB 8151
>     Ust.-Id: DE 813021663
>
>     Geschäftsführer: George Macdonald
>     ----------------------------------------------------------------------------------------------------
>
>     ----------------------------------------------------------------------------------------------------
>
>     take care of the environment - print only if necessary
>
>     ------------------------------------------------------------------------------
>     Check out the vibrant tech community on one of the world's most
>     engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>     _______________________________________________
>     Quickfix-developers mailing list
>     Qui...@li... <mailto:Qui...@li...>
>     https://lists.sourceforge.net/lists/listinfo/quickfix-developers
>
>

-- 
Christoph John
Development & Support
Direct: +49 241 557080-28
Mailto:Chr...@ma...
	


http://www.macd.com <http://www.macd.com/>
----------------------------------------------------------------------------------------------------
	
----------------------------------------------------------------------------------------------------
MACD GmbH
Oppenhoffallee 103
D-52066 Aachen
Tel: +49 241 557080-0 | Fax: +49 241 557080-10
	 Amtsgericht Aachen: HRB 8151
Ust.-Id: DE 813021663

Geschäftsführer: George Macdonald
----------------------------------------------------------------------------------------------------
	
----------------------------------------------------------------------------------------------------

take care of the environment - print only if necessary

Re: [Quickfix-developers] Fw: Disconnecting Due To First message not login

[<daw...@ya...>]

Hi Michael,

Thank you. Good point and I've tried this using FIXimulator many times - always a clean recovery without any issue.

Regards,
Dermot


----- Original Message -----
>From: Michael C. Starkie <hik...@gm...>
>To: chr...@ma...; "qui...@li..." <qui...@li...> 
>Date: 2017/6/20, Tue 20:06
>Subject: Re: [Quickfix-developers] Fw: Disconnecting Due To First message not login
> 
>QuickFIX Documentation: http://www.quickfixengine.org/quickfix/doc/html/
>
>
>
>Do you have a network at work to play with?  Perhaps you can write a simple dummy Acceptor in Java that accepts your apps connection from a different host on the same network.  How does your application behave when you disconnect and reconnect it's LAN connection? If it recovers than you have something to show NetOps.
>
>On Tue, Jun 20, 2017 at 6:26 AM Christoph John via Quickfix-developers <qui...@li...> wrote:
>
>QuickFIX Documentation: http://www.quickfixengine.org/quickfix/doc/html/
>>
>>Looks to me like a timeout on the network level. Then the other side thinks the connection is still
>>up and sends a test request while your side considers the connection broken and sends a new Logon
>>which the other side of course ignores since the session is still working.
>>
>>Chris.
>>
>>
>>On 14/06/17 09:21, daw...@ya...wrote:
>>> QuickFIX Documentation: http://www.quickfixengine.org/quickfix/doc/html/
>>>
>>>
>>> Hi Guys, Below is one I posted a month ago. Understand you're probably very busy but if you could
>>> take a look I'd appreciate it. You may have seen something similar before.
>>>
>>> Thanks!
>>> Dermot
>>>
>>>     ----- Forwarded Message -----
>>>     *From:* "daw...@ya..." <daw...@ya...>
>>>     *To:* "qui...@li..." <qui...@li...>
>>>     *Date:* 2017/5/11, Thu 04:17
>>>     *Subject:* [Quickfix-developers] Disconnecting Due To First message not login
>>>
>>>     QuickFIX Documentation: http://www.quickfixengine.org/quickfix/doc/html/
>>>
>>>
>>>     Hi Guys,
>>>
>>>     I've got a strange one here I'm hoping you might be able to help with. I've been doing some
>>>     testing with my broker, fail scenarios, specifically a "dirty disconnect" [disabling the LAN
>>>     connection] and then letting QuickFix handle the reconnect. It reconnects correctly, exchanges
>>>     heartbeats and then disconnects due to test request timeout. The broker tells me it's a
>>>     problem on my side because "first message is not LOGIN". He also says I'm connecting twice
>>>     (which I can't see is possible).
>>>
>>>     I've put together the attached file which shows what's happening. Blue rows are my outgoing
>>>     FIX messages, orange rows are the broker's incoming FIX messages, white are quickfix events
>>>     and the yellow rows are from the broker's log which he has sent me and I've spliced in there.
>>>     Could you please take a look? Start from the bottom - this is where I disconnected,
>>>
>>>     [Disconnecting Connection ID - 0 Due To First message not login] seems like a strange log
>>>     entry given the session has already exchanged sequence requests and heartbeats. One thing that
>>>     may be related.. the broker sends a TEST REQUEST at 14:58:30.341 which my side doesn't seem to
>>>     reply to. Please let me know if you have any ideas.
>>>
>>>     Thanks!
>>>     Dermot
>>>
>>>     ------------------------------------------------------------------------------
>>>     Check out the vibrant tech community on one of the world's most
>>>     engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>>>     _______________________________________________
>>>     Quickfix-developers mailing list
>>>     Qui...@li...<mailto:Qui...@li...>
>>>     https://lists.sourceforge.net/lists/listinfo/quickfix-developers
>>>
>>>
>>>
>>
>>--
>>Christoph John
>>Development & Support
>>Direct: +49 241 557080-28
>>Mailto:Chr...@ma...
>>
>>
>>
>>http://www.macd.com<http://www.macd.com/>
>>----------------------------------------------------------------------------------------------------
>>
>>----------------------------------------------------------------------------------------------------
>>MACD GmbH
>>Oppenhoffallee 103
>>D-52066 Aachen
>>Tel: +49 241 557080-0 | Fax: +49 241 557080-10
>>         Amtsgericht Aachen: HRB 8151
>>Ust.-Id: DE 813021663
>>
>>Geschäftsführer: George Macdonald
>>----------------------------------------------------------------------------------------------------
>>
>>----------------------------------------------------------------------------------------------------
>>
>>take care of the environment - print only if necessary
>>
>>------------------------------------------------------------------------------
>>Check out the vibrant tech community on one of the world's most
>>engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>>_______________________________________________
>>Quickfix-developers mailing list
>>Qui...@li...
>>https://lists.sourceforge.net/lists/listinfo/quickfix-developers
>>
>------------------------------------------------------------------------------
>Check out the vibrant tech community on one of the world's most
>engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>_______________________________________________
>Quickfix-developers mailing list
>Qui...@li...
>https://lists.sourceforge.net/lists/listinfo/quickfix-developers
>
>

Re: [Quickfix-developers] Fw: Disconnecting Due To First message not login

[<daw...@ya...>]

Hi Chris,

Thanks a lot for getting back. So I just wanna look at your response in relation to the log spreadsheet (attaching again).

[sends a test request] - You're taking about row 43 here?  

[other side thinks the connection is still up]  - That would make sense because there is no mention of a disconnect in the broker's logs (Yellow).

[while your side considers the connection broken and sends a new Logon which the other side of course ignores since the session is still working.] - the thing here is that the broker doesn't seem to be ignoring it because they send a login response (row 36). And this new connection seems to be working fine until right after heartbeat exchange (rows 13,14). If the broker still things the connection is fine back in row 43 then wouldn't they reject a new connection request (row 40)?

Help much appreciated!

Thanks,
Dermot





----- Original Message -----
>From: Christoph John via Quickfix-developers <qui...@li...>
>To: "qui...@li..." <qui...@li...> 
>Date: 2017/6/20, Tue 19:03
>Subject: Re: [Quickfix-developers] Fw: Disconnecting Due To First message not login
> 
>QuickFIX Documentation: http://www.quickfixengine.org/quickfix/doc/html/
>
>Looks to me like a timeout on the network level. Then the other side thinks the connection is still 
>up and sends a test request while your side considers the connection broken and sends a new Logon 
>which the other side of course ignores since the session is still working.
>
>Chris.
>
>
>On 14/06/17 09:21, daw...@ya...wrote:
>> QuickFIX Documentation: http://www.quickfixengine.org/quickfix/doc/html/
>>
>>
>> Hi Guys, Below is one I posted a month ago. Understand you're probably very busy but if you could 
>> take a look I'd appreciate it. You may have seen something similar before.
>>
>> Thanks!
>> Dermot
>>
>>     ----- Forwarded Message -----
>>     *From:* "daw...@ya..." <daw...@ya...>
>>     *To:* "qui...@li..." <qui...@li...>
>>     *Date:* 2017/5/11, Thu 04:17
>>     *Subject:* [Quickfix-developers] Disconnecting Due To First message not login
>>
>>     QuickFIX Documentation: http://www.quickfixengine.org/quickfix/doc/html/
>>
>>
>>     Hi Guys,
>>
>>     I've got a strange one here I'm hoping you might be able to help with. I've been doing some
>>     testing with my broker, fail scenarios, specifically a "dirty disconnect" [disabling the LAN
>>     connection] and then letting QuickFix handle the reconnect. It reconnects correctly, exchanges
>>     heartbeats and then disconnects due to test request timeout. The broker tells me it's a
>>     problem on my side because "first message is not LOGIN". He also says I'm connecting twice
>>     (which I can't see is possible).
>>
>>     I've put together the attached file which shows what's happening. Blue rows are my outgoing
>>     FIX messages, orange rows are the broker's incoming FIX messages, white are quickfix events
>>     and the yellow rows are from the broker's log which he has sent me and I've spliced in there.
>>     Could you please take a look? Start from the bottom - this is where I disconnected,
>>
>>     [Disconnecting Connection ID - 0 Due To First message not login] seems like a strange log
>>     entry given the session has already exchanged sequence requests and heartbeats. One thing that
>>     may be related.. the broker sends a TEST REQUEST at 14:58:30.341 which my side doesn't seem to
>>     reply to. Please let me know if you have any ideas.
>>
>>     Thanks!
>>     Dermot
>>
>>     ------------------------------------------------------------------------------
>>     Check out the vibrant tech community on one of the world's most
>>     engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>>     _______________________________________________
>>     Quickfix-developers mailing list
>>    Qui...@li...<mailto:Qui...@li...>
>>    https://lists.sourceforge.net/lists/listinfo/quickfix-developers
>>
>>
>>
>
>-- 
>Christoph John
>Development & Support
>Direct: +49 241 557080-28
>Mailto:Chr...@ma...
>    
>
>
>http://www.macd.com<http://www.macd.com/>
>----------------------------------------------------------------------------------------------------
>    
>----------------------------------------------------------------------------------------------------
>MACD GmbH
>Oppenhoffallee 103
>D-52066 Aachen
>Tel: +49 241 557080-0 | Fax: +49 241 557080-10
>     Amtsgericht Aachen: HRB 8151
>Ust.-Id: DE 813021663
>
>Geschäftsführer: George Macdonald
>----------------------------------------------------------------------------------------------------
>    
>----------------------------------------------------------------------------------------------------
>
>take care of the environment - print only if necessary
>
>------------------------------------------------------------------------------
>Check out the vibrant tech community on one of the world's most
>engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>_______________________________________________
>Quickfix-developers mailing list
>Qui...@li...
>https://lists.sourceforge.net/lists/listinfo/quickfix-developers
>
>
>

Re: [Quickfix-developers] Fw: Disconnecting Due To First message not login

[Michael C. S. <hik...@gm...>]

Do you have a network at work to play with? Perhaps you can write a simple
dummy Acceptor in Java that accepts your apps connection from a different
host on the same network. How does your application behave when you
disconnect and reconnect it's LAN connection? If it recovers than you have
something to show NetOps.
On Tue, Jun 20, 2017 at 6:26 AM Christoph John via Quickfix-developers <
qui...@li...> wrote:

> QuickFIX Documentation: http://www.quickfixengine.org/quickfix/doc/html/
>
> Looks to me like a timeout on the network level. Then the other side
> thinks the connection is still
> up and sends a test request while your side considers the connection
> broken and sends a new Logon
> which the other side of course ignores since the session is still working.
>
> Chris.
>
>
> On 14/06/17 09:21, daw...@ya... wrote:
> > QuickFIX Documentation: http://www.quickfixengine.org/quickfix/doc/html/
> >
> >
> > Hi Guys, Below is one I posted a month ago. Understand you're probably
> very busy but if you could
> > take a look I'd appreciate it. You may have seen something similar
> before.
> >
> > Thanks!
> > Dermot
> >
> >     ----- Forwarded Message -----
> >     *From:* "daw...@ya..." <daw...@ya...>
> >     *To:* "qui...@li..." <
> qui...@li...>
> >     *Date:* 2017/5/11, Thu 04:17
> >     *Subject:* [Quickfix-developers] Disconnecting Due To First message
> not login
> >
> >     QuickFIX Documentation:
> http://www.quickfixengine.org/quickfix/doc/html/
> >
> >
> >     Hi Guys,
> >
> >     I've got a strange one here I'm hoping you might be able to help
> with. I've been doing some
> >     testing with my broker, fail scenarios, specifically a "dirty
> disconnect" [disabling the LAN
> >     connection] and then letting QuickFix handle the reconnect. It
> reconnects correctly, exchanges
> >     heartbeats and then disconnects due to test request timeout. The
> broker tells me it's a
> >     problem on my side because "first message is not LOGIN". He also
> says I'm connecting twice
> >     (which I can't see is possible).
> >
> >     I've put together the attached file which shows what's happening.
> Blue rows are my outgoing
> >     FIX messages, orange rows are the broker's incoming FIX messages,
> white are quickfix events
> >     and the yellow rows are from the broker's log which he has sent me
> and I've spliced in there.
> >     Could you please take a look? Start from the bottom - this is where
> I disconnected,
> >
> >     [Disconnecting Connection ID - 0 Due To First message not login]
> seems like a strange log
> >     entry given the session has already exchanged sequence requests and
> heartbeats. One thing that
> >     may be related.. the broker sends a TEST REQUEST at 14:58:30.341
> which my side doesn't seem to
> >     reply to. Please let me know if you have any ideas.
> >
> >     Thanks!
> >     Dermot
> >
> >
>  ------------------------------------------------------------------------------
> >     Check out the vibrant tech community on one of the world's most
> >     engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> >     _______________________________________________
> >     Quickfix-developers mailing list
> >     Qui...@li... <mailto:
> Qui...@li...>
> >     https://lists.sourceforge.net/lists/listinfo/quickfix-developers
> >
> >
> >
>
> --
> Christoph John
> Development & Support
> Direct: +49 241 557080-28
> Mailto:Chr...@ma...
>
>
>
> http://www.macd.com <http://www.macd.com/>
>
> ----------------------------------------------------------------------------------------------------
>
>
> ----------------------------------------------------------------------------------------------------
> MACD GmbH
> Oppenhoffallee 103
> D-52066 Aachen
> Tel: +49 241 557080-0 | Fax: +49 241 557080-10
>          Amtsgericht Aachen: HRB 8151
> Ust.-Id: DE 813021663
>
> Geschäftsführer: George Macdonald
>
> ----------------------------------------------------------------------------------------------------
>
>
> ----------------------------------------------------------------------------------------------------
>
> take care of the environment - print only if necessary
>
>
> ------------------------------------------------------------------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> _______________________________________________
> Quickfix-developers mailing list
> Qui...@li...
> https://lists.sourceforge.net/lists/listinfo/quickfix-developers
>

Re: [Quickfix-developers] Fw: Disconnecting Due To First message not login

[Christoph J. <chr...@ma...>]

Looks to me like a timeout on the network level. Then the other side thinks the connection is still 
up and sends a test request while your side considers the connection broken and sends a new Logon 
which the other side of course ignores since the session is still working.

Chris.


On 14/06/17 09:21, daw...@ya... wrote:
> QuickFIX Documentation: http://www.quickfixengine.org/quickfix/doc/html/
>
>
> Hi Guys, Below is one I posted a month ago. Understand you're probably very busy but if you could 
> take a look I'd appreciate it. You may have seen something similar before.
>
> Thanks!
> Dermot
>
>     ----- Forwarded Message -----
>     *From:* "daw...@ya..." <daw...@ya...>
>     *To:* "qui...@li..." <qui...@li...>
>     *Date:* 2017/5/11, Thu 04:17
>     *Subject:* [Quickfix-developers] Disconnecting Due To First message not login
>
>     QuickFIX Documentation: http://www.quickfixengine.org/quickfix/doc/html/
>
>
>     Hi Guys,
>
>     I've got a strange one here I'm hoping you might be able to help with. I've been doing some
>     testing with my broker, fail scenarios, specifically a "dirty disconnect" [disabling the LAN
>     connection] and then letting QuickFix handle the reconnect. It reconnects correctly, exchanges
>     heartbeats and then disconnects due to test request timeout. The broker tells me it's a
>     problem on my side because "first message is not LOGIN". He also says I'm connecting twice
>     (which I can't see is possible).
>
>     I've put together the attached file which shows what's happening. Blue rows are my outgoing
>     FIX messages, orange rows are the broker's incoming FIX messages, white are quickfix events
>     and the yellow rows are from the broker's log which he has sent me and I've spliced in there.
>     Could you please take a look? Start from the bottom - this is where I disconnected,
>
>     [Disconnecting Connection ID - 0 Due To First message not login] seems like a strange log
>     entry given the session has already exchanged sequence requests and heartbeats. One thing that
>     may be related.. the broker sends a TEST REQUEST at 14:58:30.341 which my side doesn't seem to
>     reply to. Please let me know if you have any ideas.
>
>     Thanks!
>     Dermot
>
>     ------------------------------------------------------------------------------
>     Check out the vibrant tech community on one of the world's most
>     engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>     _______________________________________________
>     Quickfix-developers mailing list
>     Qui...@li... <mailto:Qui...@li...>
>     https://lists.sourceforge.net/lists/listinfo/quickfix-developers
>
>
>

-- 
Christoph John
Development & Support
Direct: +49 241 557080-28
Mailto:Chr...@ma...
	


http://www.macd.com <http://www.macd.com/>
----------------------------------------------------------------------------------------------------
	
----------------------------------------------------------------------------------------------------
MACD GmbH
Oppenhoffallee 103
D-52066 Aachen
Tel: +49 241 557080-0 | Fax: +49 241 557080-10
	 Amtsgericht Aachen: HRB 8151
Ust.-Id: DE 813021663

Geschäftsführer: George Macdonald
----------------------------------------------------------------------------------------------------
	
----------------------------------------------------------------------------------------------------

take care of the environment - print only if necessary

[Quickfix-developers] Fw: Disconnecting Due To First message not login

[<daw...@ya...>]

Hi Guys, Below is one I posted a month ago. Understand you're probably very busy but if you could take a look I'd appreciate it. You may have seen something similar before.

Thanks!
Dermot


----- Forwarded Message -----
>From: "daw...@ya..." <daw...@ya...>
>To: "qui...@li..." <qui...@li...> 
>Date: 2017/5/11, Thu 04:17
>Subject: [Quickfix-developers] Disconnecting Due To First message not login
> 
>QuickFIX Documentation: http://www.quickfixengine.org/quickfix/doc/html/
>
>
>
>Hi Guys,
>
>
>I've got a strange one here I'm hoping you might be able to help with. I've been doing some testing with my broker, fail scenarios, specifically a "dirty disconnect" [disabling the LAN connection] and then letting QuickFix handle the reconnect. It reconnects correctly, exchanges heartbeats and then disconnects due to test request timeout. The broker tells me it's a problem on my side because "first message is not LOGIN". He also says I'm connecting twice (which I can't see is possible).
>
>
>I've put together the attached file which shows what's happening. Blue rows are my outgoing FIX messages, orange rows are the broker's incoming FIX messages, white are quickfix events and the yellow rows are from the broker's log which he has sent me and I've spliced in there. Could you please take a look? Start from the bottom - this is where I disconnected,
>
>
>[Disconnecting Connection ID - 0 Due To First message not login] seems like a strange log entry given the session has already exchanged sequence requests and heartbeats. One thing that may be related.. the broker sends a TEST REQUEST at 14:58:30.341 which my side doesn't seem to reply to. Please let me know if you have any ideas.
>
>
>Thanks!
>Dermot
>------------------------------------------------------------------------------
>Check out the vibrant tech community on one of the world's most
>engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>_______________________________________________
>Quickfix-developers mailing list
>Qui...@li...
>https://lists.sourceforge.net/lists/listinfo/quickfix-developers
>
>

[Quickfix-developers] Disconnecting Due To First message not login

[<daw...@ya...>]

Hi Guys,

I've got a strange one here I'm hoping you might be able to help with. I've been doing some testing with my broker, fail scenarios, specifically a "dirty disconnect" [disabling the LAN connection] and then letting QuickFix handle the reconnect. It reconnects correctly, exchanges heartbeats and then disconnects due to test request timeout. The broker tells me it's a problem on my side because "first message is not LOGIN". He also says I'm connecting twice (which I can't see is possible).

I've put together the attached file which shows what's happening. Blue rows are my outgoing FIX messages, orange rows are the broker's incoming FIX messages, white are quickfix events and the yellow rows are from the broker's log which he has sent me and I've spliced in there. Could you please take a look? Start from the bottom - this is where I disconnected,

[Disconnecting Connection ID - 0 Due To First message not login] seems like a strange log entry given the session has already exchanged sequence requests and heartbeats. One thing that may be related.. the broker sends a TEST REQUEST at 14:58:30.341 which my side doesn't seem to reply to. Please let me know if you have any ideas.

Thanks!
Dermot

Re: [Quickfix-developers] Network disconnect recovery testing

[<daw...@ya...>]

Hi K. Frank,

Firstly apologies for my very tardy response - I got sidetracked on another project and didn't want to send a cursory reply after the effort you put in.

So you hit the nail on the head, yes our strategy will be sending limit orders on both sides until they are filled, with limits being changed continuously.  I'm using separate new order/cancel pairs rather than cancel-replace because limit orders may be replaced by a pegged algo (and visa versa) and that change isn't supported by the broker via cancel-replace.

Also, yes there will be cases where orders are cancelled outright (not individually but as a cancel-all function) and you're right it would make sense to have them included as a resend so I'll use the toApp() to control that logic for those orders.

[You still have a small potential issue.  According to you, after sending the initial order for a ticker/side, you only send pairs of cancel / new orders. It's possible that before a disconnect your cancel makes it through, but your new order doesn't] - this won't be a problem as the new order will not be triggered until I get the cancel confirm back.

[My plan is to have process which runs 120 seconds after each successful logon] - this will work probably 95% of the time but yes, point taken - it's hardly bullet proof. I could possibly use the QuickFix event "resend ..... has been satisfied" to check when a resend has ended but your suggestion of making an order status requests would lay all doubt to rest so I'm going to try that one out.

Automated trading errors.. yes I've seen the damage algos can do, especially the volume driven ones!

Thanks again for the great advice. I'll probably be back with more on this one!

Regards,
Dermot




----- Original Message -----
>From: K. Frank <kfr...@gm...>
>To: Quickfix Developers List <qui...@li...> 
>Date: 2017/4/20, Thu 06:15
>Subject: Re: [Quickfix-developers] Network disconnect recovery testing
> 
>QuickFIX Documentation: http://www.quickfixengine.org/quickfix/doc/html/
>
>Hello Dermot!
>
>On Tue, Apr 18, 2017 at 9:48 AM,  <daw...@ya...> wrote:
>> Hi K. Frank,
>>
>> Really appreciate the comprehensive response. Advice from the trenches is
>> exactly what I need!
>>
>> So actually for this business case we do NOT want to resend any orders or
>> cancels that didn't get through during downtime. The order submission is
>> fast-paced and the limit prices may be stale by the time we reconnect.
>
>This is reasonable.
>
>> Also, cancels are always linked to new orders.
>
>> (we don't allow more than one open order of the same ticker/side)
>
>>From this I take it that your trading strategy, roughly speaking,
>consists of having outstanding limit orders (possibly on both sides)
>for a set of securities, more or less continuously, at least until they
>get filled.  You do, however, adjust limit prices, by first cancelling such
>an order, and then issuing a new order with a new limit price (rather
>than issuing a single cancel-replace request that, in essence, modifies
>the limit price of an existing order.
>
>As side note: I would imagine that, if only as a matter of operational risk
>control, there would be instances where you would want to cancel an
>order without then issuing a new order.  It's your business logic, so
>maybe you really don't ever do this, but it would seem prudent to have
>this capability.  Anyway, if you do ever have "unpaired" cancels, it would
>seem that you would want to resend them in the event of a reconnect and
>a resend request from your counter-party.
>
>> This makes the approach simple -
>> any unacked messages can be 'cancelled' on our side. You're right about
>> deleting orders, that's not a good idea. Instead will set them to
>> 'Withdrawn' status.
>
>This seems reasonable.  When QuickFIX reconnects for you and responds
>to the resend request from your counter-party, it will give you a chance to
>intervene before it actually resends a given message.  It does this through
>the Application::toApp() callback that you can override in your MyApp (or
>whatever) class that you have derived from Application.  You can then tweak
>the messages before they are sent, or -- as in your use case -- throw a
>DoNotSend exception, in which case QuickFIX will gap-fill the sequence
>number (in effect telling your counter-party that the missed message was
>an ignorable admin message), rather than resend the message.
>
>> My plan is to have process which runs 120 seconds after each successful
>> logon [that should be enough time for any missing acks/fills to get back in
>> the resend].
>
>This should work.  However, as a matter of general practice, using this kind
>of magic-number wait time to insure that something has happened should
>be avoided when possible.
>
>When you reconnect, your counter-party tells you what sequence number
>it's on.  You (i.e., QuickFIX) notice that you've missed some messages so
>you (QuickFIX) issue a resend request.  You can then wait until your
>counter-party is done with its resend.
>
>I'm not entirely sure how to get QuickFIX to tell you that the resend is done,
>though.  The resent messages ought to have their PosDupFlag set, so it
>might work to wait until you get a message that was sent post reconnect
>because its PosDupFlag isn't set.  (QuickFIX queues up out-of-order
>messages for you, so messages should delivered to your application logic
>in order after the resend is complete.)  Maybe you could status a known
>good order and use its status message as a flag that the resend is complete.
>
>(It would be nice if you could get a callback from QuickFIX or query it directly
>somehow to get it to tell you that the resend is complete.  I don't know of a
>way to do this, but perhaps there is one.)
>
>> The process checks if there are any orders in 'Sent' status
>> (unacked) and created before last logon time - update these orders to
>> 'Withdrawn' status.
>
>Yes, so the logic would be something like:
>
>Inspect orders that QuickFIX is resending on your behalf in the toApp callback,
>and throw the DoNotSend exception so that you don't resend new orders and
>cancel requests that presumably haven't been sent.
>
>Wait until your counter-party's resend is complete (e.g., as outlined
>above), and
>then update unacked orders in your own order store to your proposed "Withdrawn"
>status.
>
>> Simple but should do the job. Definitely need to get unacked orders out of
>> the way as they will interfere with new orders being sent (we don't allow
>> more than one open order of the same ticker/side). There's no auto-cancel
>> requirements from the broker so the policy is up to us.
>
>You still have a small potential issue.  According to you, after
>sending the initial
>order for a ticker/side, you only send pairs of cancel / new orders.
>It's possible
>that before a disconnect your cancel makes it through, but your new
>order doesn't.
>
>So you will want to have a little bit of logic that recognizes this
>situation and lets
>you send another new order without it being part of a cancel / new order pair
>(because the cancel was already sent and correctly processed).
>
>> What do you think?
>
>If you (try to) send an order, reconnect, get all of your
>counter-party's messages
>through a proper resend request, and you don't get an ack for that
>order, then it's
>fair to say that your counter-party has told you that he did not
>receive the order.
>
>But relying on your counter-party's silence to communicate this could be less
>reliable that getting an affirmative statement that the order was not received.
>
>To be redundantly cautious, you could proactively send status requests for your
>unacked orders, and if your counter-party never got them, you should get back
>some kind of unknown order message.  I believe that the FIX specification says
>you can do this using the ClOrdID (the order id generated on your end and sent
>as part of your new order message).  However, bear in mind, some counter-parties
>may require that you status the order using the OrdID (the order id assigned by
>the counter-party and sent back in the ack).  Well, since you never got an ack,
>you wouldn't be able to do it this way.  (As I said, I don't believe
>that requiring the
>OrdID in place of the ClOrdID is really meets the spec, but it is a
>pitfall to watch
>out for.)
>
>Similarly, if you tried to send a cancel (in your case, presumably as part of a
>cancel / new order pair), but never got a ur-out, you could status the order to
>confirm that it is still alive, rather than relying on silence (i.e.,
>the absence of
>the ur-out) to indicate that the order is alive.
>
>Although this kind of double-check requires building some additional application
>logic, it will probably be worth it if you're going to trade any
>significant flow
>through your application or your application is going to be in service for more
>than, say, a few months.
>
>Trust me, stuff happens.
>
>It always seems cheaper not to invest in this kind of additional
>development cost.
>Cheaper, at least, until you're on the losing side of a trading error
>(and automated-
>trading trading errors can blow up in a hurry).
>
>> Thanks!
>> Dermot
>
>
>Happy Trading!
>
>
>K. Frank
>
>
>> ----- Original Message -----
>> From: K. Frank <kfr...@gm...>
>> To: Quickfix Developers List <qui...@li...>
>> Date: 2017/4/17, Mon 21:33
>> Subject: Re: [Quickfix-developers] Network disconnect recovery testing
>>
>> Hi Dermot!
>>
>> On Mon, Apr 17, 2017 at 5:45 AM,  <daw...@ya...> wrote:
>>> ...
>>> Hi Mike,
>>>
>>> Just getting back to orders that are sent (and don't go anywhere) during
>>> network downtime - is there a recommended approach to identifying and
>>> cleaning these up? I'm thinking to just run a process every minute or so
>>> that checks "if currently logged on and order hasn't been acked in the last
>>> minute then delete from database". What do you think?
>>
>> Some words of advice from the trenches ...
>>
>> Think through your recovery strategy carefully (as you seem to
>> be doing).
>> ...
>
>------------------------------------------------------------------------------
>Check out the vibrant tech community on one of the world's most
>engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>_______________________________________________
>Quickfix-developers mailing list
>Qui...@li...
>https://lists.sourceforge.net/lists/listinfo/quickfix-developers
>
>
>

Re: [Quickfix-developers] install Quickfix FIX engine on windows, use python API to build a client/buy-side

[Asaf B. G. <asa...@gm...>]

Hello Frank!

Thanks very much for the quick reply! I'm sorry for my late reply...

It is weird that people compile it with  "&func" and it works for them...
but as you said, it probably depends on the compiler...

These things are all about the try and error process, but the most
important thing is that eventually it works!!

Happy FIXing too!

Best,

Asaf









On Sun, May 7, 2017 at 5:10 PM, K. Frank <kfr...@gm...> wrote:

> QuickFIX Documentation: http://www.quickfixengine.org/quickfix/doc/html/
>
> Hello Asaf!
>
> On Fri, Apr 28, 2017 at 4:47 AM, Asaf Ben Gal <asa...@wh...>
> wrote:
> > ...
> > By the way, I installed quickfix by downloading the "source tar" and
> building the "quickfix_vs12.sln" in Microsoft visual studio.
> >
> > The thing is that it had a syntax error in the "utility.cpp' file in
> line number 399 in the "bool thread_spawn( THREAD_START_ROUTINE func, void*
> var, thread_id& thread )" method.
>
> I saw the same problem with quickfix-1.13.3 back in 2013.
>
> > The original (399 line) was: "result = _beginthreadex( NULL, 0, &func,
> var, 0, &id );" and I deleted the "&" sign so that it would be: "result =
> _beginthreadex( NULL, 0, func, var, 0, &id );"
> >
> > and only then the build was successful...
>
> I made the same correction and was able to build and run quickfix in a
> smallish application without any obvious problems.
>
> > Is it a known issue? Did I fix it correctly? I'm not familiar with c# so
> I have no idea why it worked....
>
> (By the way, I think you mean c++ here.)
>
> I started a thread on this list about it.  Here's a link to my original
> post:
>
>    https://sourceforge.net/p/quickfix/mailman/message/30333881/
>
> and to the entire thread:
>
>    https://sourceforge.net/p/quickfix/mailman/quickfix-developers/thread/
> CAKEnbTMwYrXCU1e55ywAzXxLnVoPQfcMyxay%3DEDkNq_OHqoBvg%
> 40mail.gmail.com/#msg30333881
>
> In that thread Mike Gatny said:
>
>    It works with the VS compiler (32-bit, anyway).
>
> I don't doubt that it worked for him, but this never made any sense to me.
> My reading of the original code (with "&func") is such that I can't see how
> it would ever compile, even with a "tolerant" compiler.
>
> (I should note that I was doing a presumably unsupported windows build
> using mingw-w64, a windows port of gcc, rather than ms visual studio.)
>
> It's weird.  You see exactly what I would expect on windows -- a compile
> error when using "&func".  But, as far as I can tell, lots of people build
> and
> run quickfix on windows, presumably without seeing this error.
>
> > Thanks again,
> >
> > Asaf
>
>
> Happy FIXing!
>
>
> K. Frank
>
> ------------------------------------------------------------
> ------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> _______________________________________________
> Quickfix-developers mailing list
> Qui...@li...
> https://lists.sourceforge.net/lists/listinfo/quickfix-developers
>

Re: [Quickfix-developers] install Quickfix FIX engine on windows, use python API to build a client/buy-side

[K. F. <kfr...@gm...>]

Hello Asaf!

On Fri, Apr 28, 2017 at 4:47 AM, Asaf Ben Gal <asa...@wh...> wrote:
> ...
> By the way, I installed quickfix by downloading the "source tar" and building the "quickfix_vs12.sln" in Microsoft visual studio.
>
> The thing is that it had a syntax error in the "utility.cpp' file in line number 399 in the "bool thread_spawn( THREAD_START_ROUTINE func, void* var, thread_id& thread )" method.

I saw the same problem with quickfix-1.13.3 back in 2013.

> The original (399 line) was: "result = _beginthreadex( NULL, 0, &func, var, 0, &id );" and I deleted the "&" sign so that it would be: "result = _beginthreadex( NULL, 0, func, var, 0, &id );"
>
> and only then the build was successful...

I made the same correction and was able to build and run quickfix in a
smallish application without any obvious problems.

> Is it a known issue? Did I fix it correctly? I'm not familiar with c# so I have no idea why it worked....

(By the way, I think you mean c++ here.)

I started a thread on this list about it.  Here's a link to my original post:

   https://sourceforge.net/p/quickfix/mailman/message/30333881/

and to the entire thread:

   https://sourceforge.net/p/quickfix/mailman/quickfix-developers/thread/CAKEnbTMwYrXCU1e55ywAzXxLnVoPQfcMyxay%3DEDkNq_OHqoBvg%40mail.gmail.com/#msg30333881

In that thread Mike Gatny said:

   It works with the VS compiler (32-bit, anyway).

I don't doubt that it worked for him, but this never made any sense to me.
My reading of the original code (with "&func") is such that I can't see how
it would ever compile, even with a "tolerant" compiler.

(I should note that I was doing a presumably unsupported windows build
using mingw-w64, a windows port of gcc, rather than ms visual studio.)

It's weird.  You see exactly what I would expect on windows -- a compile
error when using "&func".  But, as far as I can tell, lots of people build and
run quickfix on windows, presumably without seeing this error.

> Thanks again,
>
> Asaf


Happy FIXing!


K. Frank

Re: [Quickfix-developers] install Quickfix FIX engine on windows, use python API to build a client/buy-side

[Asaf B. G. <asa...@gm...>]

Hi Johnson,

Thanks for the quick reply!

Finally I managed to install quickfix and install the quickfix py.

My problem was that I had both python 3 and python 2 on my machine...
removed python 3 and it worked.


By the way, I installed quickfix by downloading the "source tar" and
building the "quickfix_vs12.sln" in Microsoft visual studio.

The thing is that it had a syntax error in the "utility.cpp' file in line
number 399 in the "bool thread_spawn( THREAD_START_ROUTINE func, void* var,
thread_id& thread )" method.

The original (399 line) was: "result = _beginthreadex( NULL, 0, &func, var,
0, &id );" and I deleted the "&" sign so that it would be: "result =
_beginthreadex( NULL, 0, func, var, 0, &id );"

and only then the build was successful...

Is it a known issue? Did I fix it correctly? I'm not familiar with c# so I
have no idea why it worked....

Thanks again,

Asaf



On Fri, Apr 28, 2017 at 5:54 AM, Zhangjianyue <zj...@16...> wrote:

> Hi,
>
> Perhaps module wheel is not installed in your system.
> Try to do as following:
> 1. Download the whl from http://www.lfd.uci.edu/~gohlke/pythonlibs/
> <http://link.zhihu.com/?target=http%3A//www.lfd.uci.edu/%7Egohlke/pythonlibs/%23lxml>
>   search wheel
> 2. Enter pip document:
> pip install *.whl
> 3. Reinstall quickfix py.
>
> It may help you, and good luck.
> Cheers.
> Johnson
>
>
> At 2017-04-27 22:01:27, "Asaf Ben Gal" <asa...@gm...> wrote:
> >QuickFIX Documentation: http://www.quickfixengine.org/quickfix/doc/html/
> >
> >Hello,
> >
> >I want to install the Quickfix FIX engine on windows and use its python API
> >to build a client/buy-side application.
> >
> >I'm using windows 10, 64bit
> >Python 2.7
> >
> >I don't know how to install the Quickfix engine, I tried to do the listed
> >below to no sucess:
> >1. pip install quickfix but it gives errors like "failed building wheel for
> >quickfix"
> >2. Downloading the "source zip" file from "http://www.quickfixengine.org/"
> >and using visual studio community 2015 to "batch build" the
> >"quickfix_vs12.sln" but it gives errors that "quickfix.lib" does not exist.
> >3. install with "quickfix‑1.14.3‑cp27‑none‑win_amd64.whl" from
> >"http://www.lfd.uci.edu/~gohlke//pythonlibs/" but it also didn't work...
> >
> >I tried more stuff to no success :(
> >
> >Can anyone help please with the steps needed to:
> >1. Install quickfix engine on windows 10 amd64
> >2. Make the Python package of quickfix available to use in order to build a
> >buy-side client.
> >
> >Thanks,
> >
> >Asaf
> >
> >
> >
> >--
> >View this message in context: http://quickfix.13857.n7.nabble.com/install-Quickfix-FIX-engine-on-windows-use-python-API-to-build-a-client-buy-side-tp6874.html
> >Sent from the QuickFIX - Dev mailing list archive at Nabble.com.
> >
> >------------------------------------------------------------------------------
> >Check out the vibrant tech community on one of the world's most
> >engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> >_______________________________________________
> >Quickfix-developers mailing list
> >Qui...@li...
> >https://lists.sourceforge.net/lists/listinfo/quickfix-developers
>
>

Re: [Quickfix-developers] install Quickfix FIX engine on windows, use python API to build a client/buy-side

[Zhangjianyue <zj...@16...>]

Hi, 


Perhaps module wheel is not installed in your system.
Try to do as following:
1. Download the whl from http://www.lfd.uci.edu/~gohlke/pythonlibs/
  search wheel
2. Enter pip document:
pip install *.whl
3. Reinstall quickfix py.


It may help you, and good luck.
Cheers.
Johnson



At 2017-04-27 22:01:27, "Asaf Ben Gal" <asa...@gm...> wrote:
>QuickFIX Documentation: http://www.quickfixengine.org/quickfix/doc/html/
>
>Hello, 
>
>I want to install the Quickfix FIX engine on windows and use its python API
>to build a client/buy-side application.
>
>I'm using windows 10, 64bit
>Python 2.7
>
>I don't know how to install the Quickfix engine, I tried to do the listed
>below to no sucess:
>1. pip install quickfix but it gives errors like "failed building wheel for
>quickfix"
>2. Downloading the "source zip" file from "http://www.quickfixengine.org/"
>and using visual studio community 2015 to "batch build" the
>"quickfix_vs12.sln" but it gives errors that "quickfix.lib" does not exist.
>3. install with "quickfix‑1.14.3‑cp27‑none‑win_amd64.whl" from
>"http://www.lfd.uci.edu/~gohlke//pythonlibs/" but it also didn't work...
>
>I tried more stuff to no success :(
>
>Can anyone help please with the steps needed to:
>1. Install quickfix engine on windows 10 amd64
>2. Make the Python package of quickfix available to use in order to build a
>buy-side client.
>
>Thanks,
>
>Asaf 
>
>
>
>--
>View this message in context: http://quickfix.13857.n7.nabble.com/install-Quickfix-FIX-engine-on-windows-use-python-API-to-build-a-client-buy-side-tp6874.html
>Sent from the QuickFIX - Dev mailing list archive at Nabble.com.
>
>------------------------------------------------------------------------------
>Check out the vibrant tech community on one of the world's most
>engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>_______________________________________________
>Quickfix-developers mailing list
>Qui...@li...
>https://lists.sourceforge.net/lists/listinfo/quickfix-developers

[Quickfix-developers] install Quickfix FIX engine on windows, use python API to build a client/buy-side

[Asaf B. G. <asa...@gm...>]

Hello, 

I want to install the Quickfix FIX engine on windows and use its python API
to build a client/buy-side application.

I'm using windows 10, 64bit
Python 2.7

I don't know how to install the Quickfix engine, I tried to do the listed
below to no sucess:
1. pip install quickfix but it gives errors like "failed building wheel for
quickfix"
2. Downloading the "source zip" file from "http://www.quickfixengine.org/"
and using visual studio community 2015 to "batch build" the
"quickfix_vs12.sln" but it gives errors that "quickfix.lib" does not exist.
3. install with "quickfix‑1.14.3‑cp27‑none‑win_amd64.whl" from
"http://www.lfd.uci.edu/~gohlke//pythonlibs/" but it also didn't work...

I tried more stuff to no success :(

Can anyone help please with the steps needed to:
1. Install quickfix engine on windows 10 amd64
2. Make the Python package of quickfix available to use in order to build a
buy-side client.

Thanks,

Asaf 



--
View this message in context: http://quickfix.13857.n7.nabble.com/install-Quickfix-FIX-engine-on-windows-use-python-API-to-build-a-client-buy-side-tp6874.html
Sent from the QuickFIX - Dev mailing list archive at Nabble.com.

Re: [Quickfix-developers] Network disconnect recovery testing

[K. F. <kfr...@gm...>]

Hello Dermot!

On Tue, Apr 18, 2017 at 9:48 AM,  <daw...@ya...> wrote:
> Hi K. Frank,
>
> Really appreciate the comprehensive response. Advice from the trenches is
> exactly what I need!
>
> So actually for this business case we do NOT want to resend any orders or
> cancels that didn't get through during downtime. The order submission is
> fast-paced and the limit prices may be stale by the time we reconnect.

This is reasonable.

> Also, cancels are always linked to new orders.

> (we don't allow more than one open order of the same ticker/side)

>From this I take it that your trading strategy, roughly speaking,
consists of having outstanding limit orders (possibly on both sides)
for a set of securities, more or less continuously, at least until they
get filled.  You do, however, adjust limit prices, by first cancelling such
an order, and then issuing a new order with a new limit price (rather
than issuing a single cancel-replace request that, in essence, modifies
the limit price of an existing order.

As side note: I would imagine that, if only as a matter of operational risk
control, there would be instances where you would want to cancel an
order without then issuing a new order.  It's your business logic, so
maybe you really don't ever do this, but it would seem prudent to have
this capability.  Anyway, if you do ever have "unpaired" cancels, it would
seem that you would want to resend them in the event of a reconnect and
a resend request from your counter-party.

> This makes the approach simple -
> any unacked messages can be 'cancelled' on our side. You're right about
> deleting orders, that's not a good idea. Instead will set them to
> 'Withdrawn' status.

This seems reasonable.  When QuickFIX reconnects for you and responds
to the resend request from your counter-party, it will give you a chance to
intervene before it actually resends a given message.  It does this through
the Application::toApp() callback that you can override in your MyApp (or
whatever) class that you have derived from Application.  You can then tweak
the messages before they are sent, or -- as in your use case -- throw a
DoNotSend exception, in which case QuickFIX will gap-fill the sequence
number (in effect telling your counter-party that the missed message was
an ignorable admin message), rather than resend the message.

> My plan is to have process which runs 120 seconds after each successful
> logon [that should be enough time for any missing acks/fills to get back in
> the resend].

This should work.  However, as a matter of general practice, using this kind
of magic-number wait time to insure that something has happened should
be avoided when possible.

When you reconnect, your counter-party tells you what sequence number
it's on.  You (i.e., QuickFIX) notice that you've missed some messages so
you (QuickFIX) issue a resend request.  You can then wait until your
counter-party is done with its resend.

I'm not entirely sure how to get QuickFIX to tell you that the resend is done,
though.  The resent messages ought to have their PosDupFlag set, so it
might work to wait until you get a message that was sent post reconnect
because its PosDupFlag isn't set.  (QuickFIX queues up out-of-order
messages for you, so messages should delivered to your application logic
in order after the resend is complete.)  Maybe you could status a known
good order and use its status message as a flag that the resend is complete.

(It would be nice if you could get a callback from QuickFIX or query it directly
somehow to get it to tell you that the resend is complete.  I don't know of a
way to do this, but perhaps there is one.)

> The process checks if there are any orders in 'Sent' status
> (unacked) and created before last logon time - update these orders to
> 'Withdrawn' status.

Yes, so the logic would be something like:

Inspect orders that QuickFIX is resending on your behalf in the toApp callback,
and throw the DoNotSend exception so that you don't resend new orders and
cancel requests that presumably haven't been sent.

Wait until your counter-party's resend is complete (e.g., as outlined
above), and
then update unacked orders in your own order store to your proposed "Withdrawn"
status.

> Simple but should do the job. Definitely need to get unacked orders out of
> the way as they will interfere with new orders being sent (we don't allow
> more than one open order of the same ticker/side). There's no auto-cancel
> requirements from the broker so the policy is up to us.

You still have a small potential issue.  According to you, after
sending the initial
order for a ticker/side, you only send pairs of cancel / new orders.
It's possible
that before a disconnect your cancel makes it through, but your new
order doesn't.

So you will want to have a little bit of logic that recognizes this
situation and lets
you send another new order without it being part of a cancel / new order pair
(because the cancel was already sent and correctly processed).

> What do you think?

If you (try to) send an order, reconnect, get all of your
counter-party's messages
through a proper resend request, and you don't get an ack for that
order, then it's
fair to say that your counter-party has told you that he did not
receive the order.

But relying on your counter-party's silence to communicate this could be less
reliable that getting an affirmative statement that the order was not received.

To be redundantly cautious, you could proactively send status requests for your
unacked orders, and if your counter-party never got them, you should get back
some kind of unknown order message.  I believe that the FIX specification says
you can do this using the ClOrdID (the order id generated on your end and sent
as part of your new order message).  However, bear in mind, some counter-parties
may require that you status the order using the OrdID (the order id assigned by
the counter-party and sent back in the ack).  Well, since you never got an ack,
you wouldn't be able to do it this way.  (As I said, I don't believe
that requiring the
OrdID in place of the ClOrdID is really meets the spec, but it is a
pitfall to watch
out for.)

Similarly, if you tried to send a cancel (in your case, presumably as part of a
cancel / new order pair), but never got a ur-out, you could status the order to
confirm that it is still alive, rather than relying on silence (i.e.,
the absence of
the ur-out) to indicate that the order is alive.

Although this kind of double-check requires building some additional application
logic, it will probably be worth it if you're going to trade any
significant flow
through your application or your application is going to be in service for more
than, say, a few months.

Trust me, stuff happens.

It always seems cheaper not to invest in this kind of additional
development cost.
Cheaper, at least, until you're on the losing side of a trading error
(and automated-
trading trading errors can blow up in a hurry).

> Thanks!
> Dermot


Happy Trading!


K. Frank


> ----- Original Message -----
> From: K. Frank <kfr...@gm...>
> To: Quickfix Developers List <qui...@li...>
> Date: 2017/4/17, Mon 21:33
> Subject: Re: [Quickfix-developers] Network disconnect recovery testing
>
> Hi Dermot!
>
> On Mon, Apr 17, 2017 at 5:45 AM,  <daw...@ya...> wrote:
>> ...
>> Hi Mike,
>>
>> Just getting back to orders that are sent (and don't go anywhere) during
>> network downtime - is there a recommended approach to identifying and
>> cleaning these up? I'm thinking to just run a process every minute or so
>> that checks "if currently logged on and order hasn't been acked in the last
>> minute then delete from database". What do you think?
>
> Some words of advice from the trenches ...
>
> Think through your recovery strategy carefully (as you seem to
> be doing).
> ...

Re: [Quickfix-developers] Network disconnect recovery testing

[<daw...@ya...>]

Hi K. Frank,

Really appreciate the comprehensive response. Advice from the trenches is exactly what I need!

So actually for this business case we do NOT want to resend any orders or cancels that didn't get through during downtime. The order submission is fast-paced and the limit prices may be stale by the time we reconnect. Also, cancels are always linked to new orders. This makes the approach simple - any unacked messages can be 'cancelled' on our side. You're right about deleting orders, that's not a good idea. Instead will set them to 'Withdrawn' status.

My plan is to have process which runs 120 seconds after each successful logon [that should be enough time for any missing acks/fills to get back in the resend]. The process checks if there are any orders in 'Sent' status (unacked) and created before last logon time - update these orders to 'Withdrawn' status.

Simple but should do the job. Definitely need to get unacked orders out of the way as they will interfere with new orders being sent (we don't allow more than one open order of the same ticker/side). There's no auto-cancel requirements from the broker so the policy is up to us.

What do you think?


Thanks!
Dermot




----- Original Message -----
>From: K. Frank <kfr...@gm...>
>To: Quickfix Developers List <qui...@li...> 
>Date: 2017/4/17, Mon 21:33
>Subject: Re: [Quickfix-developers] Network disconnect recovery testing
> 
>QuickFIX Documentation: http://www.quickfixengine.org/quickfix/doc/html/
>
>Hi Dermot!
>
>On Mon, Apr 17, 2017 at 5:45 AM,  <daw...@ya...> wrote:
>> ...
>> Hi Mike,
>>
>> Just getting back to orders that are sent (and don't go anywhere) during network downtime - is there a recommended approach to identifying and cleaning these up? I'm thinking to just run a process every minute or so that checks "if currently logged on and order hasn't been acked in the last minute then delete from database". What do you think?
>
>Some words of advice from the trenches ...
>
>Think through your recovery strategy carefully (as you seem to
>be doing).
>
>First, if you send an order and receive an ack, then your counter-
>party has accepted responsibility for that order.  (Hopefully he
>fulfills his responsibility.)
>
>If you try to send an order and don't receive an ack, you don't
>know whether your counter-party has it or not.  (Send order,
>receive and process order, send ack -- network goes down --
>don't receive ack.  Note, in this scenario, your counter-party
>doesn't know -- until and unless you request a gap fill -- that
>you didn't receive the ack.  FIX doesn't ack acks.)  Note, this
>means that you REALLY don't want to delete unacked orders
>from your database and forget about them -- they might be
>live on your counter-party's side.  So you need, at least, to
>put them in some kind of limbo state, and have a protocol
>for cleaning them up (or leaving them live when you get your
>ack after a reconnect and gap fill).
>
>You need to decide on a business protocol for handling network
>outages.  Some counter-parties will let you (or require) that you
>have them (pre-arranged -- not via FIX) auto-cancel any orders
>if connectivity drops.  Note such a "cure" can be worse than the
>problem if your network goes down for a few milliseconds or a
>few seconds or even a few minutes.  It depends on your use case.
>
>Similarly, you need to decide what you want to do with orders
>that you have attempted to issue but haven't sent yet.  Again,
>if the outage was only for a few milliseconds or seconds, you
>might be best to just send them.
>
>The problem is with orders that you might have sent.  You can't
>really not resend them in a gap fill because that could be
>inconsistent with the traffic you counter-party has already seen.
>You could negotiate with your counter-party an "auto-cancel" policy
>for new, previously unseen orders that come in a gap fill.
>
>Let's say you do decide to filter out (somehow) possibly unsent traffic
>when gap filling after an outage.  While you might want to "filter out"
>new orders (or have your counter-party ignore / auto-cancel them)
>you almost certainly do not want to filter out possibly unsent cancel
>requests.
>
>You are right that it makes a lot of sense to monitor whether you
>have connectivity with your counter-party.  Note there are several
>levels to this:
>
>Business "connectivity" -- e.g., receive acks
>FIX connectivity -- heartbeats, isLoggedOn
>Network connectivity -- e.g., a free-standing "ping" watchdog
>
>It does make sense -- to reduce potential gap-fill load and economic
>exposure if you don't manage a timely reconnect -- to pause
>sending orders on your side if you detect possible connectivity
>interruption.  You may or may not wish to queue up such unsent
>traffic to send if you reestablish connectivity after a "short" amount
>of time.  Note, there is real business logic in how you decide to
>handle this.  Would you really not want to send cancel requests
>that you generated and got queued up during a (real or imagined)
>loss of connectivity?
>
>Repeating two key points:
>
>It is well worth designing your recovery strategy with care.
>
>The details of your strategy will depend on your specific business
>use case -- auto-cancellation policies, how long an outage puts
>you into a clean-up mode, rather than recovering and proceeding
>normally (keeping orders live, etc.).
>
>And last:  Test to the extent you can afford and your business requires.
>I have NEVER seen automated recovery work completely correctly in
>institutional environments -- even with reasonably well-tested systems.
>(I'm not saying it can't happen -- I've just never seen it.)
>
>
>> Dermot
>
>
>Good luck.
>
>
>K. Frank
>
>------------------------------------------------------------------------------
>Check out the vibrant tech community on one of the world's most
>engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>_______________________________________________
>Quickfix-developers mailing list
>Qui...@li...
>https://lists.sourceforge.net/lists/listinfo/quickfix-developers
>
>
>

Re: [Quickfix-developers] Network disconnect recovery testing

[K. F. <kfr...@gm...>]

Hi Dermot!

On Mon, Apr 17, 2017 at 5:45 AM,  <daw...@ya...> wrote:
> ...
> Hi Mike,
>
> Just getting back to orders that are sent (and don't go anywhere) during network downtime - is there a recommended approach to identifying and cleaning these up? I'm thinking to just run a process every minute or so that checks "if currently logged on and order hasn't been acked in the last minute then delete from database". What do you think?

Some words of advice from the trenches ...

Think through your recovery strategy carefully (as you seem to
be doing).

First, if you send an order and receive an ack, then your counter-
party has accepted responsibility for that order.  (Hopefully he
fulfills his responsibility.)

If you try to send an order and don't receive an ack, you don't
know whether your counter-party has it or not.  (Send order,
receive and process order, send ack -- network goes down --
don't receive ack.  Note, in this scenario, your counter-party
doesn't know -- until and unless you request a gap fill -- that
you didn't receive the ack.  FIX doesn't ack acks.)  Note, this
means that you REALLY don't want to delete unacked orders
from your database and forget about them -- they might be
live on your counter-party's side.  So you need, at least, to
put them in some kind of limbo state, and have a protocol
for cleaning them up (or leaving them live when you get your
ack after a reconnect and gap fill).

You need to decide on a business protocol for handling network
outages.  Some counter-parties will let you (or require) that you
have them (pre-arranged -- not via FIX) auto-cancel any orders
if connectivity drops.  Note such a "cure" can be worse than the
problem if your network goes down for a few milliseconds or a
few seconds or even a few minutes.  It depends on your use case.

Similarly, you need to decide what you want to do with orders
that you have attempted to issue but haven't sent yet.  Again,
if the outage was only for a few milliseconds or seconds, you
might be best to just send them.

The problem is with orders that you might have sent.  You can't
really not resend them in a gap fill because that could be
inconsistent with the traffic you counter-party has already seen.
You could negotiate with your counter-party an "auto-cancel" policy
for new, previously unseen orders that come in a gap fill.

Let's say you do decide to filter out (somehow) possibly unsent traffic
when gap filling after an outage.  While you might want to "filter out"
new orders (or have your counter-party ignore / auto-cancel them)
you almost certainly do not want to filter out possibly unsent cancel
requests.

You are right that it makes a lot of sense to monitor whether you
have connectivity with your counter-party.  Note there are several
levels to this:

Business "connectivity" -- e.g., receive acks
FIX connectivity -- heartbeats, isLoggedOn
Network connectivity -- e.g., a free-standing "ping" watchdog

It does make sense -- to reduce potential gap-fill load and economic
exposure if you don't manage a timely reconnect -- to pause
sending orders on your side if you detect possible connectivity
interruption.  You may or may not wish to queue up such unsent
traffic to send if you reestablish connectivity after a "short" amount
of time.  Note, there is real business logic in how you decide to
handle this.  Would you really not want to send cancel requests
that you generated and got queued up during a (real or imagined)
loss of connectivity?

Repeating two key points:

It is well worth designing your recovery strategy with care.

The details of your strategy will depend on your specific business
use case -- auto-cancellation policies, how long an outage puts
you into a clean-up mode, rather than recovering and proceeding
normally (keeping orders live, etc.).

And last:  Test to the extent you can afford and your business requires.
I have NEVER seen automated recovery work completely correctly in
institutional environments -- even with reasonably well-tested systems.
(I'm not saying it can't happen -- I've just never seen it.)


> Dermot


Good luck.


K. Frank

Re: [Quickfix-developers] Network disconnect recovery testing

[<daw...@ya...>]

Hi Mike,

Just getting back to orders that are sent (and don't go anywhere) during network downtime - is there a recommended approach to identifying and cleaning these up? I'm thinking to just run a process every minute or so that checks "if currently logged on and order hasn't been acked in the last minute then delete from database". What do you think?

Dermot


----- Original Message -----
>From: Mike Gatny <mg...@co...>
>To: daw...@ya... 
>Cc: Grant Birchmeier <gbi...@co...>; "qui...@li..." <qui...@li...>
>Date: 2017/4/14, Fri 00:31
>Subject: Re: Network disconnect recovery testing
> 
>
>Dermot,
>
>This is one of the reasons the qf/j implementation has config options like JdbcLogHeartBeats/FileLogHeartbeats.
>I don't think it would be too hard to add this to MySQLLog for qf/c++.
>
>
>--Mike Gatny
>Connamara Systems, LLC
>
>On Thu, Apr 13, 2017 at 11:18 AM, <daw...@ya...>wrote:
>
>Hi Mike,
>>
>>
>>Thanks for the quick response. I haven't included PersistMessages in the config file but I can see it defaults to 'N' so that makes sense.
>>
>>
>>Ok noted on the onLogon()/onLogout(). Up to now I've just been catching the event text "Received logon response" and "Disconnecting" which is a bit hacky but the result is the same. The only issue I have is I've been trying to keep HeartBtInt=120 so as not to flood MySql with too many messages - this means that it takes a minute or two for quickfix to realise there's a network problem and disconnect and in the meantime my app will still be sending orders. I guess I can't have my cake and eat it so if I want to catch it sooner I'll have to up the heartbeat frequency, right?
>>
>>
>>Dermot
>>
>>
>>
>>
>>
>>
>>
>>----- Original Message -----
>>>From:Mike Gatny <mg...@co...>
>>>To: daw...@ya... 
>>>Cc:Grant Birchmeier <gbi...@co...>; "quickfix-developers@lists. sourceforge.net" <quickfix-developers@lists. sourceforge.net>
>>>Date:2017/4/13, Thu 23:55
>>>Subject:Re: Network disconnect recovery testing
>>> 
>>>
>>>Dermot,
>>>
>>>
>>>Have you set PersistMessages=Nin your SessionSettings? This is one way to prevent resend (i.e. cause GapFill over App-level msgs).
>>>
>>>
>>>Another way is to throw DoNotSendin toApp for possdup msgs:
>>>
>>>
>>>void App::toApp(FIX::Message& msg, const FIX::SessionID&) throw(FIX:: DoNotSend)
>>>>{
>>>>  try 
>>>>  {
>>>>    FIX::PossDupFlag possDupFlag;
>>>>    msg.getHeader().getField( possDupFlag);
>>>>    if (possDupFlag)
>>>>      throw FIX::DoNotSend();
>>>>  }
>>>>  catch (FIX::FieldNotFound&)
>>>>  { }
>>>>}
>>>
>>>
>>>As you say, this is almost certainly the behavior you want for sending orders, and that's the two main ways to achieve it.  If you are not doing either of those things, QF will indeed resend instead of sending GapFills.
>>>
>>>
>>>If you want to detect a connection problem, you can use onLogon() and onLogout() to alert or set some state.  You can think of onLogout() as "onDisconnect" as well -- it will get called even if you didn't get a clean Logout.
>>>Another way to do it on the fly is:
>>>
>>>
>>>FIX::Session * pSession = FIX::Session::lookupSession( sessionID);
>>>>if (pSession->isLoggedOn()) { /* ... */  }
>>>
>>>
>>>--Mike Gatny
>>>Connamara Systems, LLC
>>>
>>>On Thu, Apr 13, 2017 at 10:37 AM, <daw...@ya...>wrote:
>>>
>>>Hi Guys, I've been doing some recovery testing and I have a couple of questions.
>>>>
>>>>
>>>>Firstly, my setup is pretty simple - my trading application (initiator) and sell-side simulator (acceptor) are set up on two different networked machines, both are quickfix apps. Here's what I'm doing:- 
>>>>
>>>>
>>>>1. Establish a connection between initiator and acceptor.
>>>>2. Send some orders and ack them on the acceptor side.
>>>>2. Break the network connection on the initiator side.
>>>>3. Send some more orders from the initiator.
>>>>4. Fill existing orders on acceptor side.
>>>>5. Reconnect the initiator to the network.
>>>>
>>>>
>>>>At this point quickfix handles the reconnect and all the executions flow back to the initiator. However the new orders I sent from the initiator during the downtime are not resent to the executor [they are gap filled instead]. Is this the correct behaviour? Actually this is exactly the behaviour I want but I thought quickfix would resend them. So that's my first question.
>>>>
>>>>
>>>>My second question is what is the best way to determine a connection problem before sending any order?
>>>>
>>>>
>>>>Thanks!
>>>>Dermot
>>>
>>>
>>>
>
>
>

Re: [Quickfix-developers] Network disconnect recovery testing

[<daw...@ya...>]

Got it. Will take a look at that. Thanks a lot.
Dermot


----- Original Message -----
>From: Mike Gatny <mg...@co...>
>To: daw...@ya... 
>Cc: Grant Birchmeier <gbi...@co...>; "qui...@li..." <qui...@li...>
>Date: 2017/4/14, Fri 00:31
>Subject: Re: Network disconnect recovery testing
> 
>
>Dermot,
>
>This is one of the reasons the qf/j implementation has config options like JdbcLogHeartBeats/FileLogHeartbeats.
>I don't think it would be too hard to add this to MySQLLog for qf/c++.
>
>
>--Mike Gatny
>Connamara Systems, LLC
>
>On Thu, Apr 13, 2017 at 11:18 AM, <daw...@ya...> wrote:
>
>Hi Mike,
>>
>>
>>Thanks for the quick response. I haven't included PersistMessages in the config file but I can see it defaults to 'N' so that makes sense.
>>
>>
>>Ok noted on the onLogon()/onLogout(). Up to now I've just been catching the event text "Received logon response" and "Disconnecting" which is a bit hacky but the result is the same. The only issue I have is I've been trying to keep HeartBtInt=120 so as not to flood MySql with too many messages - this means that it takes a minute or two for quickfix to realise there's a network problem and disconnect and in the meantime my app will still be sending orders. I guess I can't have my cake and eat it so if I want to catch it sooner I'll have to up the heartbeat frequency, right?
>>
>>
>>Dermot
>>
>>
>>
>>
>>
>>
>>
>>----- Original Message -----
>>>From: Mike Gatny <mg...@co...>
>>>To: daw...@ya... 
>>>Cc: Grant Birchmeier <gbi...@co...>; "quickfix-developers@lists. sourceforge.net" <quickfix-developers@lists. sourceforge.net>
>>>Date: 2017/4/13, Thu 23:55
>>>Subject: Re: Network disconnect recovery testing
>>> 
>>>
>>>Dermot,
>>>
>>>
>>>Have you set PersistMessages=Nin your SessionSettings? This is one way to prevent resend (i.e. cause GapFill over App-level msgs).
>>>
>>>
>>>Another way is to throw DoNotSendin toApp for possdup msgs:
>>>
>>>
>>>void App::toApp(FIX::Message& msg, const FIX::SessionID&) throw(FIX:: DoNotSend)
>>>>{
>>>>  try 
>>>>  {
>>>>    FIX::PossDupFlag possDupFlag;
>>>>    msg.getHeader().getField( possDupFlag);
>>>>    if (possDupFlag)
>>>>      throw FIX::DoNotSend();
>>>>  }
>>>>  catch (FIX::FieldNotFound&)
>>>>  { }
>>>>}
>>>
>>>
>>>As you say, this is almost certainly the behavior you want for sending orders, and that's the two main ways to achieve it.  If you are not doing either of those things, QF will indeed resend instead of sending GapFills.
>>>
>>>
>>>If you want to detect a connection problem, you can use onLogon() and onLogout() to alert or set some state.  You can think of onLogout() as "onDisconnect" as well -- it will get called even if you didn't get a clean Logout.
>>>Another way to do it on the fly is:
>>>
>>>
>>>FIX::Session * pSession = FIX::Session::lookupSession( sessionID);
>>>>if (pSession->isLoggedOn()) { /* ... */  }
>>>
>>>
>>>--Mike Gatny
>>>Connamara Systems, LLC
>>>
>>>On Thu, Apr 13, 2017 at 10:37 AM, <daw...@ya...>wrote:
>>>
>>>Hi Guys, I've been doing some recovery testing and I have a couple of questions.
>>>>
>>>>
>>>>Firstly, my setup is pretty simple - my trading application (initiator) and sell-side simulator (acceptor) are set up on two different networked machines, both are quickfix apps. Here's what I'm doing:- 
>>>>
>>>>
>>>>1. Establish a connection between initiator and acceptor.
>>>>2. Send some orders and ack them on the acceptor side.
>>>>2. Break the network connection on the initiator side.
>>>>3. Send some more orders from the initiator.
>>>>4. Fill existing orders on acceptor side.
>>>>5. Reconnect the initiator to the network.
>>>>
>>>>
>>>>At this point quickfix handles the reconnect and all the executions flow back to the initiator. However the new orders I sent from the initiator during the downtime are not resent to the executor [they are gap filled instead]. Is this the correct behaviour? Actually this is exactly the behaviour I want but I thought quickfix would resend them. So that's my first question.
>>>>
>>>>
>>>>My second question is what is the best way to determine a connection problem before sending any order?
>>>>
>>>>
>>>>Thanks!
>>>>Dermot
>>>
>>>
>>>
>
>
>

Re: [Quickfix-developers] Network disconnect recovery testing

[Mike G. <mg...@co...>]

Dermot,

This is one of the reasons the qf/j implementation has config options like
JdbcLogHeartBeats/FileLogHeartbeats.
I don't think it would be too hard to add this to MySQLLog for qf/c++.

--
Mike Gatny
Connamara Systems, LLC

On Thu, Apr 13, 2017 at 11:18 AM, <daw...@ya...> wrote:

> Hi Mike,
>
> Thanks for the quick response. I haven't included PersistMessages in the
> config file but I can see it defaults to 'N' so that makes sense.
>
> Ok noted on the onLogon()/onLogout(). Up to now I've just been catching
> the event text "Received logon response" and "Disconnecting" which is a bit
> hacky but the result is the same. The only issue I have is I've been trying
> to keep HeartBtInt=120 so as not to flood MySql with too many messages -
> this means that it takes a minute or two for quickfix to realise there's a
> network problem and disconnect and in the meantime my app will still be
> sending orders. I guess I can't have my cake and eat it so if I want to
> catch it sooner I'll have to up the heartbeat frequency, right?
>
> Dermot
>
>
>
> ----- Original Message -----
> *From:* Mike Gatny <mg...@co...>
> *To:* daw...@ya...
> *Cc:* Grant Birchmeier <gbi...@co...>; "
> qui...@li..." <quickfix-developers@lists.
> sourceforge.net>
> *Date:* 2017/4/13, Thu 23:55
> *Subject:* Re: Network disconnect recovery testing
>
> Dermot,
>
> Have you set PersistMessages=N in your SessionSettings? This is one way
> to prevent resend (i.e. cause GapFill over App-level msgs).
>
> Another way is to throw DoNotSend in toApp for possdup msgs:
>
> *void App::toApp(FIX::Message& msg, const
> FIX::SessionID&) throw(FIX::DoNotSend)*
> *{*
> *  try *
> *  {*
> *    FIX::PossDupFlag possDupFlag;*
>     *msg**.getHeader().getField(possDupFlag);*
> *    if (possDupFlag)*
> *      throw FIX::DoNotSend();*
> *  }*
> *  catch (FIX::FieldNotFound&)*
> *  { }*
> *}*
>
>
> As you say, this is almost certainly the behavior you want for sending
> orders, and that's the two main ways to achieve it.  If you are not doing
> either of those things, QF will indeed resend instead of sending GapFills.
>
> If you want to detect a connection problem, you can use onLogon() and
> onLogout() to alert or set some state.  You can think of onLogout() as
> "onDisconnect" as well -- it will get called even if you didn't get a clean
> Logout.
> Another way to do it on the fly is:
>
> *FIX::Session * pSession = FIX::Session::lookupSession(sessionID);*
> *if (pSession->isLoggedOn()) { /* ... */  }*
>
>
> --
> Mike Gatny
> Connamara Systems, LLC
>
> On Thu, Apr 13, 2017 at 10:37 AM, <daw...@ya...> wrote:
>
> Hi Guys, I've been doing some recovery testing and I have a couple of
> questions.
>
> Firstly, my setup is pretty simple - my trading application (initiator)
> and sell-side simulator (acceptor) are set up on two different networked
> machines, both are quickfix apps. Here's what I'm doing:-
>
> 1. Establish a connection between initiator and acceptor.
> 2. Send some orders and ack them on the acceptor side.
> 2. Break the network connection on the initiator side.
> 3. Send some more orders from the initiator.
> 4. Fill existing orders on acceptor side.
> 5. Reconnect the initiator to the network.
>
> At this point quickfix handles the reconnect and all the executions flow
> back to the initiator. However the new orders I sent from the initiator
> during the downtime are not resent to the executor [they are gap filled
> instead]. Is this the correct behaviour? Actually this is exactly the
> behaviour I want but I thought quickfix would resend them. So that's my
> first question.
>
> My second question is what is the best way to determine a connection
> problem before sending any order?
>
> Thanks!
> Dermot
>
>
>
>
>

Re: [Quickfix-developers] Network disconnect recovery testing

[Mike G. <mg...@co...>]

Dermot,

Have you set PersistMessages=N in your SessionSettings? This is one way to
prevent resend (i.e. cause GapFill over App-level msgs).

Another way is to throw DoNotSend in toApp for possdup msgs:

*void App::toApp(FIX::Message& msg, const
FIX::SessionID&) throw(FIX::DoNotSend)*
*{*
*  try *
*  {*
*    FIX::PossDupFlag possDupFlag;*
    *msg**.getHeader().getField(possDupFlag);*
*    if (possDupFlag)*
*      throw FIX::DoNotSend();*
*  }*
*  catch (FIX::FieldNotFound&)*
*  { }*
*}*


As you say, this is almost certainly the behavior you want for sending
orders, and that's the two main ways to achieve it.  If you are not doing
either of those things, QF will indeed resend instead of sending GapFills.

If you want to detect a connection problem, you can use onLogon() and
onLogout() to alert or set some state.  You can think of onLogout() as
"onDisconnect" as well -- it will get called even if you didn't get a clean
Logout.
Another way to do it on the fly is:

*FIX::Session * pSession = FIX::Session::lookupSession(sessionID);*
*if (pSession->isLoggedOn()) { /* ... */  }*


--
Mike Gatny
Connamara Systems, LLC

On Thu, Apr 13, 2017 at 10:37 AM, <daw...@ya...> wrote:

> Hi Guys, I've been doing some recovery testing and I have a couple of
> questions.
>
> Firstly, my setup is pretty simple - my trading application (initiator)
> and sell-side simulator (acceptor) are set up on two different networked
> machines, both are quickfix apps. Here's what I'm doing:-
>
> 1. Establish a connection between initiator and acceptor.
> 2. Send some orders and ack them on the acceptor side.
> 2. Break the network connection on the initiator side.
> 3. Send some more orders from the initiator.
> 4. Fill existing orders on acceptor side.
> 5. Reconnect the initiator to the network.
>
> At this point quickfix handles the reconnect and all the executions flow
> back to the initiator. However the new orders I sent from the initiator
> during the downtime are not resent to the executor [they are gap filled
> instead]. Is this the correct behaviour? Actually this is exactly the
> behaviour I want but I thought quickfix would resend them. So that's my
> first question.
>
> My second question is what is the best way to determine a connection
> problem before sending any order?
>
> Thanks!
> Dermot
>

Re: [Quickfix-developers] Network disconnect recovery testing

[<daw...@ya...>]

Hi Mike,

Thanks for the quick response. I haven't included PersistMessages in the config file but I can see it defaults to 'N' so that makes sense.

Ok noted on the onLogon()/onLogout(). Up to now I've just been catching the event text "Received logon response" and "Disconnecting" which is a bit hacky but the result is the same. The only issue I have is I've been trying to keep HeartBtInt=120 so as not to flood MySql with too many messages - this means that it takes a minute or two for quickfix to realise there's a network problem and disconnect and in the meantime my app will still be sending orders. I guess I can't have my cake and eat it so if I want to catch it sooner I'll have to up the heartbeat frequency, right?

Dermot





----- Original Message -----
>From: Mike Gatny <mg...@co...>
>To: daw...@ya... 
>Cc: Grant Birchmeier <gbi...@co...>; "qui...@li..." <qui...@li...>
>Date: 2017/4/13, Thu 23:55
>Subject: Re: Network disconnect recovery testing
> 
>
>Dermot,
>
>
>Have you set PersistMessages=Nin your SessionSettings? This is one way to prevent resend (i.e. cause GapFill over App-level msgs).
>
>
>Another way is to throw DoNotSendin toApp for possdup msgs:
>
>
>void App::toApp(FIX::Message& msg, const FIX::SessionID&) throw(FIX::DoNotSend)
>>{
>>  try 
>>  {
>>    FIX::PossDupFlag possDupFlag;
>>    msg.getHeader().getField(possDupFlag);
>>    if (possDupFlag)
>>      throw FIX::DoNotSend();
>>  }
>>  catch (FIX::FieldNotFound&)
>>  { }
>>}
>
>
>As you say, this is almost certainly the behavior you want for sending orders, and that's the two main ways to achieve it.  If you are not doing either of those things, QF will indeed resend instead of sending GapFills.
>
>
>If you want to detect a connection problem, you can use onLogon() and onLogout() to alert or set some state.  You can think of onLogout() as "onDisconnect" as well -- it will get called even if you didn't get a clean Logout.
>Another way to do it on the fly is:
>
>
>FIX::Session * pSession = FIX::Session::lookupSession(sessionID);
>>if (pSession->isLoggedOn()) { /* ... */  }
>
>
>--Mike Gatny
>Connamara Systems, LLC
>
>On Thu, Apr 13, 2017 at 10:37 AM, <daw...@ya...>wrote:
>
>Hi Guys, I've been doing some recovery testing and I have a couple of questions.
>>
>>
>>Firstly, my setup is pretty simple - my trading application (initiator) and sell-side simulator (acceptor) are set up on two different networked machines, both are quickfix apps. Here's what I'm doing:- 
>>
>>
>>1. Establish a connection between initiator and acceptor.
>>2. Send some orders and ack them on the acceptor side.
>>2. Break the network connection on the initiator side.
>>3. Send some more orders from the initiator.
>>4. Fill existing orders on acceptor side.
>>5. Reconnect the initiator to the network.
>>
>>
>>At this point quickfix handles the reconnect and all the executions flow back to the initiator. However the new orders I sent from the initiator during the downtime are not resent to the executor [they are gap filled instead]. Is this the correct behaviour? Actually this is exactly the behaviour I want but I thought quickfix would resend them. So that's my first question.
>>
>>
>>My second question is what is the best way to determine a connection problem before sending any order?
>>
>>
>>Thanks!
>>Dermot
>
>
>