quickfix-developers

Re: [Quickfix-developers] SequenceReset bug?

[<or...@qu...>]

I'm not sure I see this as a bug in quickfix, but rather a bug in the
counterprty.  Let me make sure I understand what you are saying.

You receive message 508, but expect 507
508 is queued
you request all messages up to 507
you are now expecting 508
the queued 508 message, which is the expected number, is processed
you should now expect 509, since you have process 508 which was not part
of the resend request
They send you a new message with sequence number 508, completely
different from the previous message they tagged with 508. Eep!  That's
not allowed.

> -------- Original Message --------
> Subject: [Quickfix-developers] SequenceReset bug?
> From: Kbo Keplowsky <kbo...@gm...>
> Date: Sun, June 22, 2008 9:21 am
> To: qui...@li...
> QuickFIX Documentation: http://www.quickfixengine.org/quickfix/doc/html/index.html
> QuickFIX Support: http://www.quickfixengine.org/services.html
> Hi all,
> I've come across an issue that looks like a bug to me - i.e. not in accordance 
> with the FIX protocol.
> My system (QuickFix initiator) logs in, only to find out that the sequence 
> number is too high (508 instead of 507).
> That's cool, so it queues the current message (#508) and sends a resend 
> request (0 to 507).
> The counterparty responds, sending a sequence reset from 507 to 508 (basically 
> saying there's nothing to send, and that 508 is the next sequence number, I 
> assume).
> Now here's the problem - my system is satisfied, sets the next expected number 
> to 508, and processes the queued message (#508, remember?). This, however, 
> advances the sequence number...
> On the next heartbeat, my system gets a heartbeat response, with a sequence 
> number of (did you guess?) 508... which is reasonable for the counterparty, 
> but too low for my system, and so it disconnects. This can go on forever.
> What should I do? I've read the FIX specs, and it explicitly say that the 
> NewSeqNum field in the SequenceReset message is the next number sent by the 
> counterparty, so I believe the bug is with QuickFix, talking into account the 
> seqnum of the queued message...
> Help, please...
> Thanks,
> Kbo
> -------------------------------------------------------------------------
> Check out the new SourceForge.net Marketplace.
> It's the best place to buy or sell services for
> just about anything Open Source.
> http://sourceforge.net/services/buy/index.php
> _______________________________________________
> Quickfix-developers mailing list
> Qui...@li...
> https://lists.sourceforge.net/lists/listinfo/quickfix-developers

Re: [Quickfix-developers] SequenceReset bug?

[<or...@qu...>]

> Hi,
> Thanks for the quick reply.
> > you request all messages up to 507
> Yes. And I receive them. However, the last message sent from the counterparty 
> is a Sequence Reset message specifying NewSeqNo=508, meaning the next message 
> they will send (they do not know nor care about the queue on my side, 
> obviously) will have a sequence number of 508.

No.  That is not what it means.  All it means is that they have
completed the resend request up to message 507.  It does not mean that
508 is the next message you should be expecting.  You already received
508, and unless you explicitly request 508, they should not be sending
it to you again.  They don't need to know about your queue, they just
need to know they already sent the message.  You can do a resend request
any time for any reason.  If the last message you received was 10,000,
and you decide on a whim you want to see messages 20-50 again, this does
not mean they can start sending you new messages starting with 51.  One
has nothing to do with the other.

> I assume that by that point QuickFix does expect 508, but I guess that 
> processing the queued message causes it to expect 509 - this seems to 
> contradict FIX as it modifies expectations from the counterparty accoring to a 
> unilateral event (i.e. processing something queues on my side, older than the 
> last message received from the counterparty).

Not only does the behavior not contradict the expectations of the
protocol, it explicitly adheres to a reccomendation in the
specification:

"It is also recommended that an engine should store out of sequence
messages in a temporary queue and
process them in order when the gap is closed. This prevents generating
resend requests for n->m, n->m+1,
n->m+2, ... which can result in many resent PossDupFlag=Y messages."

> I've checked with the counterparty, and this is the way they work with all of 
> their customers...
> What do you say?

I say they are incorrect in their behavior if it is as you described and
they are.  Now, can we add a setting that will turn off queing so it can
adapt to this behavior?  Absolutely, and I think we will do this because
we want to be able to communicate with whatever is out there even if not
completely correct.

Re: [Quickfix-developers] SequenceReset bug?

[Kbo K. <kbo...@gm...>]

I see what you mean, and I agree. However, I thought I'd attach the relevant 
part of the log just to make sure I haven't made a mistake in my description 
(should have done that from the start, I guess).

In case I will have to adapt to the counterparty's behavior, is there any 
quick way to turn off queueing?

Here goes:

<20080622-13:28:53, FIX.4.4:ME->CNTRPRTY:0, outgoing>
 (8=FIX.4.4 9=105 35=A 34=192 49=ME 52=20080622-13:28:53.466
56=CNTRPRTY 50=user 98=0 108=20 553=user 554=pass 10=071 )
<20080622-13:28:53, FIX.4.4:ME->CNTRPRTY:0, event>
 (Initiated logon request)
<20080622-13:28:53, FIX.4.4:ME->CNTRPRTY:0, incoming>
 (8=FIX.4.4 9=82 35=A 34=508 49=CNTRPRTY 50=FixGateway 56=ME
52=20080622-13:32:01.445 98=0 108=20 10=072 )
<20080622-13:28:53, FIX.4.4:ME->CNTRPRTY:0, event>
 (Received logon response)
<20080622-13:28:53, FIX.4.4:ME->CNTRPRTY:0, event>
 (MsgSeqNum too high, expecting 507 but received 508)
<20080622-13:28:53, FIX.4.4:ME->CNTRPRTY:0, outgoing>
 (8=FIX.4.4 9=104 35=2 34=193 49=ME 52=20080622-13:28:53.716
56=CNTRPRTY 7=507 16=0 50=user 553=user 554=pass 10=004 )
<20080622-13:28:53, FIX.4.4:ME->CNTRPRTY:0, event>
 (Sent ResendRequest FROM: 507 TO: 0)
<20080622-13:28:53, FIX.4.4:ME->CNTRPRTY:0, incoming>
 (8=FIX.4.4 9=74 35=4 49=CNTRPRTY 56=ME 34=507 43=Y
52=20080622-13:32:01.710 123=Y 36=508 10=241 )
<20080622-13:28:53, FIX.4.4:ME->CNTRPRTY:0, event>
 (ResendRequest for messages FROM: 507 TO: 507 has been satisfied.)
<20080622-13:28:53, FIX.4.4:ME->CNTRPRTY:0, event>
 (Received SequenceReset FROM: 507 TO: 508)
<20080622-13:28:53, FIX.4.4:ME->CNTRPRTY:0, event>
 (Processing QUEUED message: 508)
<20080622-13:29:13, FIX.4.4:ME->CNTRPRTY:0, outgoing>
 (8=FIX.4.4 9=93 35=0 34=194 49=ME 52=20080622-13:29:13.248
56=CNTRPRTY 50=user 553=user 554=pass 10=241 )
<20080622-13:29:13, FIX.4.4:ME->CNTRPRTY:0, incoming>
 (8=FIX.4.4 9=70 35=0 34=508 49=CNTRPRTY 50=FixGateway 56=ME
52=20080622-13:32:21.710 10=025 )
<20080622-13:29:13, FIX.4.4:ME->CNTRPRTY:0, outgoing>
 (8=FIX.4.4 9=146 35=5 34=195 49=ME 52=20080622-13:29:13.951
56=CNTRPRTY 50=user 58=MsgSeqNum too low, expecting 509 but received
508 553=user 554=pass 10=150 )
<20080622-13:29:13, FIX.4.4:ME->CNTRPRTY:0, event>
 (MsgSeqNum too low, expecting 509 but received 508)
<20080622-13:29:13, FIX.4.4:ME->CNTRPRTY:0, event>
 (Disconnecting)

Thanks,

Kbo

Re: [Quickfix-developers] SequenceReset bug?

[<or...@qu...>]

There is not a setting for it currently, but you can go to Session.cpp
in the doTargetTooHigh method, and comment out this line: m_state.queue(
msgSeqNum, msg );

I believe that should do it.

> -------- Original Message --------
> Subject: Re: [Quickfix-developers] SequenceReset bug?
> From: Kbo Keplowsky <kbo...@gm...>
> Date: Fri, June 27, 2008 2:57 am
> To: qui...@li...
> 
> 
> QuickFIX Documentation: http://www.quickfixengine.org/quickfix/doc/html/index.html
> QuickFIX Support: http://www.quickfixengine.org/services.html
> 
> I see what you mean, and I agree. However, I thought I'd attach the relevant 
> part of the log just to make sure I haven't made a mistake in my description 
> (should have done that from the start, I guess).
> 
> In case I will have to adapt to the counterparty's behavior, is there any 
> quick way to turn off queueing?
> 
> Here goes:
> 
> <20080622-13:28:53, FIX.4.4:ME->CNTRPRTY:0, outgoing>
>  (8=FIX.4.4 9=105 35=A 34=192 49=ME 52=20080622-13:28:53.466
> 56=CNTRPRTY 50=user 98=0 108=20 553=user 554=pass 10=071 )
> <20080622-13:28:53, FIX.4.4:ME->CNTRPRTY:0, event>
>  (Initiated logon request)
> <20080622-13:28:53, FIX.4.4:ME->CNTRPRTY:0, incoming>
>  (8=FIX.4.4 9=82 35=A 34=508 49=CNTRPRTY 50=FixGateway 56=ME
> 52=20080622-13:32:01.445 98=0 108=20 10=072 )
> <20080622-13:28:53, FIX.4.4:ME->CNTRPRTY:0, event>
>  (Received logon response)
> <20080622-13:28:53, FIX.4.4:ME->CNTRPRTY:0, event>
>  (MsgSeqNum too high, expecting 507 but received 508)
> <20080622-13:28:53, FIX.4.4:ME->CNTRPRTY:0, outgoing>
>  (8=FIX.4.4 9=104 35=2 34=193 49=ME 52=20080622-13:28:53.716
> 56=CNTRPRTY 7=507 16=0 50=user 553=user 554=pass 10=004 )
> <20080622-13:28:53, FIX.4.4:ME->CNTRPRTY:0, event>
>  (Sent ResendRequest FROM: 507 TO: 0)
> <20080622-13:28:53, FIX.4.4:ME->CNTRPRTY:0, incoming>
>  (8=FIX.4.4 9=74 35=4 49=CNTRPRTY 56=ME 34=507 43=Y
> 52=20080622-13:32:01.710 123=Y 36=508 10=241 )
> <20080622-13:28:53, FIX.4.4:ME->CNTRPRTY:0, event>
>  (ResendRequest for messages FROM: 507 TO: 507 has been satisfied.)
> <20080622-13:28:53, FIX.4.4:ME->CNTRPRTY:0, event>
>  (Received SequenceReset FROM: 507 TO: 508)
> <20080622-13:28:53, FIX.4.4:ME->CNTRPRTY:0, event>
>  (Processing QUEUED message: 508)
> <20080622-13:29:13, FIX.4.4:ME->CNTRPRTY:0, outgoing>
>  (8=FIX.4.4 9=93 35=0 34=194 49=ME 52=20080622-13:29:13.248
> 56=CNTRPRTY 50=user 553=user 554=pass 10=241 )
> <20080622-13:29:13, FIX.4.4:ME->CNTRPRTY:0, incoming>
>  (8=FIX.4.4 9=70 35=0 34=508 49=CNTRPRTY 50=FixGateway 56=ME
> 52=20080622-13:32:21.710 10=025 )
> <20080622-13:29:13, FIX.4.4:ME->CNTRPRTY:0, outgoing>
>  (8=FIX.4.4 9=146 35=5 34=195 49=ME 52=20080622-13:29:13.951
> 56=CNTRPRTY 50=user 58=MsgSeqNum too low, expecting 509 but received
> 508 553=user 554=pass 10=150 )
> <20080622-13:29:13, FIX.4.4:ME->CNTRPRTY:0, event>
>  (MsgSeqNum too low, expecting 509 but received 508)
> <20080622-13:29:13, FIX.4.4:ME->CNTRPRTY:0, event>
>  (Disconnecting)
> 
> Thanks,
> 
> Kbo
> 
> 
> -------------------------------------------------------------------------
> Check out the new SourceForge.net Marketplace.
> It's the best place to buy or sell services for
> just about anything Open Source.
> http://sourceforge.net/services/buy/index.php
> _______________________________________________
> Quickfix-developers mailing list
> Qui...@li...
> https://lists.sourceforge.net/lists/listinfo/quickfix-developers

Re: [Quickfix-developers] SequenceReset bug?

[James D. <jc...@co...>]

It might be interesting to know who is the counter party  with this
behavior. If it is a big enough player, like an exchange, it might be worth
creating a setting.


On Fri, Jun 27, 2008 at 8:23 AM, <or...@qu...> wrote:

> QuickFIX Documentation:
> http://www.quickfixengine.org/quickfix/doc/html/index.html
> QuickFIX Support: http://www.quickfixengine.org/services.html
>
> There is not a setting for it currently, but you can go to Session.cpp
> in the doTargetTooHigh method, and comment out this line: m_state.queue(
> msgSeqNum, msg );
>
> I believe that should do it.
>
> > -------- Original Message --------
> > Subject: Re: [Quickfix-developers] SequenceReset bug?
> > From: Kbo Keplowsky <kbo...@gm...>
> > Date: Fri, June 27, 2008 2:57 am
> > To: qui...@li...
> >
> >
> > QuickFIX Documentation:
> http://www.quickfixengine.org/quickfix/doc/html/index.html
> > QuickFIX Support: http://www.quickfixengine.org/services.html
> >
> > I see what you mean, and I agree. However, I thought I'd attach the
> relevant
> > part of the log just to make sure I haven't made a mistake in my
> description
> > (should have done that from the start, I guess).
> >
> > In case I will have to adapt to the counterparty's behavior, is there any
> > quick way to turn off queueing?
> >
> > Here goes:
> >
> > <20080622-13:28:53, FIX.4.4:ME->CNTRPRTY:0, outgoing>
> >  (8=FIX.4.4 9=105 35=A 34=192 49=ME 52=20080622-13:28:53.466
> > 56=CNTRPRTY 50=user 98=0 108=20 553=user 554=pass 10=071 )
> > <20080622-13:28:53, FIX.4.4:ME->CNTRPRTY:0, event>
> >  (Initiated logon request)
> > <20080622-13:28:53, FIX.4.4:ME->CNTRPRTY:0, incoming>
> >  (8=FIX.4.4 9=82 35=A 34=508 49=CNTRPRTY 50=FixGateway 56=ME
> > 52=20080622-13:32:01.445 98=0 108=20 10=072 )
> > <20080622-13:28:53, FIX.4.4:ME->CNTRPRTY:0, event>
> >  (Received logon response)
> > <20080622-13:28:53, FIX.4.4:ME->CNTRPRTY:0, event>
> >  (MsgSeqNum too high, expecting 507 but received 508)
> > <20080622-13:28:53, FIX.4.4:ME->CNTRPRTY:0, outgoing>
> >  (8=FIX.4.4 9=104 35=2 34=193 49=ME 52=20080622-13:28:53.716
> > 56=CNTRPRTY 7=507 16=0 50=user 553=user 554=pass 10=004 )
> > <20080622-13:28:53, FIX.4.4:ME->CNTRPRTY:0, event>
> >  (Sent ResendRequest FROM: 507 TO: 0)
> > <20080622-13:28:53, FIX.4.4:ME->CNTRPRTY:0, incoming>
> >  (8=FIX.4.4 9=74 35=4 49=CNTRPRTY 56=ME 34=507 43=Y
> > 52=20080622-13:32:01.710 123=Y 36=508 10=241 )
> > <20080622-13:28:53, FIX.4.4:ME->CNTRPRTY:0, event>
> >  (ResendRequest for messages FROM: 507 TO: 507 has been satisfied.)
> > <20080622-13:28:53, FIX.4.4:ME->CNTRPRTY:0, event>
> >  (Received SequenceReset FROM: 507 TO: 508)
> > <20080622-13:28:53, FIX.4.4:ME->CNTRPRTY:0, event>
> >  (Processing QUEUED message: 508)
> > <20080622-13:29:13, FIX.4.4:ME->CNTRPRTY:0, outgoing>
> >  (8=FIX.4.4 9=93 35=0 34=194 49=ME 52=20080622-13:29:13.248
> > 56=CNTRPRTY 50=user 553=user 554=pass 10=241 )
> > <20080622-13:29:13, FIX.4.4:ME->CNTRPRTY:0, incoming>
> >  (8=FIX.4.4 9=70 35=0 34=508 49=CNTRPRTY 50=FixGateway 56=ME
> > 52=20080622-13:32:21.710 10=025 )
> > <20080622-13:29:13, FIX.4.4:ME->CNTRPRTY:0, outgoing>
> >  (8=FIX.4.4 9=146 35=5 34=195 49=ME 52=20080622-13:29:13.951
> > 56=CNTRPRTY 50=user 58=MsgSeqNum too low, expecting 509 but received
> > 508 553=user 554=pass 10=150 )
> > <20080622-13:29:13, FIX.4.4:ME->CNTRPRTY:0, event>
> >  (MsgSeqNum too low, expecting 509 but received 508)
> > <20080622-13:29:13, FIX.4.4:ME->CNTRPRTY:0, event>
> >  (Disconnecting)
> >
> > Thanks,
> >
> > Kbo
> >
> >
> > -------------------------------------------------------------------------
> > Check out the new SourceForge.net Marketplace.
> > It's the best place to buy or sell services for
> > just about anything Open Source.
> > http://sourceforge.net/services/buy/index.php
> > _______________________________________________
> > Quickfix-developers mailing list
> > Qui...@li...
> > https://lists.sourceforge.net/lists/listinfo/quickfix-developers
>
>
> -------------------------------------------------------------------------
> Check out the new SourceForge.net Marketplace.
> It's the best place to buy or sell services for
> just about anything Open Source.
> http://sourceforge.net/services/buy/index.php
> _______________________________________________
> Quickfix-developers mailing list
> Qui...@li...
> https://lists.sourceforge.net/lists/listinfo/quickfix-developers
>

Re: [Quickfix-developers] SequenceReset bug?

[Kbo K. <kbo...@gm...>]

> There is not a setting for it currently, but you can go to Session.cpp
> in the doTargetTooHigh method, and comment out this line: m_state.queue(
> msgSeqNum, msg );
> 
> I believe that should do it.

Thanks.

I would like, however, to make sure I understand your point, since modifying 
the DLL is not my first choice - I do trust you guys...

I looked at the description of the Sequence Reset Gap Fill message in 
fixionary.com, and it states (for all types of this message):
"The sending application will initiate the Sequence Reset. The message in all 
situations specifies NewSeqNo to reset to as the value of the next sequence 
number to be expected by the message receipient immediately following the 
messages and/or sequence numbers being skipped."

Are you saying that since the message in question explicitly specifies a Gap 
Fill mode (123=Y), then it must fit with the previously sent messages (those 
for which a resend was not requested), whether queued or not (since queuing is 
recommended but not mandatory)?

This would suggest that in Gap-Fill mode, this message cannot be used to 
arbitrarily reset the sequence number by the counterparty (it would take 123=N 
to do that) - is that correct?

Please understand I'm asking this in order to be able to report the problem to 
my counterparty in the form of a FIX protocol breach (otherwise I might be 
kicked down the stairs). Is there anywhere I can get an official quote that 
specifies this behavior?

Thanks again,

Kbo

Re: [Quickfix-developers] SequenceReset bug?

[Kbo K. <kbo...@gm...>]

 <oren@...> writes:
> There is not a setting for it currently, but you can go to Session.cpp
> in the doTargetTooHigh method, and comment out this line: m_state.queue(
> msgSeqNum, msg );
> 
> I believe that should do it.

Just did that, and it seems to work. However, I am not sure the engine 
processes the 'too high' message at all (call it message N) - it seems to, but 
I fail to understand how since I don't queue it, and the resend request is 
only for 0 to N-1.

Can anyone please explain?

Thanks,

Kbo

Re: [Quickfix-developers] SequenceReset bug?

[<or...@qu...>]

My suggestion would be to pose the question to the fix protocol message
boards.  Unfortunately the wording of the spec can sometimes be a little
ambiguous and gets subjected to different interpretations.  I think
members of the FPL can best clarify the proper behavior for everyone.

--oren

> -------- Original Message --------
> Subject: Re: [Quickfix-developers] SequenceReset bug?
> From: Kbo Keplowsky <kbo...@gm...>
> Date: Sun, June 29, 2008 4:49 pm
> To: qui...@li...
> QuickFIX Documentation: http://www.quickfixengine.org/quickfix/doc/html/index.html
> QuickFIX Support: http://www.quickfixengine.org/services.html
> > There is not a setting for it currently, but you can go to Session.cpp
> > in the doTargetTooHigh method, and comment out this line: m_state.queue(
> > msgSeqNum, msg );
> > 
> > I believe that should do it.
> Thanks.
> I would like, however, to make sure I understand your point, since modifying 
> the DLL is not my first choice - I do trust you guys...
> I looked at the description of the Sequence Reset Gap Fill message in 
> fixionary.com, and it states (for all types of this message):
> "The sending application will initiate the Sequence Reset. The message in all 
> situations specifies NewSeqNo to reset to as the value of the next sequence 
> number to be expected by the message receipient immediately following the 
> messages and/or sequence numbers being skipped."
> Are you saying that since the message in question explicitly specifies a Gap 
> Fill mode (123=Y), then it must fit with the previously sent messages (those 
> for which a resend was not requested), whether queued or not (since queuing is 
> recommended but not mandatory)?
> This would suggest that in Gap-Fill mode, this message cannot be used to 
> arbitrarily reset the sequence number by the counterparty (it would take 123=N 
> to do that) - is that correct?
> Please understand I'm asking this in order to be able to report the problem to 
> my counterparty in the form of a FIX protocol breach (otherwise I might be 
> kicked down the stairs). Is there anywhere I can get an official quote that 
> specifies this behavior?
> Thanks again,
> Kbo
> -------------------------------------------------------------------------
> Check out the new SourceForge.net Marketplace.
> It's the best place to buy or sell services for
> just about anything Open Source.
> http://sourceforge.net/services/buy/index.php
> _______________________________________________
> Quickfix-developers mailing list
> Qui...@li...
> https://lists.sourceforge.net/lists/listinfo/quickfix-developers

Re: [Quickfix-developers] SequenceReset bug?

[Kbo K. <kbo...@gm...>]

Hi,

Thanks for the quick reply.

> you request all messages up to 507

Yes. And I receive them. However, the last message sent from the counterparty 
is a Sequence Reset message specifying NewSeqNo=508, meaning the next message 
they will send (they do not know nor care about the queue on my side, 
obviously) will have a sequence number of 508.

I assume that by that point QuickFix does expect 508, but I guess that 
processing the queued message causes it to expect 509 - this seems to 
contradict FIX as it modifies expectations from the counterparty accoring to a 
unilateral event (i.e. processing something queues on my side, older than the 
last message received from the counterparty).

I've checked with the counterparty, and this is the way they work with all of 
their customers...

What do you say?

Thanks again,

Kbo