quickfix-developers

[Quickfix-developers] non logged-on messages

[John G. <joh...@wa...>]

Hi there,

Using C++ version of QF 1.10.2. on linux.

I don't get what I am doing wrong here. The documentation at 
http://www.quickfixengine.org/quickfix/doc/html/application.html 
stipulates that " As soon as a session is created, you can begin sending 
messages to it. If no one is logged on, the messages will be sent at the 
time a connection is established with the counterparty."

I launch an Initiator, and a trace in onCreate() says the session has been 
created. I send an order to it, which gets written in the directory 
defined by FileStorePath with the name 
BeginString-SenderCompID-TargetCompID.body

Then I launch the ordermatch.exe test program and here is what happens 
from its side :

<20050829-09:47:53, FIX.4.1:JGSERVER->JGCLIENT, event>
(Created session) 

<20050829-09:48:10, FIX.4.1:JGSERVER->JGCLIENT, incoming>
 (8=FIX.4.19=6735=A34=249=JGCLIENT52=20050829-09:48:1056=JGSERVER98=0108=6010=254)

<20050829-09:48:10, FIX.4.1:JGSERVER->JGCLIENT, event>
(Received logon request) 

<20050829-09:48:10, FIX.4.1:JGSERVER->JGCLIENT,outgoing>
  
(8=FIX.4.19=6735=A34=149=JGSERVER52=20050829-09:48:1056=JGCLIENT98=0108=6010=253)
<20050829-09:48:10, FIX.4.1:JGSERVER->JGCLIENT, event>
  (Responding to logon request) 
<20050829-09:48:10, FIX.4.1:JGSERVER->JGCLIENT, event>
  (MsgSeqNum too high, expecting 1 but received 2) 
<20050829-09:48:10, FIX.4.1:JGSERVER->JGCLIENT, outgoing>
(8=FIX.4.19=6935=234=249=JGSERVER52=20050829-09:48:1056=JGCLIENT7=116=99999910=118)
<20050829-09:48:10, FIX.4.1:JGSERVER->JGCLIENT, event>
  (Sent ResendRequest FROM: 1 TO: 999999) 
Logon - FIX.4.1:JGSERVER->JGCLIENT <20050829-09:48:10, 
FIX.4.1:JGSERVER->JGCLIENT,incoming>
(8=FIX.4.19=9335=434=143=Y49=JGCLIENT52=20050829-09:48:1056=JGSERVER122=20050829-09:48:1036=3123=Y10=028)
<20050829-09:48:10, FIX.4.1:JGSERVER->JGCLIENT, event>
(Received SequenceReset FROM: 1 TO: 3)

The same kind of things happens in the following scenario :
- I let both initiator and ordermatch(acceptor) chat a little while
- I kill ordermatch with a CTRL-C
- I send my order to the initiator
- I boot up ordermatch again 
=> ordermatch asks for ResendRequest but the client never sends its order, 
only a SequenceReset

Is there something special I must do in my code to handle this situation 
or is it a "bug" of some kind ?

If I start ordermatch first, everything works just fine. 

Sincerely,
JG

Re: [Quickfix-developers] non logged-on messages

[Caleb E. <cal...@gm...>]

On 8/29/05, John GALLET <joh...@wa...> wrote:

> The same kind of things happens in the following scenario :
> - I let both initiator and ordermatch(acceptor) chat a little while
> - I kill ordermatch with a CTRL-C
> - I send my order to the initiator

How's that?  The Initiator class doesn't provide any methods for
sending messages.  Only the Session class does.

> - I boot up ordermatch again
> =3D> ordermatch asks for ResendRequest but the client never sends its ord=
er,
> only a SequenceReset
=20
> Is there something special I must do in my code to handle this situation
> or is it a "bug" of some kind ?

No, this sort of thing Just Works.

Its hard to tell what is happening in your tests without more
information.  Just the event log isn't enough.

It would be helpful if you could include the code you'e using and
configuration you have for the session, as well as the logs from the
FileStore (e.g. at least the .body and .header files).  Are you
certain you've sent a NewOrderSingle and not some admin-type message?

> If I start ordermatch first, everything works just fine.

Perhaps you could provide a log from one of these scenarios.

--=20
Caleb Epstein
caleb dot epstein at gmail dot com

Re: [Quickfix-developers] non logged-on messages

[John G. <joh...@wa...>]

Hi and thanks for your answer,

> > - I send my order to the initiator
> 
> How's that?  The Initiator class doesn't provide any methods for
> sending messages.  Only the Session class does.

I was refering to the exe, not the implementation, sorry for not being 
clear.

> No, this sort of thing Just Works.
I can't make it. So something must be wrong in my code, but I don't know 
where to start hunting for it.

> Perhaps you could provide a log from one of these scenarios.

You can find at http://www.saphirtech.com/QF/ a tarball
non-logged-on-messages.tar.gz that includes the programs tradeclient and
ordermatch that I modified so they would launch correctly (can't make the
shell wrapper work) and work on a FIX 4.1 standard (my clients'
requirements, don't ask why). They are pretty close to the ones in 
examples/ distrib.
 
If you don't compile on linux, I renamed "config.h" to qf_config.h 
because we have zillions of files named this way in here and one more did 
not appeal to me, but it's the same one.

You should just have to modify the path of QF and LIBXML in the scripts 
named "compile", and maybe the port (5556 in my config files).

Both QF_CLI.TXT match one another. 

Here is what I do : 

1) start the command line client. You'll notice the "SESSION CREATED" 
trace added in onCreate() immediatly on startup.

2) manually input a FIX 41 buy order, limited, GTD. SenderCompID is 
"JGCLIENT" and TargetCompId is "JGSERVER" if you use the provided 
QF_CLI.TXTs files. No subID.

3) Now answer Y(es) to "send order". 

OUT: 
8=FIX.4.19=13735=D34=149=JGCLIENT52=20050830-06:39:1356=JGSERVER
1=my_account11=myOrderID21=138=240=244=1.3554=155=FR00013000759=0207=02510=197  

4) lets launch the ordermatch part that I called vmarket.exe and wait 
until they synchronise. 

SESSION CREATED. 
<20050830-06:39:53, FIX.4.1:JGSERVER->JGCLIENT, event>
  (Created session) 
<20050830-06:40:13, FIX.4.1:JGSERVER->JGCLIENT,incoming>
  (8=FIX.4.19=6735=A34=249=JGCLIENT52=20050830-06:40:1356=JGSERVER98=0108=4010=236)
<20050830-06:40:13, FIX.4.1:JGSERVER->JGCLIENT, event>
  (Received logon request) 
<20050830-06:40:13, FIX.4.1:JGSERVER->JGCLIENT,outgoing>
(8=FIX.4.19=6735=A34=149=JGSERVER52=20050830-06:40:1356=JGCLIENT98=0108=4010=235)
<20050830-06:40:13, FIX.4.1:JGSERVER->JGCLIENT, event>
  (Responding to logon request) 
<20050830-06:40:13, FIX.4.1:JGSERVER->JGCLIENT, event>
  (MsgSeqNum too high, expecting 1 but received 2) <20050830-06:40:13,
FIX.4.1:JGSERVER->JGCLIENT, outgoing>
(8=FIX.4.19=6935=234=249=JGSERVER52=20050830-06:40:1356=JGCLIENT7=116=99999910=102)
<20050830-06:40:13, FIX.4.1:JGSERVER->JGCLIENT, event>
  (Sent ResendRequest FROM: 1 TO: 999999) 
Logon - FIX.4.1:JGSERVER->JGCLIENT 
<20050830-06:40:13, FIX.4.1:JGSERVER->JGCLIENT,incoming>
(8=FIX.4.19=9335=434=143=Y49=JGCLIENT52=20050830-06:40:1356=JGSERVER122=20050830-06:40:1336=3123=Y10=252)
<20050830-06:40:13, FIX.4.1:JGSERVER->JGCLIENT, event>
  (Received SequenceReset FROM: 1 TO: 3)   

And from then on, only heartbeats are exchanged. The order was never sent. 
If I input a new order, using another security to distinguish them, no 
problem, the new is sent (and rejected because whatever tag is missing, 
but that's another story). 

I must have done something wrong, but I don't understand what. 

Sincerely,
JG

Re: [Quickfix-developers] non logged-on messages

[Caleb E. <cal...@gm...>]

Here's the problem in the example code (tradeclient/Application.cpp):

void Application::toApp( FIX::Message& message, const FIX::SessionID&=20
sessionID )
throw( FIX::DoNotSend )
{
try
{
FIX::PossDupFlag possDupFlag;
message.getHeader().getField( possDupFlag );
 if ( possDupFlag ) throw FIX::DoNotSend();
}
catch ( FIX::FieldNotFound& ) {}

std::cout << std::endl
<< "OUT: " << message << std::endl;
}

This application will *never* resend messages, since they always have=20
PossDupFlag=3DY

--=20
Caleb Epstein
caleb dot epstein at gmail dot com

Re: [Quickfix-developers] non logged-on messages

[John G. <joh...@wa...>]

Hi there,

> Here's the problem in the example code (tradeclient/Application.cpp):
Thanks VERY MUCH for the explanation.

> This application will *never* resend messages, since they always have 
> PossDupFlag=Y

So the way it works is that the messages are written to the sending queue 
with PossDupFlag=Y ? I would never have guessed the logic behind that. Is 
there some documentation explaining the inner works of QF for message 
exchanges that I missed ? 

Thanks again.

Sincerely,
JG

Re: [Quickfix-developers] non logged-on messages

[Oren M. <or...@qu...>]

Any message that is sent as a result of a resend request, will always  
have a PosDupFlag=Y attached to it.  This isn't QuickFIX specific,  
but is mandated by the FIX specification itself.  In your case, if  
something is added to the queue, it means that the message was not  
successfully sent the first time around.  So when it does finally get  
sent, it would be due to a resend request.  Even though this message  
was never sent, it is still labeled PosDup. This is why they call the  
field PosDupFlag instead of DefinatelyDupFlag.

--oren

On Aug 31, 2005, at 2:26 AM, John GALLET wrote:

> QuickFIX Documentation: http://www.quickfixengine.org/quickfix/doc/ 
> html/index.html
> QuickFIX Support: http://www.quickfixengine.org/services.html
>
> Hi there,
>
>
>> Here's the problem in the example code (tradeclient/Application.cpp):
>>
> Thanks VERY MUCH for the explanation.
>
>
>> This application will *never* resend messages, since they always have
>> PossDupFlag=Y
>>
>
> So the way it works is that the messages are written to the sending  
> queue
> with PossDupFlag=Y ? I would never have guessed the logic behind  
> that. Is
> there some documentation explaining the inner works of QF for message
> exchanges that I missed ?
>
> Thanks again.
>
> Sincerely,
> JG
>
>
>
>
> -------------------------------------------------------
> SF.Net email is Sponsored by the Better Software Conference & EXPO
> September 19-22, 2005 * San Francisco, CA * Development Lifecycle  
> Practices
> Agile & Plan-Driven Development * Managing Projects & Teams *  
> Testing & QA
> Security * Process Improvement & Measurement * http://www.sqe.com/ 
> bsce5sf
> _______________________________________________
> Quickfix-developers mailing list
> Qui...@li...
> https://lists.sourceforge.net/lists/listinfo/quickfix-developers
>
>

Re: [Quickfix-developers] non logged-on messages

[Brian E. <azz...@ya...>]

Oren -

Does FIX mandate PosDup on messages that were never even *attempted* to
be sent? That seems to be an odd requirement.

I would expect that perhaps the first message that gets enqueued due to
a missing socket connection might have to be PosDup'd (perhaps the
socket connection disappeared during a send), but on subsequent
messages the engine would know for sure whether that message had ever
been sent to the counterparty (even though it may not know whether it
reached the counterparty). Almost by definition, any message with a
sequence number greater than the sequence number of last message that
actually reached a valid socket/network-stream object is NOT a PosDup.

In our own home-rolled FIX engine we don't set the PosDup flag unless
the message is sent due to a sequence-resend request and the sequence
number is below the sequence number of the last known attempt at
transmission.

- Brian Erst
Thynk Software, Inc.

--- Oren Miller <or...@qu...> wrote:

> QuickFIX Documentation:
> http://www.quickfixengine.org/quickfix/doc/html/index.html
> QuickFIX Support: http://www.quickfixengine.org/services.html
> 
> Any message that is sent as a result of a resend request, will always
>  
> have a PosDupFlag=Y attached to it.  This isn't QuickFIX specific,  
> but is mandated by the FIX specification itself.  In your case, if  
> something is added to the queue, it means that the message was not  
> successfully sent the first time around.  So when it does finally get
>  
> sent, it would be due to a resend request.  Even though this message 
> 
> was never sent, it is still labeled PosDup. This is why they call the
>  
> field PosDupFlag instead of DefinatelyDupFlag.
> 
> --oren
> 
> On Aug 31, 2005, at 2:26 AM, John GALLET wrote:
> 
> > QuickFIX Documentation: http://www.quickfixengine.org/quickfix/doc/
> 
> > html/index.html
> > QuickFIX Support: http://www.quickfixengine.org/services.html
> >
> > Hi there,
> >
> >
> >> Here's the problem in the example code
> (tradeclient/Application.cpp):
> >>
> > Thanks VERY MUCH for the explanation.
> >
> >
> >> This application will *never* resend messages, since they always
> have
> >> PossDupFlag=Y
> >>
> >
> > So the way it works is that the messages are written to the sending
>  
> > queue
> > with PossDupFlag=Y ? I would never have guessed the logic behind  
> > that. Is
> > there some documentation explaining the inner works of QF for
> message
> > exchanges that I missed ?
> >
> > Thanks again.
> >
> > Sincerely,
> > JG
> >
> >
> >
> >
> > -------------------------------------------------------
> > SF.Net email is Sponsored by the Better Software Conference & EXPO
> > September 19-22, 2005 * San Francisco, CA * Development Lifecycle  
> > Practices
> > Agile & Plan-Driven Development * Managing Projects & Teams *  
> > Testing & QA
> > Security * Process Improvement & Measurement * http://www.sqe.com/ 
> > bsce5sf
> > _______________________________________________
> > Quickfix-developers mailing list
> > Qui...@li...
> > https://lists.sourceforge.net/lists/listinfo/quickfix-developers
> >
> >
> 
> 
> 
> -------------------------------------------------------
> SF.Net email is Sponsored by the Better Software Conference & EXPO
> September 19-22, 2005 * San Francisco, CA * Development Lifecycle
> Practices
> Agile & Plan-Driven Development * Managing Projects & Teams * Testing
> & QA
> Security * Process Improvement & Measurement *
> http://www.sqe.com/bsce5sf
> _______________________________________________
> Quickfix-developers mailing list
> Qui...@li...
> https://lists.sourceforge.net/lists/listinfo/quickfix-developers
> 

Re: [Quickfix-developers] non logged-on messages

[Caleb E. <cal...@gm...>]

On 8/31/05, Brian Erst <azz...@ya...> wrote:


I would expect that perhaps the first message that gets enqueued due to
> a missing socket connection might have to be PosDup'd (perhaps the
> socket connection disappeared during a send), but on subsequent
> messages the engine would know for sure whether that message had ever
> been sent to the counterparty (even though it may not know whether it
> reached the counterparty). Almost by definition, any message with a
> sequence number greater than the sequence number of last message that
> actually reached a valid socket/network-stream object is NOT a PosDup.
>=20

But you can't know which of your sequence numbers your counterparty is up=
=20
to! Well, you could reconnect, or initiate a Test request and wait for the=
=20
response. But sockets have buffers and networks go down and applications=20
crash, so a successful send() on your end doesn't always mean the other sid=
e=20
will recv() it. So you may have think you have sent several messages that=
=20
your counterparty never got.

In our own home-rolled FIX engine we don't set the PosDup flag unless
> the message is sent due to a sequence-resend request and the sequence
> number is below the sequence number of the last known attempt at
> transmission.


But what if your last-known-attempt counter is too high (as I think I=20
describe above)? I think this behavior is in violation of the spec, which=
=20
says that the PossDupFlag will be set to Y on any message sent in response=
=20
to a resend request. If a counterparty asks you for a bunch of messages, yo=
u=20
shouldn't presume to know better than they do which ones they haven't seen=
=20
yet :-)

--=20
Caleb Epstein
caleb dot epstein at gmail dot com

Re: [Quickfix-developers] non logged-on messages

[Brian E. <azz...@ya...>]

Caleb -

I would distinguish between messages that may have been forwarded to
your communications object and those that have simply been enqueued.

I misspoke when I mentioned sequence-resend requests - obviously those
require PosDup. I was envisioning the scenario when messages are being
queued while there is no active login or socket - once the login
occurs, the enqueued messages are sent. Would QF set the PosDup flag in
this case?

Another scenario is the case of messages that are queued due to the
engine processing a previous sequence-resend request. For instance,
counterparty requests messages 1-0 (all messages). At the time of the
request, the next sequence number is 996, so messages 1-995 are
"resends" and anything after that are not. While QF is sending messages
1-995, messages 996-1000 are enqueued awaiting transmission (there are
995 messages in front of them and the socket takes time). Would
messages 996-1000 have PosDup? I presume not, otherwise you might never
be able to get out of the PosDup trap.

I believe the answer to both questions above should be "no" and assume
that QF works that way as well.

- Brian

--- Caleb Epstein <cal...@gm...> wrote:

> On 8/31/05, Brian Erst <azz...@ya...> wrote:
> 
> 
> I would expect that perhaps the first message that gets enqueued due
> to
> > a missing socket connection might have to be PosDup'd (perhaps the
> > socket connection disappeared during a send), but on subsequent
> > messages the engine would know for sure whether that message had
> ever
> > been sent to the counterparty (even though it may not know whether
> it
> > reached the counterparty). Almost by definition, any message with a
> > sequence number greater than the sequence number of last message
> that
> > actually reached a valid socket/network-stream object is NOT a
> PosDup.
> > 
> 
> But you can't know which of your sequence numbers your counterparty
> is up 
> to! Well, you could reconnect, or initiate a Test request and wait
> for the 
> response. But sockets have buffers and networks go down and
> applications 
> crash, so a successful send() on your end doesn't always mean the
> other side 
> will recv() it. So you may have think you have sent several messages
> that 
> your counterparty never got.
> 
> In our own home-rolled FIX engine we don't set the PosDup flag unless
> > the message is sent due to a sequence-resend request and the
> sequence
> > number is below the sequence number of the last known attempt at
> > transmission.
> 
> 
> But what if your last-known-attempt counter is too high (as I think I
> 
> describe above)? I think this behavior is in violation of the spec,
> which 
> says that the PossDupFlag will be set to Y on any message sent in
> response 
> to a resend request. If a counterparty asks you for a bunch of
> messages, you 
> shouldn't presume to know better than they do which ones they haven't
> seen 
> yet :-)
> 
> -- 
> Caleb Epstein
> caleb dot epstein at gmail dot com
> 

Re: [Quickfix-developers] non logged-on messages

[John G. <joh...@wa...>]

Hi there,

> I would distinguish between messages that may have been forwarded to
> your communications object and those that have simply been enqueued.
So I thought it worked, but it is not the case if I understood well.

> require PosDup. I was envisioning the scenario when messages are being
> queued while there is no active login or socket - once the login
> occurs, the enqueued messages are sent. Would QF set the PosDup flag in
> this case?

This is exactly the case I had. I was enqueuing messages before trying to 
connect, and the snippet of code Caleb copied then prevented these 
messages from being sent as they were PosDup tagged.

> I believe the answer to both questions above should be "no" and assume
> that QF works that way as well.
Can't say about the second case, but the first looks to me a "yes".

JG

Re: [Quickfix-developers] non logged-on messages

[Caleb E. <cal...@gm...>]

On 8/31/05, Brian Erst <azz...@ya...> wrote:

I would distinguish between messages that may have been forwarded to
> your communications object and those that have simply been enqueued.
>=20
> I misspoke when I mentioned sequence-resend requests - obviously those
> require PosDup. I was envisioning the scenario when messages are being
> queued while there is no active login or socket - once the login
> occurs, the enqueued messages are sent. Would QF set the PosDup flag in
> this case?


I believe QF will set the PossDup flag on any message that is sent (or=20
re-sent) after having been read back from the MessageStore. This means any=
=20
message an application "sends" while a connection is down will be sent as a=
=20
PossDup, even if it could never have been sent before. For example, suppose=
=20
we have:

   - Session is up and running, sequence numbers are 100:100=20
   (yours:theirs)
    - Session is dropped=20
   - Your application "sends" 100 more messages, making the sequence=20
   numbers 200:100=20
   - Session reconnects, login is sent as message 201=20
   - Counterparty requests resend of messages 101-0=20
   - QF resends messages 101-200 all with PossDupFlag=3DY
   =20
QuickFIX doesn't keep track of the fact that the transport associated with =
a=20
session is up or down at the time a message is sent, so it doesn't have any=
=20
way to know if a message might have been sent before. Basically, if it has=
=20
to read a message from the MessageStore (which it will only do when=20
processing a ResendRequest), it will send it as a PossDup.

Another scenario is the case of messages that are queued due to the
> engine processing a previous sequence-resend request. For instance,
> counterparty requests messages 1-0 (all messages). At the time of the
> request, the next sequence number is 996, so messages 1-995 are
> "resends" and anything after that are not. While QF is sending messages
> 1-995, messages 996-1000 are enqueued awaiting transmission (there are
> 995 messages in front of them and the socket takes time). Would
> messages 996-1000 have PosDup? I presume not, otherwise you might never
> be able to get out of the PosDup trap.


Given a resend request from N to 0, the Session::nextResendRequest code wil=
l=20
resend all messages from N to the next-expected-sender-sequence-number - 1=
=20
with PossDupFlag=3DY. In this case it would be 1-995. This is done while=20
holding the Session's mutex locked, so it will not be possible to send any=
=20
additional messages. Your application code would be blocked waiting on that=
=20
mutex. I think this is more or less what you'd expect.

I believe the answer to both questions above should be "no" and assume that=
=20
> QF works that way as well.


You know the old saying about "assume" :-)

Seriously though. You raise some interesting points about keeping closer=20
track of which messages could/could not have possibly been sent previously.=
=20
This is something that the SessionState could perhaps keep track of, but it=
=20
would complicate the code for questionable benefit.

What is the harm in sending a message that might never before have been sen=
t=20
as a PossDup? I know in our trading systems we don't even really make use o=
f=20
this flag at all. We do dupe detection based on IDs like ClOrdID, ExecID an=
d=20
the like and this has stood us in good stead.

--=20
Caleb Epstein
caleb dot epstein at gmail dot com

Re: [Quickfix-developers] non logged-on messages

[John G. <joh...@wa...>]

Hi (again),

> Seriously though. You raise some interesting points about keeping closer 
> track of which messages could/could not have possibly been sent previously. 
> This is something that the SessionState could perhaps keep track of, but it 
> would complicate the code for questionable benefit.
> What is the harm in sending a message that might never before have been sent 
> as a PossDup? 

I know at least one system that just drops them, but they are wrong and 
should use fonctionnal data to ensure uniqueness as you describe.

The only thing that bugs me is that I don't like the idea of relying on 
the counterparty to basically say "eh, are you sure you did not forget 
something ?" when I perfectly know I did. As I understand it, resend 
resquests are a way of recovering unforeseen errors, not a means to 
say "it's time to empty the sending queue".

It works like this, it's just a question of having the right information. 

Now on the other hand : why then in examples, disable totally the 
resending of a message tagged (we know wrongly, because it works like 
this) PossDup ? In fact, why even try to disable it ? It is sure to cause 
losses of application messages that will go totally unnoticed for a long 
time. 

Sincerely,
JG

Re: [Quickfix-developers] non logged-on messages

[John G. <joh...@wa...>]

Hi,

> Any message that is sent as a result of a resend request, will always  
> have a PosDupFlag=Y attached to it.

Thanks for these explanations. In fact I had not figured that a message
that had never been sent would be sent because of a resend request from
the counterparty. This seems dangerous to me, relying *only* on the other
side to determine which messages were sent and which were not. In fact I
presumed (wrongly) that the "sending queue" was emptied after the
connection went into "logged-on" state and all synchronisation issues were
dealt with.

>  This isn't QuickFIX specific,  
> but is mandated by the FIX specification itself. 
Totally agreed.

> In your case, if  
> something is added to the queue, it means that the message was not  
> successfully sent the first time around.  So when it does finally get  
> sent, it would be due to a resend request.  

This is the piece of logic that I was getting wrong. I was thinking of the 
emission queue as a buffer independant from the admin messages, that 
would be emptyied when the admin part was done. In fact, I see the admin 
messages vs "real" messages a little as the two FTP channels (control and 
data), and this vision is not QF's (I am not pretending you guys are 
wrong, just explaining what I was missing).

> Even though this message  was never sent, it is still labeled PosDup. 
This was the information I needed. 

Thanks again to all for explanations.

JG