quickfix-users — QuickFIX user support

Re: [Quickfix-users] Modifying toAdmin/toApp messages

[Brian E. <azz...@ya...>]

Oren -

I think that sounds great. My main concern was whether non-C++
platforms (.Net, Java and Python) handled overloaded methods that
differed only by the "constness" of one of the arguments.

I also wondered about backward compatibility - if the new method
differed only by signature, it might break existing applications which
are currently using the cracker on toApp/toAdmin messages and
performing the cast. (If you called the new cracker from toApp/toAdmin,
it would crack to the non-const method, but the existing application
would be overriding the const method, causing the messages to be
skipped.)

A best of both worlds approach might be to simply make the "non-const"
versions have a different method name (as in decorateMessage() or
something that better fits your naming scheme) - existing applications
would not be affected, but you wouldn't pollute the namespace with
another class definition.

If/when this gets implemented, I'll try to update the Wiki to document
the new behavior.

Thanks, Oren!

- Brian Erst


--- Oren Miller <or...@qu...> wrote:

> No need to go typing all that up.  We certainly didn't type up the 
> MessageCracker.  It is being generated off of the XML.
> 
> I think instead of creating a new Decorator class, we can add methods
> that 
> receive non-const messages alongside the const ones.  This way the
> same 
> message cracker can be used and C++ will be able to resolve which
> method to 
> call based on the constness of the message.  So for instance we could
> have:
> 
> virtual void onMessage( const SessionID& sessionID, const 
> FIX42::NewOrderSingle& message )
> virtual void onMessage( const SessionID& sessionID,
> FIX42::NewOrderSingle& 
> message )
> 
> Both generated for the MessageCracker.  What do you think of that?
> 
> --oren
> 
> ----- Original Message ----- 
> From: "Brian Erst" <azz...@ya...>
> To: <qui...@li...>
> Sent: Thursday, December 30, 2004 11:13 PM
> Subject: [Quickfix-users] Modifying toAdmin/toApp messages
> 
> 
> > QuickFIX Documentation: 
> > http://www.quickfixengine.org/quickfix/doc/html/index.html
> > QuickFIX FAQ:
> http://www.quickfixengine.org/wikifix/index.php?QuickFixFAQ
> > QuickFIX Support: http://www.quickfixengine.org/services.html
> >
> > As I am relatively new to QuickFIX, this may have already been
> covered
> > in the past (I did a search across the mailing list archive and
> didn't
> > see much on it though).
> >
> > It seems to me that the use of toApp and toAdmin needs to be better
> > documented and that there might be a need for a message cracker
> > specific to these messages. None of the examples, for instance,
> make
> > use of either of these methods (except for a token check of the
> > PossDupFlag in TradeClient).
> >
> > The FIX server I was connecting to required the use of the Username
> and
> > Password fields in a Logon message. It took a long time perusing
> the
> > mailing list and reading the source code to figure out that I'd
> need to
> > use the toApp method to trap the automatically generated Logon
> message
> > and modify that message.
> >
> > After that, I bumped into a slight problem with message cracking.
> The
> > MessageCracker class can crack any message, but will only dispatch
> > those messages as "const", which requires a cast in order to add
> the
> > appropriate fields (I created a non-const pointer to the Logon
> message
> > and set the fields).
> >
> > Is there/should there be an "inbound" message cracker and an
> "outbound"
> > message cracker? "Inbound" messages would have a "const" modifier
> to
> > prevent changes to the message, while "outbound" messages would be
> > non-const so the message could be modified (or "decorated") prior
> to
> > being sent.
> >
> > For backward compatibility, the "inbound" message cracker could
> > continue to be MessageCracker, while a new class could handle the
> > "outbound"  message cracking/decorating.
> >
> > I'd foresee something like:
> >
> > class MessageDecorator
> > {
> > public:
> >
> >    virtual ~MessageDecorator() {}
> >    virtual void decorateMessage(Message&, const FIX::SessionID&)
> >        {throw FIX::UnsupportedMessageType();}
> >    virtual void decorateMessage(Heartbeat&, const FIX::SessionID&)
> {}
> >    virtual void decorateMessage(Logon &, const FIX::SessionID&) {}
> >    ...
> >
> > public:
> >
> >    void decorate(Message & message, const FIX::SessionID &
> sessionID)
> >    {
> >        FIX::MsgType msgType;
> >        message.getHeader().getField(msgType);
> >        std::string msgTypeValue = msgType.getValue();
> >
> >        if (msgTypeValue == "0")
> >            decorateMessage((Heartbeat&)message, sessionID);
> >        else
> >        if (msgTypeValue == "A")
> >            decorateMessage((Logon&)message, sessionID);
> >        ...
> >    }
> > }
> >
> > and Application could inherit from both MessageCracker and
> > MessageDecorator.
> >
> > Then, when an Application-derived class needs to add fields to
> outbound
> > messages, they simply override the toApp() or toAdmin() methods and
> > override the appropriate decorator.
> >
> > Maybe this is overkill to avoid casting, but it also helps to
> separate
> > the message flow - otherwise, a single "onMessage" method might be
> > called by both inbound messages AND outbound messages (when systems
> are
> > closer to being peers, you can often see the same messages going
> both
> > ways) and you have to add additional checks (who does the
> SenderCompID
> > point to in this message).
> >
> > Just an idea. I'm willing to put my money where my mouth is an
> write a
> > C++ MessageDecorator class - it's a pretty simple modification of
> the
> > existing MessageCracker class(es).
> >
> > I'll also look into updating the Wiki to document how things work
> now -
> > assuming I accurately understand the process.
> >
> > - Brian Erst
> >
> > p.s. QuickFIX is a very impressive piece of work - none of the
> above
> > should be construed as anything other than a wishlist item.
> >
> >
> > -------------------------------------------------------
> > The SF.Net email is sponsored by: Beat the post-holiday blues
> > Get a FREE limited edition SourceForge.net t-shirt from ThinkGeek.
> > It's fun and FREE -- well,
> almost....http://www.thinkgeek.com/sfshirt
> > _______________________________________________
> > Quickfix-users mailing list
> > Qui...@li...
> > https://lists.sourceforge.net/lists/listinfo/quickfix-users
> > 
> 
> 

Re: [Quickfix-users] Modifying toAdmin/toApp messages

[Oren M. <or...@qu...>]

No need to go typing all that up.  We certainly didn't type up the 
MessageCracker.  It is being generated off of the XML.

I think instead of creating a new Decorator class, we can add methods that 
receive non-const messages alongside the const ones.  This way the same 
message cracker can be used and C++ will be able to resolve which method to 
call based on the constness of the message.  So for instance we could have:

virtual void onMessage( const SessionID& sessionID, const 
FIX42::NewOrderSingle& message )
virtual void onMessage( const SessionID& sessionID, FIX42::NewOrderSingle& 
message )

Both generated for the MessageCracker.  What do you think of that?

--oren

----- Original Message ----- 
From: "Brian Erst" <azz...@ya...>
To: <qui...@li...>
Sent: Thursday, December 30, 2004 11:13 PM
Subject: [Quickfix-users] Modifying toAdmin/toApp messages


> QuickFIX Documentation: 
> http://www.quickfixengine.org/quickfix/doc/html/index.html
> QuickFIX FAQ: http://www.quickfixengine.org/wikifix/index.php?QuickFixFAQ
> QuickFIX Support: http://www.quickfixengine.org/services.html
>
> As I am relatively new to QuickFIX, this may have already been covered
> in the past (I did a search across the mailing list archive and didn't
> see much on it though).
>
> It seems to me that the use of toApp and toAdmin needs to be better
> documented and that there might be a need for a message cracker
> specific to these messages. None of the examples, for instance, make
> use of either of these methods (except for a token check of the
> PossDupFlag in TradeClient).
>
> The FIX server I was connecting to required the use of the Username and
> Password fields in a Logon message. It took a long time perusing the
> mailing list and reading the source code to figure out that I'd need to
> use the toApp method to trap the automatically generated Logon message
> and modify that message.
>
> After that, I bumped into a slight problem with message cracking. The
> MessageCracker class can crack any message, but will only dispatch
> those messages as "const", which requires a cast in order to add the
> appropriate fields (I created a non-const pointer to the Logon message
> and set the fields).
>
> Is there/should there be an "inbound" message cracker and an "outbound"
> message cracker? "Inbound" messages would have a "const" modifier to
> prevent changes to the message, while "outbound" messages would be
> non-const so the message could be modified (or "decorated") prior to
> being sent.
>
> For backward compatibility, the "inbound" message cracker could
> continue to be MessageCracker, while a new class could handle the
> "outbound"  message cracking/decorating.
>
> I'd foresee something like:
>
> class MessageDecorator
> {
> public:
>
>    virtual ~MessageDecorator() {}
>    virtual void decorateMessage(Message&, const FIX::SessionID&)
>        {throw FIX::UnsupportedMessageType();}
>    virtual void decorateMessage(Heartbeat&, const FIX::SessionID&) {}
>    virtual void decorateMessage(Logon &, const FIX::SessionID&) {}
>    ...
>
> public:
>
>    void decorate(Message & message, const FIX::SessionID & sessionID)
>    {
>        FIX::MsgType msgType;
>        message.getHeader().getField(msgType);
>        std::string msgTypeValue = msgType.getValue();
>
>        if (msgTypeValue == "0")
>            decorateMessage((Heartbeat&)message, sessionID);
>        else
>        if (msgTypeValue == "A")
>            decorateMessage((Logon&)message, sessionID);
>        ...
>    }
> }
>
> and Application could inherit from both MessageCracker and
> MessageDecorator.
>
> Then, when an Application-derived class needs to add fields to outbound
> messages, they simply override the toApp() or toAdmin() methods and
> override the appropriate decorator.
>
> Maybe this is overkill to avoid casting, but it also helps to separate
> the message flow - otherwise, a single "onMessage" method might be
> called by both inbound messages AND outbound messages (when systems are
> closer to being peers, you can often see the same messages going both
> ways) and you have to add additional checks (who does the SenderCompID
> point to in this message).
>
> Just an idea. I'm willing to put my money where my mouth is an write a
> C++ MessageDecorator class - it's a pretty simple modification of the
> existing MessageCracker class(es).
>
> I'll also look into updating the Wiki to document how things work now -
> assuming I accurately understand the process.
>
> - Brian Erst
>
> p.s. QuickFIX is a very impressive piece of work - none of the above
> should be construed as anything other than a wishlist item.
>
>
> -------------------------------------------------------
> The SF.Net email is sponsored by: Beat the post-holiday blues
> Get a FREE limited edition SourceForge.net t-shirt from ThinkGeek.
> It's fun and FREE -- well, almost....http://www.thinkgeek.com/sfshirt
> _______________________________________________
> Quickfix-users mailing list
> Qui...@li...
> https://lists.sourceforge.net/lists/listinfo/quickfix-users
> 

[Quickfix-users] Modifying toAdmin/toApp messages

[Brian E. <azz...@ya...>]

As I am relatively new to QuickFIX, this may have already been covered
in the past (I did a search across the mailing list archive and didn't
see much on it though).

It seems to me that the use of toApp and toAdmin needs to be better
documented and that there might be a need for a message cracker
specific to these messages. None of the examples, for instance, make
use of either of these methods (except for a token check of the
PossDupFlag in TradeClient).

The FIX server I was connecting to required the use of the Username and
Password fields in a Logon message. It took a long time perusing the
mailing list and reading the source code to figure out that I'd need to
use the toApp method to trap the automatically generated Logon message
and modify that message. 

After that, I bumped into a slight problem with message cracking. The
MessageCracker class can crack any message, but will only dispatch
those messages as "const", which requires a cast in order to add the
appropriate fields (I created a non-const pointer to the Logon message
and set the fields).

Is there/should there be an "inbound" message cracker and an "outbound"
message cracker? "Inbound" messages would have a "const" modifier to
prevent changes to the message, while "outbound" messages would be
non-const so the message could be modified (or "decorated") prior to
being sent.

For backward compatibility, the "inbound" message cracker could
continue to be MessageCracker, while a new class could handle the
"outbound"  message cracking/decorating.

I'd foresee something like:

class MessageDecorator
{
public:

    virtual ~MessageDecorator() {}
    virtual void decorateMessage(Message&, const FIX::SessionID&)
        {throw FIX::UnsupportedMessageType();}
    virtual void decorateMessage(Heartbeat&, const FIX::SessionID&) {}
    virtual void decorateMessage(Logon &, const FIX::SessionID&) {}
    ...

public:

    void decorate(Message & message, const FIX::SessionID & sessionID)
    {
        FIX::MsgType msgType;
        message.getHeader().getField(msgType);
        std::string msgTypeValue = msgType.getValue();
    
        if (msgTypeValue == "0")
            decorateMessage((Heartbeat&)message, sessionID);
        else
        if (msgTypeValue == "A")
            decorateMessage((Logon&)message, sessionID);
        ...
    }    
}

and Application could inherit from both MessageCracker and
MessageDecorator.

Then, when an Application-derived class needs to add fields to outbound
messages, they simply override the toApp() or toAdmin() methods and
override the appropriate decorator.

Maybe this is overkill to avoid casting, but it also helps to separate
the message flow - otherwise, a single "onMessage" method might be
called by both inbound messages AND outbound messages (when systems are
closer to being peers, you can often see the same messages going both
ways) and you have to add additional checks (who does the SenderCompID
point to in this message).

Just an idea. I'm willing to put my money where my mouth is an write a
C++ MessageDecorator class - it's a pretty simple modification of the
existing MessageCracker class(es).

I'll also look into updating the Wiki to document how things work now -
assuming I accurately understand the process.

- Brian Erst

p.s. QuickFIX is a very impressive piece of work - none of the above
should be construed as anything other than a wishlist item.

RE: [Quickfix-users] some question on QuickFix usage

[Alexey Z. <ale...@in...>]

Hi Oren,
=20
I=92ve got the reject message in the toAdmin callback. It helps me to =
know
at least that the problem has happened.
=20
Regarding to the XML changes:
I was afraid that changing of type of a field may cause QuickFIX
internal problems.
Anyway, I changed MDReqRejReason field from CHAR to STRING and it works.
=20
Thank you for help.
=20
Regards,=20
Alexey.
=20
=20
-----Original Message-----
From: Oren Miller [mailto:or...@qu...]=20
Sent: Tuesday, December 28, 2004 4:31 PM
To: Alexey Zubko; qui...@li...
Subject: Re: [Quickfix-users] some question on QuickFix usage
=20
The message itself will not go to your application, but the reject that
goes out to the counterparty should via the toAdmin callback.
=20
You should never have to send your own ResendRequest messages.  QuickFIX
is very smart about doing this on its own when necessary.  The .outgoing
file should contain everything that is sent to the counterparty,
including ResendRequests.
=20
I'm not sure why you can't solve your problem with a new xml.  From your
description it seems you just need an xml that declares that field as a
string.  Is there anything else?
=20
--oren

Re: [Quickfix-users] some question on QuickFix usage

[Oren M. <or...@qu...>]

The message itself will not go to your application, but the reject that =
goes out to the counterparty should via the toAdmin callback.

You should never have to send your own ResendRequest messages.  QuickFIX =
is very smart about doing this on its own when necessary.  The .outgoing =
file should contain everything that is sent to the counterparty, =
including ResendRequests.

I'm not sure why you can't solve your problem with a new xml.  From your =
description it seems you just need an xml that declares that field as a =
string.  Is there anything else?

--oren
  ----- Original Message -----=20
  From: Alexey Zubko=20
  To: qui...@li...=20
  Sent: Thursday, December 23, 2004 2:02 PM
  Subject: RE: [Quickfix-users] some question on QuickFix usage


  Hi Oren,

  =20

  Thank you for the fast reply.

  =20

  I read this documentation of course, but it's more like a quick start =
reference.

  It's easy to build a server or a client application, but if you need =
to reset the sequence number manually, for example, you have to look =
through the code.

  =20

  I'm absolutely agreed that validated messages are less dangerous.

  But it's more danger to lose a message than to know that it was not =
passed validation.

  =20

  I write a client for non-QuickFix server which misuses a FIX tag =
(sends string instead of char (enum)).

  I can't solve the problem by defining a new xml, I need to change the =
QuickFix source code and as a result I have a separate library for this =
particular project.

  By the way, this field is not required by protocol and I really don't =
care of the value.

  =20

  What did you mean saying to monitor the reject message?

  The message is rejected in the dictionary and is not go to my =
application class.

  =20

  Regarding resend:=20

  Do you mean I don't need to create my ResendRequest callback to resend =
messages?

  If QuickFix sends these messages by themselves should the *.outgoing =
file contains them too?

  =20

  =20

  Thank you again.

  =20

  =20

  Regards,

  Alexey.

    =20

Re: [Quickfix-users] need for SwingUtilities in trading system application?

[Oren M. <or...@qu...>]

Well, the only reason that invokeLater is being called is due to the 
requirement that Swing components need to be updated in the dispatch 
thread.  If you are not using Swing, this requirement goes away 
completely and there isn't really a need to make an asynchronous call 
at all.  Just do whatever you want to do in the fromApp method.  Just 
make sure to synchronize your objects where necessary.  If you do not 
want to deal with threads at all then you should use the poll() method 
in place of start().

--oren

On Dec 28, 2004, at 6:51 AM, Murray Smigel wrote:

> Hi,
> I notice that you are using the SwingUtilities function invokeLater to 
> invoke
> the run function of doFromApp in your java example program
> BanzaiApplication.java
>
> public void fromApp(quickfix.Message message, SessionID sessionID)
>     throws FieldNotFound, IncorrectDataFormat, IncorrectTagValue,
> UnsupportedMessageType {
>       try {
> 	SwingUtilities.invokeLater(new doFromApp(message, sessionID));
>       } catch(Exception e) {}
>     }
>
> I am trying to build a version of the Application that works in a 
> non-graphic
> environment and so would like to avoid the dependency on 
> SwingUtilities. Is
> there an easy way to do the asynchronous invokation of the run method 
> without
> using the SwingUtilities.invokeLater function?
>
> Thanks,
> murray smigel
>

[Quickfix-users] need for SwingUtilities in trading system application?

[Murray S. <mu...@sm...>]

Hi,
I notice that you are using the SwingUtilities function invokeLater to invo=
ke=20
the run function of doFromApp in your java example program =20
BanzaiApplication.java

public void fromApp(quickfix.Message message, SessionID sessionID)
    throws FieldNotFound, IncorrectDataFormat, IncorrectTagValue,=20
UnsupportedMessageType {
      try {
	SwingUtilities.invokeLater(new doFromApp(message, sessionID));
      } catch(Exception e) {}
    }

I am trying to build a version of the Application that works in a non-graph=
ic=20
environment and so would like to avoid the dependency on SwingUtilities. Is=
=20
there an easy way to do the asynchronous invokation of the run method witho=
ut=20
using the SwingUtilities.invokeLater function?

Thanks,
murray smigel

[Quickfix-users] Client - Server behavior

[Radhika S. <Rad...@Pi...>]

Hi,

In trying to understand better how QuickFix works, what exactly
determines on the client side, the next sequence number?
Is it reading solely from the FIX.4.0-*.seqnums file from the server?


Thanks,
Radhika

RE: [Quickfix-users] some question on QuickFix usage

[Alexey Z. <ale...@in...>]

Hi Oren,
=20
Thank you for the fast reply.
=20
I read this documentation of course, but it=92s more like a quick start
reference.
It=92s easy to build a server or a client application, but if you need =
to
reset the sequence number manually, for example, you have to look
through the code.
=20
I=92m absolutely agreed that validated messages are less dangerous.
But it=92s more danger to lose a message than to know that it was not
passed validation.
=20
I write a client for non-QuickFix server which misuses a FIX tag (sends
string instead of char (enum)).
I can=92t solve the problem by defining a new xml, I need to change the
QuickFix source code and as a result I have a separate library for this
particular project.
By the way, this field is not required by protocol and I really don=92t
care of the value.
=20
What did you mean saying to monitor the reject message?
The message is rejected in the dictionary and is not go to my
application class.
=20
Regarding resend:=20
Do you mean I don=92t need to create my ResendRequest callback to resend
messages?
If QuickFix sends these messages by themselves should the *.outgoing
file contains them too?
=20
=20
Thank you again.
=20
=20
Regards,
Alexey.
=20

Re: [Quickfix-users] some question on QuickFix usage

[Oren M. <or...@qu...>]

Alexey,

First of all, if you haven't found it already, the documentation is =
located here: http://www.quickfixengine.org/quickfix/doc/html/index.html =
and you can locate the examples as per this page: =
http://www.quickfixengine.org/quickfix/doc/html/examples.html

Not sure if you were unable to find the documentation at all or if you =
meant that it did not answer your questions.

1) QuickFIX only sends messages that pass validation back to the =
application.  Messages that do not pass basic validation are considered =
to dangerous to pass back since there can be a variety of issues that =
make them unstable.  What you can do is monitor for the Reject message =
that QuickFIX generates.  This message will contain the sequence number, =
message type, and specific tag (if applicable) that caused QuickFIX to =
reject this message.

2) No, only messages that pass validation make it through to the =
application.  The log will contain messages that are well formatted =
enough to make it through initial parsing.

3) Typically QuickFIX will take care of this when necessary.  You should =
not ever resend the same message yourself since there are complex =
interaction involved that identify when this is necessary.  For user =
level resend, FIX supports a special field called PossResend, which will =
allow you to send an identical message, but with a new sequence number, =
and indicate that for some reason you believe that the original may not =
have been processed.  You need to make sure that your counterparty =
supports the use of this field for the message type you are resending.

4) You can get access to the log by using the lookupSession call to get =
a pointer to a session, and the the getLog method to pull out a pointer =
to it's logger.  You cannot pull out data from the log, but you can add =
your own events there.

--oren
  ----- Original Message -----=20
  From: Alexey Zubko=20
  To: qui...@li...=20
  Sent: Thursday, December 23, 2004 10:35 AM
  Subject: [Quickfix-users] some question on QuickFix usage


  Hi All,

  =20

  I just started to use the engine (VC++). I found it pretty =
interesting, but unfortunately there is a lack of documentation and =
examples.

  I understand that open source gives much, but sometimes it's necessary =
to get results fast.

  =20

  It would be very nice to get answers for some of my questions:

  =20

    1.. Is it possible to handle some exceptions in the application =
class?=20
  For example:

  The engine rejects a message (invalid tag) but my application class =
knows nothing about this.

  =20

    2.. Is it possible to get a message (and possibly change) before =
validation in dictionary?=20
  =20

    3.. Is there an easy way to resend previous messages (on resend =
message)?=20
  =20

    4.. How can I handle/get/add mine events in log?=20
  =20

  =20

  Thank you in advance.

  =20

  =20

  Regards,

              Alexey.

Re: [Quickfix-users] Sequence number resets question

[Oren M. <or...@qu...>]

Radhika,

The standard case for FIX is to logon using the last sequence number before 
the logoff.  So you are seeing normal behavior from QuickFIX.  QuickFIX also 
supports the behavior of always logging on with a sequence number of 1.  To 
enable this you will need to set the ResetOnDisconnect and ResetOnLogout 
configuration settings to Y.

--oren

----- Original Message ----- 
From: "Radhika Sambamurti" <Rad...@Pi...>
To: <qui...@li...>
Sent: Thursday, December 23, 2004 7:38 AM
Subject: [Quickfix-users] Sequence number resets question


QuickFIX Documentation: 
http://www.quickfixengine.org/quickfix/doc/html/index.html
QuickFIX FAQ: http://www.quickfixengine.org/wikifix/index.php?QuickFixFAQ
QuickFIX Support: http://www.quickfixengine.org/services.html

Hi,

I need some help with message sequence numbers not being set correctly
by the fix client.
The client I am writing uses a fix session to connect to the server.
Every time I logoff, I issue a session.logoff() - which according to the
docs says resets sequence numbers.
The behavior I am facing is that every time I logon using the same fix
session, the client sends the last used sequence number.
A very first time login (even though last used sequence numbers are
sent, and then a message sequence number of 1 is sent) is successful.
Logging off and re-logging on creates a mis-match in the session
numbers.
The server seems to be resetting and asking for a sequence number of 1.
The client seems to be providing a larger number.
Any ideas or help on this will be helpful. I have included both Client
and Server event logs below.
Thank you.
Radhika


Client event logs - Connecting for the first time - starts with seqnum
of 4 and reverts to 1 - but is still connected

20041222-21:34:00 : Created session
20041222-21:34:01 : Connecting to fix_server
20041222-21:34:01 : Connection succeeded
20041222-21:34:02 : Initiated logon request
20041222-21:34:02 : MsgSeqNum too low, expecting 4 but received 1
20041222-21:34:02 : Disconnecting
20041222-21:34:06 : Connecting to fix_server
20041222-21:34:06 : Connection succeeded
20041222-21:34:06 : Initiated logon request
20041222-21:34:06 : Received logon response

connecting for the second time after logging off from above session

20041222-21:39:37 : Created session
20041222-21:39:38 : Connecting to fix_server on port 0000
20041222-21:39:38 : Connection succeeded
20041222-21:39:38 : Initiated logon request
20041222-21:39:38 : Received logon response
20041222-21:39:41 : Connecting to fix_server on port 0000
20041222-21:39:41 : Connection succeeded
20041222-21:39:41 : Initiated logon request
20041222-21:39:41 : Dropped Connection
20041222-21:39:46 : Connecting to fix_server on port 0000
20041222-21:39:46 : Connection succeeded
20041222-21:39:46 : Initiated logon request
20041222-21:39:46 : Received logon response
20041222-21:39:46 : Received ResendRequest FROM: 1 TO: 999999
20041222-21:39:46 : Sent SequenceReset TO: 3
20041222-21:40:08 : Dropped Connection
20041222-21:40:11 : Connecting to fix_server on port 0000
20041222-21:40:11 : Connection succeeded
20041222-21:40:11 : Initiated logon request
20041222-21:40:11 : MsgSeqNum too low, expecting 2 but received 1
20041222-21:40:11 : Disconnecting
20041222-21:40:14 : Sent test request TEST
20041222-21:40:16 : Connecting to fix_server on port 0000
20041222-21:40:16 : Connection succeeded
20041222-21:40:16 : Initiated logon request
20041222-21:40:16 : MsgSeqNum too low, expecting 2 but received 1
20041222-21:40:16 : Disconnecting
20041222-21:40:21 : Connecting to fix_server on port 0000
20041222-21:40:21 : Connection succeeded
20041222-21:40:21 : Initiated logon request
20041222-21:40:21 : MsgSeqNum too low, expecting 2 but received 1
20041222-21:40:21 : Disconnecting
20041222-21:40:26 : Connecting to fix_server on port 0000
20041222-21:40:26 : Connection succeeded
20041222-21:40:26 : Initiated logon request
20041222-21:40:26 : MsgSeqNum too low, expecting 2 but received 1
20041222-21:40:26 : Disconnecting
20041222-21:40:31 : Connecting to fix_server on port 0000
20041222-21:40:31 : Connection succeeded
20041222-21:40:31 : Initiated logon request
20041222-21:40:31 : MsgSeqNum too low, expecting 2 but received 1
20041222-21:40:31 : Disconnecting


Server event logs:

20041222-21:34:11 : Received logon request
20041222-21:34:11 : Responding to logon request
20041222-21:34:11 : MsgSeqNum too high RECEIVED: 4 EXPECTED: 1
20041222-21:34:11 : Sent ResendRequest FROM: 1 TO: 999999
20041222-21:34:11 : Received logout request
20041222-21:34:11 : Sending logout response
20041222-21:34:11 : Disconnecting
20041222-21:34:15 : Received logon request
20041222-21:34:15 : Responding to logon request
20041222-21:39:36 : Received logout request
20041222-21:39:36 : Sending logout response
20041222-21:39:36 : Disconnecting
20041222-21:39:47 : Received logon request
20041222-21:39:47 : Responding to logon request 20041222-21:39:50 :
Logon state is not valid for message 20041222-21:39:50 : Disconnecting
20041222-21:39:55 : Received logon request
20041222-21:39:55 : Responding to logon request
20041222-21:39:55 : MsgSeqNum too high RECEIVED: 2 EXPECTED: 1
20041222-21:39:55 : Sent ResendRequest FROM: 1 TO: 999999
20041222-21:39:55 : Received SequenceReset FROM: 1 TO: 3
20041222-21:40:18 : Field not found
20041222-21:40:18 : Disconnecting
20041222-21:40:21 : Received logon request
20041222-21:40:21 : Responding to logon request
20041222-21:40:21 : MsgSeqNum too high RECEIVED: 3 EXPECTED: 1
20041222-21:40:21 : Sent ResendRequest FROM: 1 TO: 999999
20041222-21:40:21 : Received logout request
20041222-21:40:21 : Sending logout response
20041222-21:40:21 : Disconnecting
20041222-21:40:24 : Logon state is not valid for message
20041222-21:40:26 : Received logon request
20041222-21:40:26 : Responding to logon request
20041222-21:40:26 : MsgSeqNum too high RECEIVED: 5 EXPECTED: 1
20041222-21:40:26 : Sent ResendRequest FROM: 1 TO: 999999
20041222-21:40:26 : Received logout request
20041222-21:40:26 : Sending logout response
20041222-21:40:26 : Disconnecting
20041222-21:40:31 : Received logon request
20041222-21:40:31 : Responding to logon request
20041222-21:40:31 : MsgSeqNum too high RECEIVED: 7 EXPECTED: 1
20041222-21:40:31 : Sent ResendRequest FROM: 1 TO: 999999
20041222-21:40:31 : Received logout request
20041222-21:40:31 : Sending logout response
20041222-21:40:31 : Disconnecting
20041222-21:40:36 : Received logon request
20041222-21:40:36 : Responding to logon request
20041222-21:40:36 : MsgSeqNum too high RECEIVED: 9 EXPECTED: 1
20041222-21:40:36 : Sent ResendRequest FROM: 1 TO: 999999
20041222-21:40:36 : Received logout request
20041222-21:40:36 : Sending logout response
20041222-21:40:36 : Disconnecting
20041222-21:40:41 : Received logon request
20041222-21:40:41 : Responding to logon request
20041222-21:40:41 : MsgSeqNum too high RECEIVED: 11 EXPECTED: 1
20041222-21:40:41 : Sent ResendRequest FROM: 1 TO: 999999
20041222-21:40:41 : Received logout request
20041222-21:40:41 : Sending logout response
20041222-21:40:41 : Disconnecting


Radhika Sambamurti
Pipeline Trading Systems
Rad...@pi...
212-370-8343


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://productguide.itmanagersjournal.com/
_______________________________________________
Quickfix-users mailing list
Qui...@li...
https://lists.sourceforge.net/lists/listinfo/quickfix-users

[Quickfix-users] some question on QuickFix usage

[Alexey Z. <ale...@in...>]

Hi All,
=20
I just started to use the engine (VC++). I found it pretty interesting,
but unfortunately there is a lack of documentation and examples.
I understand that open source gives much, but sometimes it=92s necessary
to get results fast.
=20
It would be very nice to get answers for some of my questions:
=20
1.	Is it possible to handle some exceptions in the application
class?
For example:
The engine rejects a message (invalid tag) but my application class
knows nothing about this.
=20
2.	Is it possible to get a message (and possibly change) before
validation in dictionary?
=20
3.	Is there an easy way to resend previous messages (on resend
message)?
=20
4.	How can I handle/get/add mine events in log?
=20
=20
Thank you in advance.
=20
=20
Regards,
            Alexey.

[Quickfix-users] Sequence number resets question

[Radhika S. <Rad...@Pi...>]

Hi,
=20
I need some help with message sequence numbers not being set correctly
by the fix client.
The client I am writing uses a fix session to connect to the server.
Every time I logoff, I issue a session.logoff() - which according to the
docs says resets sequence numbers.
The behavior I am facing is that every time I logon using the same fix
session, the client sends the last used sequence number.
A very first time login (even though last used sequence numbers are
sent, and then a message sequence number of 1 is sent) is successful.
Logging off and re-logging on creates a mis-match in the session
numbers.
The server seems to be resetting and asking for a sequence number of 1.
The client seems to be providing a larger number.
Any ideas or help on this will be helpful. I have included both Client
and Server event logs below.
Thank you.
Radhika
=20
=20
Client event logs - Connecting for the first time - starts with seqnum
of 4 and reverts to 1 - but is still connected
=20
20041222-21:34:00 : Created session
20041222-21:34:01 : Connecting to fix_server
20041222-21:34:01 : Connection succeeded
20041222-21:34:02 : Initiated logon request
20041222-21:34:02 : MsgSeqNum too low, expecting 4 but received 1
20041222-21:34:02 : Disconnecting
20041222-21:34:06 : Connecting to fix_server
20041222-21:34:06 : Connection succeeded
20041222-21:34:06 : Initiated logon request
20041222-21:34:06 : Received logon response

connecting for the second time after logging off from above session
=20
20041222-21:39:37 : Created session
20041222-21:39:38 : Connecting to fix_server on port 0000
20041222-21:39:38 : Connection succeeded
20041222-21:39:38 : Initiated logon request
20041222-21:39:38 : Received logon response
20041222-21:39:41 : Connecting to fix_server on port 0000
20041222-21:39:41 : Connection succeeded
20041222-21:39:41 : Initiated logon request
20041222-21:39:41 : Dropped Connection
20041222-21:39:46 : Connecting to fix_server on port 0000
20041222-21:39:46 : Connection succeeded
20041222-21:39:46 : Initiated logon request
20041222-21:39:46 : Received logon response
20041222-21:39:46 : Received ResendRequest FROM: 1 TO: 999999
20041222-21:39:46 : Sent SequenceReset TO: 3
20041222-21:40:08 : Dropped Connection
20041222-21:40:11 : Connecting to fix_server on port 0000
20041222-21:40:11 : Connection succeeded
20041222-21:40:11 : Initiated logon request
20041222-21:40:11 : MsgSeqNum too low, expecting 2 but received 1
20041222-21:40:11 : Disconnecting
20041222-21:40:14 : Sent test request TEST
20041222-21:40:16 : Connecting to fix_server on port 0000
20041222-21:40:16 : Connection succeeded
20041222-21:40:16 : Initiated logon request
20041222-21:40:16 : MsgSeqNum too low, expecting 2 but received 1
20041222-21:40:16 : Disconnecting
20041222-21:40:21 : Connecting to fix_server on port 0000
20041222-21:40:21 : Connection succeeded
20041222-21:40:21 : Initiated logon request
20041222-21:40:21 : MsgSeqNum too low, expecting 2 but received 1
20041222-21:40:21 : Disconnecting
20041222-21:40:26 : Connecting to fix_server on port 0000
20041222-21:40:26 : Connection succeeded
20041222-21:40:26 : Initiated logon request
20041222-21:40:26 : MsgSeqNum too low, expecting 2 but received 1
20041222-21:40:26 : Disconnecting
20041222-21:40:31 : Connecting to fix_server on port 0000
20041222-21:40:31 : Connection succeeded
20041222-21:40:31 : Initiated logon request
20041222-21:40:31 : MsgSeqNum too low, expecting 2 but received 1
20041222-21:40:31 : Disconnecting

=20
Server event logs:
=20
20041222-21:34:11 : Received logon request
20041222-21:34:11 : Responding to logon request
20041222-21:34:11 : MsgSeqNum too high RECEIVED: 4 EXPECTED: 1
20041222-21:34:11 : Sent ResendRequest FROM: 1 TO: 999999
20041222-21:34:11 : Received logout request
20041222-21:34:11 : Sending logout response
20041222-21:34:11 : Disconnecting
20041222-21:34:15 : Received logon request
20041222-21:34:15 : Responding to logon request
20041222-21:39:36 : Received logout request
20041222-21:39:36 : Sending logout response
20041222-21:39:36 : Disconnecting
20041222-21:39:47 : Received logon request
20041222-21:39:47 : Responding to logon request 20041222-21:39:50 :
Logon state is not valid for message 20041222-21:39:50 : Disconnecting
20041222-21:39:55 : Received logon request
20041222-21:39:55 : Responding to logon request
20041222-21:39:55 : MsgSeqNum too high RECEIVED: 2 EXPECTED: 1
20041222-21:39:55 : Sent ResendRequest FROM: 1 TO: 999999
20041222-21:39:55 : Received SequenceReset FROM: 1 TO: 3
20041222-21:40:18 : Field not found
20041222-21:40:18 : Disconnecting
20041222-21:40:21 : Received logon request
20041222-21:40:21 : Responding to logon request
20041222-21:40:21 : MsgSeqNum too high RECEIVED: 3 EXPECTED: 1
20041222-21:40:21 : Sent ResendRequest FROM: 1 TO: 999999
20041222-21:40:21 : Received logout request
20041222-21:40:21 : Sending logout response
20041222-21:40:21 : Disconnecting
20041222-21:40:24 : Logon state is not valid for message
20041222-21:40:26 : Received logon request
20041222-21:40:26 : Responding to logon request
20041222-21:40:26 : MsgSeqNum too high RECEIVED: 5 EXPECTED: 1
20041222-21:40:26 : Sent ResendRequest FROM: 1 TO: 999999
20041222-21:40:26 : Received logout request
20041222-21:40:26 : Sending logout response
20041222-21:40:26 : Disconnecting
20041222-21:40:31 : Received logon request
20041222-21:40:31 : Responding to logon request
20041222-21:40:31 : MsgSeqNum too high RECEIVED: 7 EXPECTED: 1
20041222-21:40:31 : Sent ResendRequest FROM: 1 TO: 999999
20041222-21:40:31 : Received logout request
20041222-21:40:31 : Sending logout response
20041222-21:40:31 : Disconnecting
20041222-21:40:36 : Received logon request
20041222-21:40:36 : Responding to logon request
20041222-21:40:36 : MsgSeqNum too high RECEIVED: 9 EXPECTED: 1
20041222-21:40:36 : Sent ResendRequest FROM: 1 TO: 999999
20041222-21:40:36 : Received logout request
20041222-21:40:36 : Sending logout response
20041222-21:40:36 : Disconnecting
20041222-21:40:41 : Received logon request
20041222-21:40:41 : Responding to logon request
20041222-21:40:41 : MsgSeqNum too high RECEIVED: 11 EXPECTED: 1
20041222-21:40:41 : Sent ResendRequest FROM: 1 TO: 999999
20041222-21:40:41 : Received logout request
20041222-21:40:41 : Sending logout response
20041222-21:40:41 : Disconnecting


Radhika Sambamurti
Pipeline Trading Systems
Rad...@pi...
212-370-8343

Re: [Quickfix-users] TCP NO DELAY

[Oren M. <or...@qu...>]

Yeah, we should probably support more runtime tcp options.  It would be nice 
to set such things in the config file on a per session basis.

----- Original Message ----- 
From: <que...@bn...>
To: <qui...@li...>
Sent: Wednesday, December 15, 2004 4:35 AM
Subject: [Quickfix-users] TCP NO DELAY


> QuickFIX Documentation: 
> http://www.quickfixengine.org/quickfix/doc/html/index.html
> QuickFIX FAQ: http://www.quickfixengine.org/wikifix/index.php?QuickFixFAQ
> QuickFIX Support: http://www.quickfixengine.org/services.html
>
>
> Hello all,
>
>      To improve performances, I had to add the TCP_NODELAY socket option.
>
>      What do yo think of addng a parameter tcp_nodelay in the config file
> ?
>
> Regards,
> Quentin.
>
>
>
>
>
> This message and any attachments (the "message") is
> intended solely for the addressees and is confidential.
> If you receive this message in error, please delete it and
> immediately notify the sender. Any use not in accord with
> its purpose, any dissemination or disclosure, either whole
> or partial, is prohibited except formal approval. The internet
> can not guarantee the integrity of this message.
> BNP PARIBAS (and its subsidiaries) shall (will) not
> therefore be liable for the message if modified.
>
>                ---------------------------------------------
>
> Ce message et toutes les pieces jointes (ci-apres le
> "message") sont etablis a l'intention exclusive de ses
> destinataires et sont confidentiels. Si vous recevez ce
> message par erreur, merci de le detruire et d'en avertir
> immediatement l'expediteur. Toute utilisation de ce
> message non conforme a sa destination, toute diffusion
> ou toute publication, totale ou partielle, est interdite, sauf
> autorisation expresse. L'internet ne permettant pas
> d'assurer l'integrite de ce message, BNP PARIBAS (et ses
> filiales) decline(nt) toute responsabilite au titre de ce
> message, dans l'hypothese ou il aurait ete modifie.
>
>
>
> -------------------------------------------------------
> SF email is sponsored by - The IT Product Guide
> Read honest & candid reviews on hundreds of IT Products from real users.
> Discover which products truly live up to the hype. Start reading now.
> http://productguide.itmanagersjournal.com/
> _______________________________________________
> Quickfix-users mailing list
> Qui...@li...
> https://lists.sourceforge.net/lists/listinfo/quickfix-users
> 

Re: [Quickfix-users] Possible bug in the Parser when receiving hu ge amounts through th e socket

[Oren M. <or...@qu...>]

No, I got it, I'm just designing a test to expose the problem.  Should be 
able to duplicate in a unit test to the Parser by feeding in the data in the 
same increments that caused your problem.

--oren

----- Original Message ----- 
From: <em...@co...>
To: <or...@qu...>
Cc: <qui...@li...>
Sent: Wednesday, December 15, 2004 3:36 AM
Subject: RE: [Quickfix-users] Possible bug in the Parser when receiving hu 
ge amounts through th e socket


> This is my post from earlier. I guess you didnt see it Oren. Here it goes
> anyway....
>
>
>
>
> Oren,
>
> AM_INIT_AUTOMAKE(quickfix, 1.5.0), but have also take some code from
> AM_INIT_AUTOMAKE(quickfix, 1.9.2) so I can have an upto date version of 
> the
> engine. But,  most of it is 1.5.0 with changes made to a few files in 
> order
> to have an upto date track of the bug fixes.
>
> Also, my test platform is Windows Server 2000 SP4 (quad 550Mhz processor
> machine, 1Gb memory), and have actually also made some changes to how data
> is received on the socket now, and how the socket is created. The socket 
> is
> now created as an overlapped socket explicitly through
> ::WSASocket( PF_INET, SOCK_STREAM, IPPROTO_TCP, NULL, 0, 
> WSA_FLAG_OVERLAPPED
> );
> So, this creates an overlapped socket and allows me to perform 
> asynchronous
> calls on the socket.
>
> I have started of a new thread which creates a completion port, and calls
> 'WSARecv' and then calls 'GetQueuedCompletionStatus' depending on the 
> value
> of the return of 'WSARecv'. The reason I have gone for this approach is 
> that
> when calling 'recv' on the socket and there is 5,000 bytes on the socket,
> and you give it a sufficiently large buffer (lets says 10,000 bytes), it 
> is
> not guaranteed that you are going to receive all 5,000 bytes in that 
> single
> receive. This is explained in the MSDN when calling 'ioctlsocket', or in 
> the
> code 'socket_fionread( m_socket, bytes )'. The below is copied direct from
> MSDN:
> ----------------------------------------------------------------------------
> -
> FIONREAD
> Use to determine the amount of data pending in the network's input buffer
> that can be read from socket s. The argp parameter points to an unsigned
> long value in which ioctlsocket stores the result. FIONREAD returns the
> amount of data that can be read in a single call to the recv function, 
> which
> may not be the same as the total amount of data queued on the socket. If s
> is message oriented (for example, type SOCK_DGRAM), FIONREAD still returns
> the amount of pending data in the network buffer, however, the amount that
> can actually be read in a single call to the recv function is limited to 
> the
> data size written in the send or sendto function call.
> ----------------------------------------------------------------------------
> -
>
> So, we can see that 'ioctlsocket' will let us know how much we can 
> actually
> read from the socket, even though there may be more data pending on the
> socket. So, I came to a scenario where I would have read a buffer of lets
> says 15-20 messages, but the end of the buffer would have looked like
> '8=FI', and on the next 'recv' call I would have got 'X.4.2' ....etc.
>
> I think that it is important to check for the existence of the standard
> trailer in the buffer before trying to extract the length, as it seems at
> least on the Microsoft platform you are not guaranteed to receive all the
> data from socket even if you give it a large enough buffer. I have only 
> been
> able to use a Microsoft platform as this is what we work on, and would be
> very interested to know on what platform you carried out your test.
>
> I do not think that this a problem specifically with any version, because 
> I
> check the very latest version of the parser on the internet
> (http://cvs.sourceforge.net/viewcvs.py/quickfix/quickfix/src/C%2B%2B/Parser.
> cpp?rev=1.11&view=markup) and
>
>
> bool Parser::readFixMessage( std::string& str )
> throw( MessageParseError, RecvFailed )
> { QF_STACK_PUSH(Parser::readFixMessage)
>
>  readFromStream();
>
>  std::string::size_type pos = 0;
>  if( m_buffer.length() < 2 ) return false;
>  pos = m_buffer.find( "8=" );
>  if( pos == std::string::npos ) return false;
>  m_buffer.erase( 0, pos );
>
> /*
> * NEED TO CHECK FOR STANDARD TRAILER HERE TO COMPENSATE FOR
> THE SCENARIO DESCRIBED ABOVE
> */
>
>  int length = 0;
>
>  try
>  {
>    if( extractLength(length, pos, m_buffer) )
>    {
>      pos += length;
>      if( m_buffer.size() < pos )
>        return false;
> -----------------------------------------------------
>
>
> That is the change I had made and have not seen the problem again.
>
> I hope this helps in some way, and once again I say that I have only seen
> this when creating huge amounts of traffic (ie. filling an order for 100
> lots and getting back 100 partial fills, etc), and on Microsoft platform.
>
>
> Regards,
>
>
> Emir Kamber
>
> -----Original Message-----
> From: Oren Miller [mailto:or...@qu...]
> Sent: Tuesday, December 14, 2004 2:57 PM
> To: em...@co...
> Cc: qui...@li...
> Subject: Re: [Quickfix-users] Possible bug in the Parser when receiving
> hu ge amounts through th e socket
>
>
> QuickFIX Documentation:
> http://www.quickfixengine.org/quickfix/doc/html/index.html
> QuickFIX FAQ: http://www.quickfixengine.org/wikifix/index.php?QuickFixFAQ
> QuickFIX Support: http://www.quickfixengine.org/services.html
>
> Emir,
>
> Can you tell me the version that is listed in your configure.in at the
> top, should be in this format AM_INIT_AUTOMAKE(quickfix,
> major.minor.revision)
>
> Here we can at least tell the release yours is based on.  My goal is to
> write a test for that version which duplicates the issue in a
> repeatable manner, and then run the test against the latest version to
> see if the problem still exists.  If not I will apply your patch and
> see if that does the trick.  Thanks.
>
> --oren
>
> On Dec 13, 2004, at 5:16 AM, em...@co... wrote:
>
>> QuickFIX Documentation:
>> http://www.quickfixengine.org/quickfix/doc/html/index.html
>> QuickFIX FAQ:
>> http://www.quickfixengine.org/wikifix/index.php?QuickFixFAQ
>> QuickFIX Support: http://www.quickfixengine.org/services.html
>>
>>
>> I am not quite suire of the version I am using but this is the code
>> that was
>> giving me trouble. I am not sure of the version becase I have
>> customised the
>> code heavily, so I have lately taken code from your updated versions
>> and big
>> fixes and just added, instead of taking a whole new release, sorry
>> about
>> this ;)...
>>
>>
>>
>> The following was giving me trouble. I think the problem is that the
>> 'extractLength' below should not be called before 'pos = m_buffer.find(
>> "\00110=", pos-1 );', this is because you cannot check for the length
>> unless
>> the 'standard trailer' is part of the message. This is the problem I
>> was
>> faced with where the only thing I would have in the buffer when trying
>> to
>> extract the length is '8=FI', and obviously at this point the
>> extractLength
>> would throw an exception. So, I have just modified it to check for the
>> standard message trailer before attempting to extract length and have
>> not
>> seen this problem.
>>
>> Here is the code below.
>>
>> bool Parser::readFixMessage( std::string& str )
>> throw( MessageParseError&, RecvFailed& )
>> { QF_STACK_PUSH(Parser::readFixMessage)
>>
>>   readFromStream();
>>
>>   std::string::size_type pos = 0;
>>
>>   if( m_buffer.length() < 2 )
>>   return false;
>>   pos = m_buffer.find( "8=" );
>>   if( pos == std::string::npos )
>>   return false;
>>
>>
>>   int pos_end_trailer_ = m_buffer.find( "\00110="); \\ I HAVE ADDED
>> THE FOLLOWING 3 LINES
>>   if(pos_end_trailer_ == std::string::npos) \\ AND HAVE
>> SEEMED TO SOLVED THE PRBLEM
>>   return false; \\
>>
>>   if(pos)
>>   m_buffer.erase( 0, pos );
>>
>>   int length = 0;
>>
>>   try
>>   {
>>     if( extractLength(length, pos, m_buffer) )
>>     {
>>       pos += length;
>>       if( m_buffer.size() < pos )
>>         return false;
>>
>>       pos = m_buffer.find( "\00110=", pos-1 );
>>       if( pos == std::string::npos ) return false;
>>       pos += 4;
>>       pos = m_buffer.find( "\001", pos );
>>       if( pos == std::string::npos ) return false;
>>       pos += 1;
>>
>>       str = m_buffer.substr( 0, pos );
>>       m_buffer.erase( 0, pos );
>>       return true;
>>     }
>>   }
>>   catch( MessageParseError& e )
>>   {
>>     m_buffer.erase( 0, pos + length );
>>     throw e;
>>   }
>>
>>   readFromStream();
>>   return false;
>>
>>   QF_STACK_POP
>> }
>>
>>
>> I hope this helps.
>>
>> Regards,
>>
>> Emir
>>
>> -----Original Message-----
>> From: Oren Miller [mailto:or...@qu...]
>> Sent: Friday, December 10, 2004 10:04 PM
>> To: em...@co...; qui...@li...
>> Subject: Re: [Quickfix-users] Possible bug in the Parser when receiving
>> huge amounts through th e socket
>>
>>
>> QuickFIX Documentation:
>> http://www.quickfixengine.org/quickfix/doc/html/index.html
>> QuickFIX FAQ:
>> http://www.quickfixengine.org/wikifix/index.php?QuickFixFAQ
>> QuickFIX Support: http://www.quickfixengine.org/services.html
>>
>> I think I do need some more information about the version you are
>> using.  I
>> wrote a test and ran it against 1.9.4 and was not able to duplicate the
>> behavior described.  The parser has changed over time so it may have
>> been
>> resolved somewhere along the way.  What version are you using?
>>
>> --oren
>>
>> ----- Original Message -----
>> From: <em...@co...>
>> To: <qui...@li...>
>> Sent: Thursday, December 09, 2004 10:14 AM
>> Subject: [Quickfix-users] Possible bug in the Parser when receiving
>> huge
>> amounts through th e socket
>>
>>
>>> QuickFIX Documentation:
>>> http://www.quickfixengine.org/quickfix/doc/html/index.html
>>> QuickFIX FAQ:
>>> http://www.quickfixengine.org/wikifix/index.php?QuickFixFAQ
>>> QuickFIX Support: http://www.quickfixengine.org/services.html
>>>
>>> Here is the scenario.
>>>
>>> At the moment the socket buffer in the parser is 4096. So, I have
>>> recently
>>> generated a lot of traffic so I constantly receive of 4096 bytes on
>>> the
>>> socket and have encountered the following problem.
>>>
>>> The end of the incoming buffer (m_buffer) of 4096 is as follows (some
>>> of
>>> the
>>> values have been substituted to hide trading details):
>>>
>>>
>> 8=FIX.4.2
9=291
35=8
34=156110
369=31450
52=20041209-15:35:
>> 32.687
49=BLA
50=
>>>
>> G
56=D33030N
60=20041209-15:35:
>> 32
59=0
55=GE
54=1
48=BLA000060467
107=BLF5
1
>>>
>> 67=FUT
44=9740.00
41=0
40=2
39=0
38=10
37=20041209004077
151=10
150=0
2
>> 0=0
9
>>>
>> 717=67960
17=00687120041209093532
14=0
11=00067960
6=0
432=20041209
1=1
>> 23456
>>> 78
10=097
8=F
>>>
>>> As we can see, when the parser gets to parsing this part of the
>>> buffer, it
>>> will extract the length of the valid message which is 294 (9=291) and
>>> in
>>> turn send of the message for processing into the application code,
>>> etc.
>>> After, it has processed the message the buffer then looks like:
>>>
>>> 8=F
>>>
>>> This is because the rest of the message is resting on the socket to be
>>> read
>>> so a valid FIX message can be composed.
>>>
>>> As you can see there is a 8=F at the end which was received into the
>>> buffer,
>>> and more data is left over on the socket to be read into the buffer.
>>> But,
>>> at
>>> the moment what the parser will do is make 8=F into a fix message as
>>> this
>>> is
>>> where the buffer is ending, and will try and extract length from it,
>>> and
>>> this generates a series of exceptions (FieldConvertError,
>>> MessageParseError,
>>> FieldNotFound and InvalidMessage).
>>>
>>>
>>> Once the parser tries to parse this message, and throws its
>>> exceptions, it
>>> will get around to reading the socket again and receiving the
>>> following
>>> into
>>> buffer:
>>>
>>>
>> IX.4.2
9=349
35=8
34=156111
369=31450
52=20041209-15:35:
>> 32.692
49=BLA
50=G
5
>>>
>> 6=D33030N
60=20041209-15:35:
>> 32
59=0
55=GE
54=1
48=CME000060467
107=BLF5
167=
>>>
>> FUT
44=9740.00
41=0
40=2
39=2
38=10
37=20041209004077
337=0C0
32=10
31=
>> 9740.
>>>
>> 00
151=0
150=2
20=0
9717=67960
17=00687220041209093532TN0002843
75=2004
>> 1209

>>> 14=0
11=00067960
375=BLA030A1
6=0
432=20041209
1=12345678
10=178

>>>
>>> Once this is received the m_buffer is "8=F" and m_readBuffer is as
>>> above.
>>> When the parser appends m_readBuffer to m_buffer the message looks
>>> like:
>>>
>>>
>> 8=FIX.4.2
9=349
35=8
34=156111
369=31450
52=20041209-15:35:
>> 32.692
49=BLA
50=
>>>
>> G
56=D33030N
60=20041209-15:35:
>> 32
59=0
55=GE
54=1
48=CME000060467
107=BLF5
1
>>>
>> 67=FUT
44=9740.00
41=0
40=2
39=2
38=10
37=20041209004077
337=0C0
32=10

>> 31=97
>>>
>> 40.00
151=0
150=2
20=0
9717=67960
17=00687220041209093532TN0002843
75=2
>> 00412
>>> 09
14=0
11=00067960
375=BLA030A1
6=0
432=20041209
1=12345678
10=178

>>>
>>> a COMPLETE message as it should be.
>>>
>>> Now surely, if this kind of scenario is encountered the parser should
>>> not
>>> even attempt to parse the message, as it is incomplete. I think that
>>> there
>>> needs to be an extra safeguard in the parser that needs to check
>>> presence
>>> of
>>> field 10 (Standard Trailer) in order to verify that it is a complete
>>> message.
>>>
>>> If anyone has any ideas how to solve this, please be my guess as it
>>> has
>>> taken my over 2 days just to track this problem down.
>>>
>>> Thanks a lot
>>>
>>> Emir
>>>
>>> ---
>>> Outgoing mail is certified Virus Free.
>>> Checked by AVG anti-virus system (http://www.grisoft.com).
>>> Version: 6.0.766 / Virus Database: 513 - Release Date: 9/17/2004
>>>
>>>
>>>
>>> -------------------------------------------------------
>>> SF email is sponsored by - The IT Product Guide
>>> Read honest & candid reviews on hundreds of IT Products from real
>>> users.
>>> Discover which products truly live up to the hype. Start reading now.
>>> http://productguide.itmanagersjournal.com/
>>> _______________________________________________
>>> Quickfix-users mailing list
>>> Qui...@li...
>>> https://lists.sourceforge.net/lists/listinfo/quickfix-users
>>>
>>
>>
>>
>> -------------------------------------------------------
>> SF email is sponsored by - The IT Product Guide
>> Read honest & candid reviews on hundreds of IT Products from real
>> users.
>> Discover which products truly live up to the hype. Start reading now.
>> http://productguide.itmanagersjournal.com/
>> _______________________________________________
>> Quickfix-users mailing list
>> Qui...@li...
>> https://lists.sourceforge.net/lists/listinfo/quickfix-users
>>
>> ---
>> Incoming mail is certified Virus Free.
>> Checked by AVG anti-virus system (http://www.grisoft.com).
>> Version: 6.0.766 / Virus Database: 513 - Release Date: 9/17/2004
>>
>>
>> ---
>> Outgoing mail is certified Virus Free.
>> Checked by AVG anti-virus system (http://www.grisoft.com).
>> Version: 6.0.766 / Virus Database: 513 - Release Date: 9/17/2004
>>
>>
>> ---
>> Outgoing mail is certified Virus Free.
>> Checked by AVG anti-virus system (http://www.grisoft.com).
>> Version: 6.0.766 / Virus Database: 513 - Release Date: 9/17/2004
>>
>>
>>
>> -------------------------------------------------------
>> SF email is sponsored by - The IT Product Guide
>> Read honest & candid reviews on hundreds of IT Products from real
>> users.
>> Discover which products truly live up to the hype. Start reading now.
>> http://productguide.itmanagersjournal.com/
>> _______________________________________________
>> Quickfix-users mailing list
>> Qui...@li...
>> https://lists.sourceforge.net/lists/listinfo/quickfix-users
>>
>
>
>
> -------------------------------------------------------
> SF email is sponsored by - The IT Product Guide
> Read honest & candid reviews on hundreds of IT Products from real users.
> Discover which products truly live up to the hype. Start reading now.
> http://productguide.itmanagersjournal.com/
> _______________________________________________
> Quickfix-users mailing list
> Qui...@li...
> https://lists.sourceforge.net/lists/listinfo/quickfix-users
>
> ---
> Incoming mail is certified Virus Free.
> Checked by AVG anti-virus system (http://www.grisoft.com).
> Version: 6.0.766 / Virus Database: 513 - Release Date: 9/17/2004
>
>
> ---
> Outgoing mail is certified Virus Free.
> Checked by AVG anti-virus system (http://www.grisoft.com).
> Version: 6.0.766 / Virus Database: 513 - Release Date: 9/17/2004
>
> 

RE: [Quickfix-users] TCP NO DELAY

[<em...@co...>]

I have added this as well, but this will not necessarily improve
performance. MSDN extract:

-------------------------------------------------
TCP_NODELAY 
The TCP_NODELAY option is specific to TCP/IP service providers. The Nagle
algorithm is disabled if the TCP_NODELAY option is enabled (and vice versa).
The process involves buffering send data when there is unacknowledged data
already in flight or buffering send data until a full-size packet can be
sent. It is highly recommended that TCP/IP service providers enable the
Nagle Algorithm by default, and for the vast majority of application
protocols the Nagle Algorithm can deliver significant performance
enhancements. However, for some applications this algorithm can impede
performance, and TCP_NODELAY can be used to turn it off. These are
applications where many small messages are sent, and the time delays between
the messages are maintained. Application writers should not set TCP_NODELAY
unless the impact of doing so is well-understood and desired because setting
TCP_NODELAY can have a significant negative impact on network and
application performance. 
-------------------------------------------------

Just make sure that you have added for the right reasons, and just because
it says that it will put data onto the wire immediately. This option can
cause a vast number of collisions and retransmissions on a very busy line.

Regards,


Emir

-----Original Message-----
From: que...@bn...
[mailto:que...@bn...]
Sent: Wednesday, December 15, 2004 10:36 AM
To: qui...@li...
Subject: [Quickfix-users] TCP NO DELAY


QuickFIX Documentation:
http://www.quickfixengine.org/quickfix/doc/html/index.html
QuickFIX FAQ: http://www.quickfixengine.org/wikifix/index.php?QuickFixFAQ
QuickFIX Support: http://www.quickfixengine.org/services.html


Hello all,

      To improve performances, I had to add the TCP_NODELAY socket option.

      What do yo think of addng a parameter tcp_nodelay in the config file
?

Regards,
Quentin.





This message and any attachments (the "message") is
intended solely for the addressees and is confidential. 
If you receive this message in error, please delete it and 
immediately notify the sender. Any use not in accord with 
its purpose, any dissemination or disclosure, either whole 
or partial, is prohibited except formal approval. The internet
can not guarantee the integrity of this message. 
BNP PARIBAS (and its subsidiaries) shall (will) not 
therefore be liable for the message if modified. 

                ---------------------------------------------

Ce message et toutes les pieces jointes (ci-apres le 
"message") sont etablis a l'intention exclusive de ses 
destinataires et sont confidentiels. Si vous recevez ce 
message par erreur, merci de le detruire et d'en avertir 
immediatement l'expediteur. Toute utilisation de ce 
message non conforme a sa destination, toute diffusion 
ou toute publication, totale ou partielle, est interdite, sauf 
autorisation expresse. L'internet ne permettant pas 
d'assurer l'integrite de ce message, BNP PARIBAS (et ses
filiales) decline(nt) toute responsabilite au titre de ce 
message, dans l'hypothese ou il aurait ete modifie.



-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now. 
http://productguide.itmanagersjournal.com/
_______________________________________________
Quickfix-users mailing list
Qui...@li...
https://lists.sourceforge.net/lists/listinfo/quickfix-users

---
Incoming mail is certified Virus Free.
Checked by AVG anti-virus system (http://www.grisoft.com).
Version: 6.0.766 / Virus Database: 513 - Release Date: 9/17/2004
 

---
Outgoing mail is certified Virus Free.
Checked by AVG anti-virus system (http://www.grisoft.com).
Version: 6.0.766 / Virus Database: 513 - Release Date: 9/17/2004
 

[Quickfix-users] TCP NO DELAY

[<que...@bn...>]

Hello all,

      To improve performances, I had to add the TCP_NODELAY socket option.

      What do yo think of addng a parameter tcp_nodelay in the config file
?

Regards,
Quentin.





This message and any attachments (the "message") is
intended solely for the addressees and is confidential. 
If you receive this message in error, please delete it and 
immediately notify the sender. Any use not in accord with 
its purpose, any dissemination or disclosure, either whole 
or partial, is prohibited except formal approval. The internet
can not guarantee the integrity of this message. 
BNP PARIBAS (and its subsidiaries) shall (will) not 
therefore be liable for the message if modified. 

                ---------------------------------------------

Ce message et toutes les pieces jointes (ci-apres le 
"message") sont etablis a l'intention exclusive de ses 
destinataires et sont confidentiels. Si vous recevez ce 
message par erreur, merci de le detruire et d'en avertir 
immediatement l'expediteur. Toute utilisation de ce 
message non conforme a sa destination, toute diffusion 
ou toute publication, totale ou partielle, est interdite, sauf 
autorisation expresse. L'internet ne permettant pas 
d'assurer l'integrite de ce message, BNP PARIBAS (et ses
filiales) decline(nt) toute responsabilite au titre de ce 
message, dans l'hypothese ou il aurait ete modifie.

RE: [Quickfix-users] Possible bug in the Parser when receiving hu ge amounts through th e socket

[<em...@co...>]

This is my post from earlier. I guess you didnt see it Oren. Here it goes
anyway....




Oren,

AM_INIT_AUTOMAKE(quickfix, 1.5.0), but have also take some code from
AM_INIT_AUTOMAKE(quickfix, 1.9.2) so I can have an upto date version of the
engine. But,  most of it is 1.5.0 with changes made to a few files in order
to have an upto date track of the bug fixes.

Also, my test platform is Windows Server 2000 SP4 (quad 550Mhz processor
machine, 1Gb memory), and have actually also made some changes to how data
is received on the socket now, and how the socket is created. The socket is
now created as an overlapped socket explicitly through
::WSASocket( PF_INET, SOCK_STREAM, IPPROTO_TCP, NULL, 0, WSA_FLAG_OVERLAPPED
);
So, this creates an overlapped socket and allows me to perform asynchronous
calls on the socket. 

I have started of a new thread which creates a completion port, and calls
'WSARecv' and then calls 'GetQueuedCompletionStatus' depending on the value
of the return of 'WSARecv'. The reason I have gone for this approach is that
when calling 'recv' on the socket and there is 5,000 bytes on the socket,
and you give it a sufficiently large buffer (lets says 10,000 bytes), it is
not guaranteed that you are going to receive all 5,000 bytes in that single
receive. This is explained in the MSDN when calling 'ioctlsocket', or in the
code 'socket_fionread( m_socket, bytes )'. The below is copied direct from
MSDN:
----------------------------------------------------------------------------
-
FIONREAD 
Use to determine the amount of data pending in the network's input buffer
that can be read from socket s. The argp parameter points to an unsigned
long value in which ioctlsocket stores the result. FIONREAD returns the
amount of data that can be read in a single call to the recv function, which
may not be the same as the total amount of data queued on the socket. If s
is message oriented (for example, type SOCK_DGRAM), FIONREAD still returns
the amount of pending data in the network buffer, however, the amount that
can actually be read in a single call to the recv function is limited to the
data size written in the send or sendto function call. 
----------------------------------------------------------------------------
-

So, we can see that 'ioctlsocket' will let us know how much we can actually
read from the socket, even though there may be more data pending on the
socket. So, I came to a scenario where I would have read a buffer of lets
says 15-20 messages, but the end of the buffer would have looked like
'8=FI', and on the next 'recv' call I would have got 'X.4.2' ....etc.

I think that it is important to check for the existence of the standard
trailer in the buffer before trying to extract the length, as it seems at
least on the Microsoft platform you are not guaranteed to receive all the
data from socket even if you give it a large enough buffer. I have only been
able to use a Microsoft platform as this is what we work on, and would be
very interested to know on what platform you carried out your test.

I do not think that this a problem specifically with any version, because I
check the very latest version of the parser on the internet
(http://cvs.sourceforge.net/viewcvs.py/quickfix/quickfix/src/C%2B%2B/Parser.
cpp?rev=1.11&view=markup) and 


bool Parser::readFixMessage( std::string& str )
throw( MessageParseError, RecvFailed )
{ QF_STACK_PUSH(Parser::readFixMessage)

  readFromStream();

  std::string::size_type pos = 0;
  if( m_buffer.length() < 2 ) return false;
  pos = m_buffer.find( "8=" );
  if( pos == std::string::npos ) return false;
  m_buffer.erase( 0, pos );

	/*
	*	NEED TO CHECK FOR STANDARD TRAILER HERE TO COMPENSATE FOR
THE SCENARIO DESCRIBED ABOVE
	*/

  int length = 0;

  try
  {
    if( extractLength(length, pos, m_buffer) )
    {
      pos += length;
      if( m_buffer.size() < pos )
        return false;
-----------------------------------------------------


That is the change I had made and have not seen the problem again.

I hope this helps in some way, and once again I say that I have only seen
this when creating huge amounts of traffic (ie. filling an order for 100
lots and getting back 100 partial fills, etc), and on Microsoft platform.


Regards,


Emir Kamber

-----Original Message-----
From: Oren Miller [mailto:or...@qu...]
Sent: Tuesday, December 14, 2004 2:57 PM
To: em...@co...
Cc: qui...@li...
Subject: Re: [Quickfix-users] Possible bug in the Parser when receiving
hu ge amounts through th e socket


QuickFIX Documentation:
http://www.quickfixengine.org/quickfix/doc/html/index.html
QuickFIX FAQ: http://www.quickfixengine.org/wikifix/index.php?QuickFixFAQ
QuickFIX Support: http://www.quickfixengine.org/services.html

Emir,

Can you tell me the version that is listed in your configure.in at the  
top, should be in this format AM_INIT_AUTOMAKE(quickfix,  
major.minor.revision)

Here we can at least tell the release yours is based on.  My goal is to  
write a test for that version which duplicates the issue in a  
repeatable manner, and then run the test against the latest version to  
see if the problem still exists.  If not I will apply your patch and  
see if that does the trick.  Thanks.

--oren

On Dec 13, 2004, at 5:16 AM, em...@co... wrote:

> QuickFIX Documentation:  
> http://www.quickfixengine.org/quickfix/doc/html/index.html
> QuickFIX FAQ:  
> http://www.quickfixengine.org/wikifix/index.php?QuickFixFAQ
> QuickFIX Support: http://www.quickfixengine.org/services.html
>
>
> I am not quite suire of the version I am using but this is the code  
> that was
> giving me trouble. I am not sure of the version becase I have  
> customised the
> code heavily, so I have lately taken code from your updated versions  
> and big
> fixes and just added, instead of taking a whole new release, sorry  
> about
> this ;)...
>
>
>
> The following was giving me trouble. I think the problem is that the
> 'extractLength' below should not be called before 'pos = m_buffer.find(
> "\00110=", pos-1 );', this is because you cannot check for the length  
> unless
> the 'standard trailer' is part of the message. This is the problem I  
> was
> faced with where the only thing I would have in the buffer when trying  
> to
> extract the length is '8=FI', and obviously at this point the  
> extractLength
> would throw an exception. So, I have just modified it to check for the
> standard message trailer before attempting to extract length and have  
> not
> seen this problem.
>
> Here is the code below.
>
> bool Parser::readFixMessage( std::string& str )
> throw( MessageParseError&, RecvFailed& )
> { QF_STACK_PUSH(Parser::readFixMessage)
>
>   readFromStream();
>
>   std::string::size_type pos = 0;
>
>   if( m_buffer.length() < 2 )
> 	  return false;
>   pos = m_buffer.find( "8=" );
>   if( pos == std::string::npos )
> 	  return false;
>
>
>   int pos_end_trailer_ = m_buffer.find( "\00110=");	\\	I HAVE ADDED
> THE FOLLOWING 3 LINES
>   if(pos_end_trailer_ == std::string::npos)		\\ 	AND HAVE
> SEEMED TO SOLVED THE PRBLEM
> 	  return false;						\\
>
>   if(pos)
> 	  m_buffer.erase( 0, pos );
>
>   int length = 0;
>
>   try
>   {
>     if( extractLength(length, pos, m_buffer) )
>     {
>       pos += length;
>       if( m_buffer.size() < pos )
>         return false;
>
>       pos = m_buffer.find( "\00110=", pos-1 );
>       if( pos == std::string::npos ) return false;
>       pos += 4;
>       pos = m_buffer.find( "\001", pos );
>       if( pos == std::string::npos ) return false;
>       pos += 1;
>
>       str = m_buffer.substr( 0, pos );
>       m_buffer.erase( 0, pos );
>       return true;
>     }
>   }
>   catch( MessageParseError& e )
>   {
>     m_buffer.erase( 0, pos + length );
>     throw e;
>   }
>
>   readFromStream();
>   return false;
>
>   QF_STACK_POP
> }
>
>
> I hope this helps.
>
> Regards,
>
> Emir
>
> -----Original Message-----
> From: Oren Miller [mailto:or...@qu...]
> Sent: Friday, December 10, 2004 10:04 PM
> To: em...@co...; qui...@li...
> Subject: Re: [Quickfix-users] Possible bug in the Parser when receiving
> huge amounts through th e socket
>
>
> QuickFIX Documentation:
> http://www.quickfixengine.org/quickfix/doc/html/index.html
> QuickFIX FAQ:  
> http://www.quickfixengine.org/wikifix/index.php?QuickFixFAQ
> QuickFIX Support: http://www.quickfixengine.org/services.html
>
> I think I do need some more information about the version you are  
> using.  I
> wrote a test and ran it against 1.9.4 and was not able to duplicate the
> behavior described.  The parser has changed over time so it may have  
> been
> resolved somewhere along the way.  What version are you using?
>
> --oren
>
> ----- Original Message -----
> From: <em...@co...>
> To: <qui...@li...>
> Sent: Thursday, December 09, 2004 10:14 AM
> Subject: [Quickfix-users] Possible bug in the Parser when receiving  
> huge
> amounts through th e socket
>
>
>> QuickFIX Documentation:
>> http://www.quickfixengine.org/quickfix/doc/html/index.html
>> QuickFIX FAQ:  
>> http://www.quickfixengine.org/wikifix/index.php?QuickFixFAQ
>> QuickFIX Support: http://www.quickfixengine.org/services.html
>>
>> Here is the scenario.
>>
>> At the moment the socket buffer in the parser is 4096. So, I have  
>> recently
>> generated a lot of traffic so I constantly receive of 4096 bytes on  
>> the
>> socket and have encountered the following problem.
>>
>> The end of the incoming buffer (m_buffer) of 4096 is as follows (some  
>> of
>> the
>> values have been substituted to hide trading details):
>>
>>
> 8=FIX.4.2
9=291
35=8
34=156110
369=31450
52=20041209-15:35: 
> 32.687
49=BLA
50=
>>
> G
56=D33030N
60=20041209-15:35: 
> 32
59=0
55=GE
54=1
48=BLA000060467
107=BLF5
1
>>
> 67=FUT
44=9740.00
41=0
40=2
39=0
38=10
37=20041209004077
151=10
150=0
2 
> 0=0
9
>>
> 717=67960
17=00687120041209093532
14=0
11=00067960
6=0
432=20041209
1=1 
> 23456
>> 78
10=097
8=F
>>
>> As we can see, when the parser gets to parsing this part of the  
>> buffer, it
>> will extract the length of the valid message which is 294 (9=291) and  
>> in
>> turn send of the message for processing into the application code,  
>> etc.
>> After, it has processed the message the buffer then looks like:
>>
>> 8=F
>>
>> This is because the rest of the message is resting on the socket to be
>> read
>> so a valid FIX message can be composed.
>>
>> As you can see there is a 8=F at the end which was received into the
>> buffer,
>> and more data is left over on the socket to be read into the buffer.  
>> But,
>> at
>> the moment what the parser will do is make 8=F into a fix message as  
>> this
>> is
>> where the buffer is ending, and will try and extract length from it,  
>> and
>> this generates a series of exceptions (FieldConvertError,
>> MessageParseError,
>> FieldNotFound and InvalidMessage).
>>
>>
>> Once the parser tries to parse this message, and throws its  
>> exceptions, it
>> will get around to reading the socket again and receiving the  
>> following
>> into
>> buffer:
>>
>>
> IX.4.2
9=349
35=8
34=156111
369=31450
52=20041209-15:35: 
> 32.692
49=BLA
50=G
5
>>
> 6=D33030N
60=20041209-15:35: 
> 32
59=0
55=GE
54=1
48=CME000060467
107=BLF5
167=
>>
> FUT
44=9740.00
41=0
40=2
39=2
38=10
37=20041209004077
337=0C0
32=10
31= 
> 9740.
>>
> 00
151=0
150=2
20=0
9717=67960
17=00687220041209093532TN0002843
75=2004 
> 1209

>> 14=0
11=00067960
375=BLA030A1
6=0
432=20041209
1=12345678
10=178

>>
>> Once this is received the m_buffer is "8=F" and m_readBuffer is as  
>> above.
>> When the parser appends m_readBuffer to m_buffer the message looks  
>> like:
>>
>>
> 8=FIX.4.2
9=349
35=8
34=156111
369=31450
52=20041209-15:35: 
> 32.692
49=BLA
50=
>>
> G
56=D33030N
60=20041209-15:35: 
> 32
59=0
55=GE
54=1
48=CME000060467
107=BLF5
1
>>
> 67=FUT
44=9740.00
41=0
40=2
39=2
38=10
37=20041209004077
337=0C0
32=10
 
> 31=97
>>
> 40.00
151=0
150=2
20=0
9717=67960
17=00687220041209093532TN0002843
75=2 
> 00412
>> 09
14=0
11=00067960
375=BLA030A1
6=0
432=20041209
1=12345678
10=178

>>
>> a COMPLETE message as it should be.
>>
>> Now surely, if this kind of scenario is encountered the parser should  
>> not
>> even attempt to parse the message, as it is incomplete. I think that  
>> there
>> needs to be an extra safeguard in the parser that needs to check  
>> presence
>> of
>> field 10 (Standard Trailer) in order to verify that it is a complete
>> message.
>>
>> If anyone has any ideas how to solve this, please be my guess as it  
>> has
>> taken my over 2 days just to track this problem down.
>>
>> Thanks a lot
>>
>> Emir
>>
>> ---
>> Outgoing mail is certified Virus Free.
>> Checked by AVG anti-virus system (http://www.grisoft.com).
>> Version: 6.0.766 / Virus Database: 513 - Release Date: 9/17/2004
>>
>>
>>
>> -------------------------------------------------------
>> SF email is sponsored by - The IT Product Guide
>> Read honest & candid reviews on hundreds of IT Products from real  
>> users.
>> Discover which products truly live up to the hype. Start reading now.
>> http://productguide.itmanagersjournal.com/
>> _______________________________________________
>> Quickfix-users mailing list
>> Qui...@li...
>> https://lists.sourceforge.net/lists/listinfo/quickfix-users
>>
>
>
>
> -------------------------------------------------------
> SF email is sponsored by - The IT Product Guide
> Read honest & candid reviews on hundreds of IT Products from real  
> users.
> Discover which products truly live up to the hype. Start reading now.
> http://productguide.itmanagersjournal.com/
> _______________________________________________
> Quickfix-users mailing list
> Qui...@li...
> https://lists.sourceforge.net/lists/listinfo/quickfix-users
>
> ---
> Incoming mail is certified Virus Free.
> Checked by AVG anti-virus system (http://www.grisoft.com).
> Version: 6.0.766 / Virus Database: 513 - Release Date: 9/17/2004
>
>
> ---
> Outgoing mail is certified Virus Free.
> Checked by AVG anti-virus system (http://www.grisoft.com).
> Version: 6.0.766 / Virus Database: 513 - Release Date: 9/17/2004
>
>
> ---
> Outgoing mail is certified Virus Free.
> Checked by AVG anti-virus system (http://www.grisoft.com).
> Version: 6.0.766 / Virus Database: 513 - Release Date: 9/17/2004
>
>
>
> -------------------------------------------------------
> SF email is sponsored by - The IT Product Guide
> Read honest & candid reviews on hundreds of IT Products from real  
> users.
> Discover which products truly live up to the hype. Start reading now.
> http://productguide.itmanagersjournal.com/
> _______________________________________________
> Quickfix-users mailing list
> Qui...@li...
> https://lists.sourceforge.net/lists/listinfo/quickfix-users
>



-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now. 
http://productguide.itmanagersjournal.com/
_______________________________________________
Quickfix-users mailing list
Qui...@li...
https://lists.sourceforge.net/lists/listinfo/quickfix-users

---
Incoming mail is certified Virus Free.
Checked by AVG anti-virus system (http://www.grisoft.com).
Version: 6.0.766 / Virus Database: 513 - Release Date: 9/17/2004
 

---
Outgoing mail is certified Virus Free.
Checked by AVG anti-virus system (http://www.grisoft.com).
Version: 6.0.766 / Virus Database: 513 - Release Date: 9/17/2004
 

RE: [Quickfix-users] Possible bug in the Parser when receiving hu ge amounts through th e socket

[<em...@co...>]

Oren,

AM_INIT_AUTOMAKE(quickfix, 1.5.0), but have also take some code from
AM_INIT_AUTOMAKE(quickfix, 1.9.2) so I can have an upto date version of the
engine. But,  most of it is 1.5.0 with changes made to a few files in order
to have an upto date track of the bug fixes.

Also, my test platform is Windows Server 2000 SP4 (quad 550Mhz processor
machine, 1Gb memory), and have actually also made some changes to how data
is received on the socket now, and how the socket is created. The socket is
now created as an overlapped socket explicitly through
::WSASocket( PF_INET, SOCK_STREAM, IPPROTO_TCP, NULL, 0, WSA_FLAG_OVERLAPPED
);
So, this creates an overlapped socket and allows me to perform asynchronous
calls on the socket. 

I have started of a new thread which creates a completion port, and calls
'WSARecv' and then calls 'GetQueuedCompletionStatus' depending on the value
of the return of 'WSARecv'. The reason I have gone for this approach is that
when calling 'recv' on the socket and there is 5,000 bytes on the socket,
and you give it a sufficiently large buffer (lets says 10,000 bytes), it is
not guaranteed that you are going to receive all 5,000 bytes in that single
receive. This is explained in the MSDN when calling 'ioctlsocket', or in the
code 'socket_fionread( m_socket, bytes )'. The below is copied direct from
MSDN:
----------------------------------------------------------------------------
-
FIONREAD 
Use to determine the amount of data pending in the network's input buffer
that can be read from socket s. The argp parameter points to an unsigned
long value in which ioctlsocket stores the result. FIONREAD returns the
amount of data that can be read in a single call to the recv function, which
may not be the same as the total amount of data queued on the socket. If s
is message oriented (for example, type SOCK_DGRAM), FIONREAD still returns
the amount of pending data in the network buffer, however, the amount that
can actually be read in a single call to the recv function is limited to the
data size written in the send or sendto function call. 
----------------------------------------------------------------------------
-

So, we can see that 'ioctlsocket' will let us know how much we can actually
read from the socket, even though there may be more data pending on the
socket. So, I came to a scenario where I would have read a buffer of lets
says 15-20 messages, but the end of the buffer would have looked like
'8=FI', and on the next 'recv' call I would have got 'X.4.2' ....etc.

I think that it is important to check for the existence of the standard
trailer in the buffer before trying to extract the length, as it seems at
least on the Microsoft platform you are not guaranteed to receive all the
data from socket even if you give it a large enough buffer. I have only been
able to use a Microsoft platform as this is what we work on, and would be
very interested to know on what platform you carried out your test.

I do not think that this a problem specifically with any version, because I
check the very latest version of the parser on the internet
(http://cvs.sourceforge.net/viewcvs.py/quickfix/quickfix/src/C%2B%2B/Parser.
cpp?rev=1.11&view=markup) and 


bool Parser::readFixMessage( std::string& str )
throw( MessageParseError, RecvFailed )
{ QF_STACK_PUSH(Parser::readFixMessage)

  readFromStream();

  std::string::size_type pos = 0;
  if( m_buffer.length() < 2 ) return false;
  pos = m_buffer.find( "8=" );
  if( pos == std::string::npos ) return false;
  m_buffer.erase( 0, pos );

	/*
	*	NEED TO CHECK FOR STANDARD TRAILER HERE TO COMPENSATE FOR
THE SCENARIO DESCRIBED ABOVE
	*/

  int length = 0;

  try
  {
    if( extractLength(length, pos, m_buffer) )
    {
      pos += length;
      if( m_buffer.size() < pos )
        return false;
-----------------------------------------------------


That is the change I had made and have not seen the problem again.

I hope this helps in some way, and once again I say that I have only seen
this when creating huge amounts of traffic (ie. filling an order for 100
lots and getting back 100 partial fills, etc), and on Microsoft platform.


Regards,


Emir Kamber



-----Original Message-----
From: Oren Miller [mailto:or...@qu...]
Sent: Tuesday, December 14, 2004 2:57 PM
To: em...@co...
Cc: qui...@li...
Subject: Re: [Quickfix-users] Possible bug in the Parser when receiving
hu ge amounts through th e socket


Emir,

Can you tell me the version that is listed in your configure.in at the  
top, should be in this format AM_INIT_AUTOMAKE(quickfix,  
major.minor.revision)

Here we can at least tell the release yours is based on.  My goal is to  
write a test for that version which duplicates the issue in a  
repeatable manner, and then run the test against the latest version to  
see if the problem still exists.  If not I will apply your patch and  
see if that does the trick.  Thanks.

--oren

On Dec 13, 2004, at 5:16 AM, em...@co... wrote:

> QuickFIX Documentation:  
> http://www.quickfixengine.org/quickfix/doc/html/index.html
> QuickFIX FAQ:  
> http://www.quickfixengine.org/wikifix/index.php?QuickFixFAQ
> QuickFIX Support: http://www.quickfixengine.org/services.html
>
>
> I am not quite suire of the version I am using but this is the code  
> that was
> giving me trouble. I am not sure of the version becase I have  
> customised the
> code heavily, so I have lately taken code from your updated versions  
> and big
> fixes and just added, instead of taking a whole new release, sorry  
> about
> this ;)...
>
>
>
> The following was giving me trouble. I think the problem is that the
> 'extractLength' below should not be called before 'pos = m_buffer.find(
> "\00110=", pos-1 );', this is because you cannot check for the length  
> unless
> the 'standard trailer' is part of the message. This is the problem I  
> was
> faced with where the only thing I would have in the buffer when trying  
> to
> extract the length is '8=FI', and obviously at this point the  
> extractLength
> would throw an exception. So, I have just modified it to check for the
> standard message trailer before attempting to extract length and have  
> not
> seen this problem.
>
> Here is the code below.
>
> bool Parser::readFixMessage( std::string& str )
> throw( MessageParseError&, RecvFailed& )
> { QF_STACK_PUSH(Parser::readFixMessage)
>
>   readFromStream();
>
>   std::string::size_type pos = 0;
>
>   if( m_buffer.length() < 2 )
> 	  return false;
>   pos = m_buffer.find( "8=" );
>   if( pos == std::string::npos )
> 	  return false;
>
>
>   int pos_end_trailer_ = m_buffer.find( "\00110=");	\\	I HAVE ADDED
> THE FOLLOWING 3 LINES
>   if(pos_end_trailer_ == std::string::npos)		\\ 	AND HAVE
> SEEMED TO SOLVED THE PRBLEM
> 	  return false;						\\
>
>   if(pos)
> 	  m_buffer.erase( 0, pos );
>
>   int length = 0;
>
>   try
>   {
>     if( extractLength(length, pos, m_buffer) )
>     {
>       pos += length;
>       if( m_buffer.size() < pos )
>         return false;
>
>       pos = m_buffer.find( "\00110=", pos-1 );
>       if( pos == std::string::npos ) return false;
>       pos += 4;
>       pos = m_buffer.find( "\001", pos );
>       if( pos == std::string::npos ) return false;
>       pos += 1;
>
>       str = m_buffer.substr( 0, pos );
>       m_buffer.erase( 0, pos );
>       return true;
>     }
>   }
>   catch( MessageParseError& e )
>   {
>     m_buffer.erase( 0, pos + length );
>     throw e;
>   }
>
>   readFromStream();
>   return false;
>
>   QF_STACK_POP
> }
>
>
> I hope this helps.
>
> Regards,
>
> Emir
>
> -----Original Message-----
> From: Oren Miller [mailto:or...@qu...]
> Sent: Friday, December 10, 2004 10:04 PM
> To: em...@co...; qui...@li...
> Subject: Re: [Quickfix-users] Possible bug in the Parser when receiving
> huge amounts through th e socket
>
>
> QuickFIX Documentation:
> http://www.quickfixengine.org/quickfix/doc/html/index.html
> QuickFIX FAQ:  
> http://www.quickfixengine.org/wikifix/index.php?QuickFixFAQ
> QuickFIX Support: http://www.quickfixengine.org/services.html
>
> I think I do need some more information about the version you are  
> using.  I
> wrote a test and ran it against 1.9.4 and was not able to duplicate the
> behavior described.  The parser has changed over time so it may have  
> been
> resolved somewhere along the way.  What version are you using?
>
> --oren
>
> ----- Original Message -----
> From: <em...@co...>
> To: <qui...@li...>
> Sent: Thursday, December 09, 2004 10:14 AM
> Subject: [Quickfix-users] Possible bug in the Parser when receiving  
> huge
> amounts through th e socket
>
>
>> QuickFIX Documentation:
>> http://www.quickfixengine.org/quickfix/doc/html/index.html
>> QuickFIX FAQ:  
>> http://www.quickfixengine.org/wikifix/index.php?QuickFixFAQ
>> QuickFIX Support: http://www.quickfixengine.org/services.html
>>
>> Here is the scenario.
>>
>> At the moment the socket buffer in the parser is 4096. So, I have  
>> recently
>> generated a lot of traffic so I constantly receive of 4096 bytes on  
>> the
>> socket and have encountered the following problem.
>>
>> The end of the incoming buffer (m_buffer) of 4096 is as follows (some  
>> of
>> the
>> values have been substituted to hide trading details):
>>
>>
> 8=FIX.4.2
9=291
35=8
34=156110
369=31450
52=20041209-15:35: 
> 32.687
49=BLA
50=
>>
> G
56=D33030N
60=20041209-15:35: 
> 32
59=0
55=GE
54=1
48=BLA000060467
107=BLF5
1
>>
> 67=FUT
44=9740.00
41=0
40=2
39=0
38=10
37=20041209004077
151=10
150=0
2 
> 0=0
9
>>
> 717=67960
17=00687120041209093532
14=0
11=00067960
6=0
432=20041209
1=1 
> 23456
>> 78
10=097
8=F
>>
>> As we can see, when the parser gets to parsing this part of the  
>> buffer, it
>> will extract the length of the valid message which is 294 (9=291) and  
>> in
>> turn send of the message for processing into the application code,  
>> etc.
>> After, it has processed the message the buffer then looks like:
>>
>> 8=F
>>
>> This is because the rest of the message is resting on the socket to be
>> read
>> so a valid FIX message can be composed.
>>
>> As you can see there is a 8=F at the end which was received into the
>> buffer,
>> and more data is left over on the socket to be read into the buffer.  
>> But,
>> at
>> the moment what the parser will do is make 8=F into a fix message as  
>> this
>> is
>> where the buffer is ending, and will try and extract length from it,  
>> and
>> this generates a series of exceptions (FieldConvertError,
>> MessageParseError,
>> FieldNotFound and InvalidMessage).
>>
>>
>> Once the parser tries to parse this message, and throws its  
>> exceptions, it
>> will get around to reading the socket again and receiving the  
>> following
>> into
>> buffer:
>>
>>
> IX.4.2
9=349
35=8
34=156111
369=31450
52=20041209-15:35: 
> 32.692
49=BLA
50=G
5
>>
> 6=D33030N
60=20041209-15:35: 
> 32
59=0
55=GE
54=1
48=CME000060467
107=BLF5
167=
>>
> FUT
44=9740.00
41=0
40=2
39=2
38=10
37=20041209004077
337=0C0
32=10
31= 
> 9740.
>>
> 00
151=0
150=2
20=0
9717=67960
17=00687220041209093532TN0002843
75=2004 
> 1209

>> 14=0
11=00067960
375=BLA030A1
6=0
432=20041209
1=12345678
10=178

>>
>> Once this is received the m_buffer is "8=F" and m_readBuffer is as  
>> above.
>> When the parser appends m_readBuffer to m_buffer the message looks  
>> like:
>>
>>
> 8=FIX.4.2
9=349
35=8
34=156111
369=31450
52=20041209-15:35: 
> 32.692
49=BLA
50=
>>
> G
56=D33030N
60=20041209-15:35: 
> 32
59=0
55=GE
54=1
48=CME000060467
107=BLF5
1
>>
> 67=FUT
44=9740.00
41=0
40=2
39=2
38=10
37=20041209004077
337=0C0
32=10
 
> 31=97
>>
> 40.00
151=0
150=2
20=0
9717=67960
17=00687220041209093532TN0002843
75=2 
> 00412
>> 09
14=0
11=00067960
375=BLA030A1
6=0
432=20041209
1=12345678
10=178

>>
>> a COMPLETE message as it should be.
>>
>> Now surely, if this kind of scenario is encountered the parser should  
>> not
>> even attempt to parse the message, as it is incomplete. I think that  
>> there
>> needs to be an extra safeguard in the parser that needs to check  
>> presence
>> of
>> field 10 (Standard Trailer) in order to verify that it is a complete
>> message.
>>
>> If anyone has any ideas how to solve this, please be my guess as it  
>> has
>> taken my over 2 days just to track this problem down.
>>
>> Thanks a lot
>>
>> Emir
>>
>> ---
>> Outgoing mail is certified Virus Free.
>> Checked by AVG anti-virus system (http://www.grisoft.com).
>> Version: 6.0.766 / Virus Database: 513 - Release Date: 9/17/2004
>>
>>
>>
>> -------------------------------------------------------
>> SF email is sponsored by - The IT Product Guide
>> Read honest & candid reviews on hundreds of IT Products from real  
>> users.
>> Discover which products truly live up to the hype. Start reading now.
>> http://productguide.itmanagersjournal.com/
>> _______________________________________________
>> Quickfix-users mailing list
>> Qui...@li...
>> https://lists.sourceforge.net/lists/listinfo/quickfix-users
>>
>
>
>
> -------------------------------------------------------
> SF email is sponsored by - The IT Product Guide
> Read honest & candid reviews on hundreds of IT Products from real  
> users.
> Discover which products truly live up to the hype. Start reading now.
> http://productguide.itmanagersjournal.com/
> _______________________________________________
> Quickfix-users mailing list
> Qui...@li...
> https://lists.sourceforge.net/lists/listinfo/quickfix-users
>
> ---
> Incoming mail is certified Virus Free.
> Checked by AVG anti-virus system (http://www.grisoft.com).
> Version: 6.0.766 / Virus Database: 513 - Release Date: 9/17/2004
>
>
> ---
> Outgoing mail is certified Virus Free.
> Checked by AVG anti-virus system (http://www.grisoft.com).
> Version: 6.0.766 / Virus Database: 513 - Release Date: 9/17/2004
>
>
> ---
> Outgoing mail is certified Virus Free.
> Checked by AVG anti-virus system (http://www.grisoft.com).
> Version: 6.0.766 / Virus Database: 513 - Release Date: 9/17/2004
>
>
>
> -------------------------------------------------------
> SF email is sponsored by - The IT Product Guide
> Read honest & candid reviews on hundreds of IT Products from real  
> users.
> Discover which products truly live up to the hype. Start reading now.
> http://productguide.itmanagersjournal.com/
> _______________________________________________
> Quickfix-users mailing list
> Qui...@li...
> https://lists.sourceforge.net/lists/listinfo/quickfix-users
>

---
Incoming mail is certified Virus Free.
Checked by AVG anti-virus system (http://www.grisoft.com).
Version: 6.0.766 / Virus Database: 513 - Release Date: 9/17/2004
 

---
Outgoing mail is certified Virus Free.
Checked by AVG anti-virus system (http://www.grisoft.com).
Version: 6.0.766 / Virus Database: 513 - Release Date: 9/17/2004
 

Re: [Quickfix-users] Possible bug in the Parser when receiving hu ge amounts through th e socket

[Oren M. <or...@qu...>]

Emir,

Can you tell me the version that is listed in your configure.in at the =20=

top, should be in this format AM_INIT_AUTOMAKE(quickfix, =20
major.minor.revision)

Here we can at least tell the release yours is based on.  My goal is to =20=

write a test for that version which duplicates the issue in a =20
repeatable manner, and then run the test against the latest version to =20=

see if the problem still exists.  If not I will apply your patch and =20
see if that does the trick.  Thanks.

--oren

On Dec 13, 2004, at 5:16 AM, em...@co... wrote:

> QuickFIX Documentation: =20
> http://www.quickfixengine.org/quickfix/doc/html/index.html
> QuickFIX FAQ: =20
> http://www.quickfixengine.org/wikifix/index.php?QuickFixFAQ
> QuickFIX Support: http://www.quickfixengine.org/services.html
>
>
> I am not quite suire of the version I am using but this is the code =20=

> that was
> giving me trouble. I am not sure of the version becase I have =20
> customised the
> code heavily, so I have lately taken code from your updated versions =20=

> and big
> fixes and just added, instead of taking a whole new release, sorry =20
> about
> this ;)...
>
>
>
> The following was giving me trouble. I think the problem is that the
> 'extractLength' below should not be called before 'pos =3D =
m_buffer.find(
> "\00110=3D", pos-1 );', this is because you cannot check for the =
length =20
> unless
> the 'standard trailer' is part of the message. This is the problem I =20=

> was
> faced with where the only thing I would have in the buffer when trying =
=20
> to
> extract the length is '8=3DFI', and obviously at this point the =20
> extractLength
> would throw an exception. So, I have just modified it to check for the
> standard message trailer before attempting to extract length and have =20=

> not
> seen this problem.
>
> Here is the code below.
>
> bool Parser::readFixMessage( std::string& str )
> throw( MessageParseError&, RecvFailed& )
> { QF_STACK_PUSH(Parser::readFixMessage)
>
>   readFromStream();
>
>   std::string::size_type pos =3D 0;
>
>   if( m_buffer.length() < 2 )
> 	  return false;
>   pos =3D m_buffer.find( "8=3D" );
>   if( pos =3D=3D std::string::npos )
> 	  return false;
>
>
>   int pos_end_trailer_ =3D m_buffer.find( "\00110=3D");	\\	=
I HAVE ADDED
> THE FOLLOWING 3 LINES
>   if(pos_end_trailer_ =3D=3D std::string::npos)		\\ 	=
AND HAVE
> SEEMED TO SOLVED THE PRBLEM
> 	  return false;						\\
>
>   if(pos)
> 	  m_buffer.erase( 0, pos );
>
>   int length =3D 0;
>
>   try
>   {
>     if( extractLength(length, pos, m_buffer) )
>     {
>       pos +=3D length;
>       if( m_buffer.size() < pos )
>         return false;
>
>       pos =3D m_buffer.find( "\00110=3D", pos-1 );
>       if( pos =3D=3D std::string::npos ) return false;
>       pos +=3D 4;
>       pos =3D m_buffer.find( "\001", pos );
>       if( pos =3D=3D std::string::npos ) return false;
>       pos +=3D 1;
>
>       str =3D m_buffer.substr( 0, pos );
>       m_buffer.erase( 0, pos );
>       return true;
>     }
>   }
>   catch( MessageParseError& e )
>   {
>     m_buffer.erase( 0, pos + length );
>     throw e;
>   }
>
>   readFromStream();
>   return false;
>
>   QF_STACK_POP
> }
>
>
> I hope this helps.
>
> Regards,
>
> Emir
>
> -----Original Message-----
> From: Oren Miller [mailto:or...@qu...]
> Sent: Friday, December 10, 2004 10:04 PM
> To: em...@co...; qui...@li...
> Subject: Re: [Quickfix-users] Possible bug in the Parser when =
receiving
> huge amounts through th e socket
>
>
> QuickFIX Documentation:
> http://www.quickfixengine.org/quickfix/doc/html/index.html
> QuickFIX FAQ: =20
> http://www.quickfixengine.org/wikifix/index.php?QuickFixFAQ
> QuickFIX Support: http://www.quickfixengine.org/services.html
>
> I think I do need some more information about the version you are =20
> using.  I
> wrote a test and ran it against 1.9.4 and was not able to duplicate =
the
> behavior described.  The parser has changed over time so it may have =20=

> been
> resolved somewhere along the way.  What version are you using?
>
> --oren
>
> ----- Original Message -----
> From: <em...@co...>
> To: <qui...@li...>
> Sent: Thursday, December 09, 2004 10:14 AM
> Subject: [Quickfix-users] Possible bug in the Parser when receiving =20=

> huge
> amounts through th e socket
>
>
>> QuickFIX Documentation:
>> http://www.quickfixengine.org/quickfix/doc/html/index.html
>> QuickFIX FAQ: =20
>> http://www.quickfixengine.org/wikifix/index.php?QuickFixFAQ
>> QuickFIX Support: http://www.quickfixengine.org/services.html
>>
>> Here is the scenario.
>>
>> At the moment the socket buffer in the parser is 4096. So, I have =20
>> recently
>> generated a lot of traffic so I constantly receive of 4096 bytes on =20=

>> the
>> socket and have encountered the following problem.
>>
>> The end of the incoming buffer (m_buffer) of 4096 is as follows (some =
=20
>> of
>> the
>> values have been substituted to hide trading details):
>>
>>
> 8=3DFIX.4.2=019=3D291=0135=3D8=0134=3D156110=01369=3D31450=0152=3D200412=
09-15:35:=20
> 32.687=0149=3DBLA=0150=3D
>>
> G=0156=3DD33030N=0160=3D20041209-15:35:=20
> 32=0159=3D0=0155=3DGE=0154=3D1=0148=3DBLA000060467=01107=3DBLF5=011
>>
> =
67=3DFUT=0144=3D9740.00=0141=3D0=0140=3D2=0139=3D0=0138=3D10=0137=3D200412=
09004077=01151=3D10=01150=3D0=012=20
> 0=3D0=019
>>
> =
717=3D67960=0117=3D00687120041209093532=0114=3D0=0111=3D00067960=016=3D0=01=
432=3D20041209=011=3D1=20
> 23456
>> 78=0110=3D097=018=3DF
>>
>> As we can see, when the parser gets to parsing this part of the =20
>> buffer, it
>> will extract the length of the valid message which is 294 (9=3D291) =
and =20
>> in
>> turn send of the message for processing into the application code, =20=

>> etc.
>> After, it has processed the message the buffer then looks like:
>>
>> 8=3DF
>>
>> This is because the rest of the message is resting on the socket to =
be
>> read
>> so a valid FIX message can be composed.
>>
>> As you can see there is a 8=3DF at the end which was received into =
the
>> buffer,
>> and more data is left over on the socket to be read into the buffer. =20=

>> But,
>> at
>> the moment what the parser will do is make 8=3DF into a fix message =
as =20
>> this
>> is
>> where the buffer is ending, and will try and extract length from it, =20=

>> and
>> this generates a series of exceptions (FieldConvertError,
>> MessageParseError,
>> FieldNotFound and InvalidMessage).
>>
>>
>> Once the parser tries to parse this message, and throws its =20
>> exceptions, it
>> will get around to reading the socket again and receiving the =20
>> following
>> into
>> buffer:
>>
>>
> IX.4.2=019=3D349=0135=3D8=0134=3D156111=01369=3D31450=0152=3D20041209-15=
:35:=20
> 32.692=0149=3DBLA=0150=3DG=015
>>
> 6=3DD33030N=0160=3D20041209-15:35:=20
> 32=0159=3D0=0155=3DGE=0154=3D1=0148=3DCME000060467=01107=3DBLF5=01167=3D=

>>
> =
FUT=0144=3D9740.00=0141=3D0=0140=3D2=0139=3D2=0138=3D10=0137=3D20041209004=
077=01337=3D0C0=0132=3D10=0131=3D=20
> 9740.
>>
> =
00=01151=3D0=01150=3D2=0120=3D0=019717=3D67960=0117=3D00687220041209093532=
TN0002843=0175=3D2004=20
> 1209=01
>> 14=3D0=0111=3D00067960=01375=3DBLA030A1=016=3D0=01432=3D20041209=011=3D=
12345678=0110=3D178=01
>>
>> Once this is received the m_buffer is "8=3DF" and m_readBuffer is as =20=

>> above.
>> When the parser appends m_readBuffer to m_buffer the message looks =20=

>> like:
>>
>>
> 8=3DFIX.4.2=019=3D349=0135=3D8=0134=3D156111=01369=3D31450=0152=3D200412=
09-15:35:=20
> 32.692=0149=3DBLA=0150=3D
>>
> G=0156=3DD33030N=0160=3D20041209-15:35:=20
> 32=0159=3D0=0155=3DGE=0154=3D1=0148=3DCME000060467=01107=3DBLF5=011
>>
> =
67=3DFUT=0144=3D9740.00=0141=3D0=0140=3D2=0139=3D2=0138=3D10=0137=3D200412=
09004077=01337=3D0C0=0132=3D10=01=20
> 31=3D97
>>
> =
40.00=01151=3D0=01150=3D2=0120=3D0=019717=3D67960=0117=3D00687220041209093=
532TN0002843=0175=3D2=20
> 00412
>> 09=0114=3D0=0111=3D00067960=01375=3DBLA030A1=016=3D0=01432=3D20041209=01=
1=3D12345678=0110=3D178=01
>>
>> a COMPLETE message as it should be.
>>
>> Now surely, if this kind of scenario is encountered the parser should =
=20
>> not
>> even attempt to parse the message, as it is incomplete. I think that =20=

>> there
>> needs to be an extra safeguard in the parser that needs to check =20
>> presence
>> of
>> field 10 (Standard Trailer) in order to verify that it is a complete
>> message.
>>
>> If anyone has any ideas how to solve this, please be my guess as it =20=

>> has
>> taken my over 2 days just to track this problem down.
>>
>> Thanks a lot
>>
>> Emir
>>
>> ---
>> Outgoing mail is certified Virus Free.
>> Checked by AVG anti-virus system (http://www.grisoft.com).
>> Version: 6.0.766 / Virus Database: 513 - Release Date: 9/17/2004
>>
>>
>>
>> -------------------------------------------------------
>> SF email is sponsored by - The IT Product Guide
>> Read honest & candid reviews on hundreds of IT Products from real =20
>> users.
>> Discover which products truly live up to the hype. Start reading now.
>> http://productguide.itmanagersjournal.com/
>> _______________________________________________
>> Quickfix-users mailing list
>> Qui...@li...
>> https://lists.sourceforge.net/lists/listinfo/quickfix-users
>>
>
>
>
> -------------------------------------------------------
> SF email is sponsored by - The IT Product Guide
> Read honest & candid reviews on hundreds of IT Products from real =20
> users.
> Discover which products truly live up to the hype. Start reading now.
> http://productguide.itmanagersjournal.com/
> _______________________________________________
> Quickfix-users mailing list
> Qui...@li...
> https://lists.sourceforge.net/lists/listinfo/quickfix-users
>
> ---
> Incoming mail is certified Virus Free.
> Checked by AVG anti-virus system (http://www.grisoft.com).
> Version: 6.0.766 / Virus Database: 513 - Release Date: 9/17/2004
>
>
> ---
> Outgoing mail is certified Virus Free.
> Checked by AVG anti-virus system (http://www.grisoft.com).
> Version: 6.0.766 / Virus Database: 513 - Release Date: 9/17/2004
>
>
> ---
> Outgoing mail is certified Virus Free.
> Checked by AVG anti-virus system (http://www.grisoft.com).
> Version: 6.0.766 / Virus Database: 513 - Release Date: 9/17/2004
>
>
>
> -------------------------------------------------------
> SF email is sponsored by - The IT Product Guide
> Read honest & candid reviews on hundreds of IT Products from real =20
> users.
> Discover which products truly live up to the hype. Start reading now.
> http://productguide.itmanagersjournal.com/
> _______________________________________________
> Quickfix-users mailing list
> Qui...@li...
> https://lists.sourceforge.net/lists/listinfo/quickfix-users
>

RE: [Quickfix-users] Possible bug in the Parser when receiving hu ge amounts through th e socket

[<em...@co...>]

I am not quite suire of the version I am using but this is the code that was
giving me trouble. I am not sure of the version becase I have customised the
code heavily, so I have lately taken code from your updated versions and big
fixes and just added, instead of taking a whole new release, sorry about
this ;)...



The following was giving me trouble. I think the problem is that the
'extractLength' below should not be called before 'pos = m_buffer.find(
"\00110=", pos-1 );', this is because you cannot check for the length unless
the 'standard trailer' is part of the message. This is the problem I was
faced with where the only thing I would have in the buffer when trying to
extract the length is '8=FI', and obviously at this point the extractLength
would throw an exception. So, I have just modified it to check for the
standard message trailer before attempting to extract length and have not
seen this problem.

Here is the code below.

bool Parser::readFixMessage( std::string& str )
throw( MessageParseError&, RecvFailed& )
{ QF_STACK_PUSH(Parser::readFixMessage)

  readFromStream();

  std::string::size_type pos = 0;

  if( m_buffer.length() < 2 ) 
	  return false;
  pos = m_buffer.find( "8=" );
  if( pos == std::string::npos ) 
	  return false;


  int pos_end_trailer_ = m_buffer.find( "\00110=");	\\	I HAVE ADDED
THE FOLLOWING 3 LINES
  if(pos_end_trailer_ == std::string::npos)		\\ 	AND HAVE
SEEMED TO SOLVED THE PRBLEM
	  return false;						\\

  if(pos)
	  m_buffer.erase( 0, pos );

  int length = 0;

  try
  {
    if( extractLength(length, pos, m_buffer) )
    {
      pos += length;
      if( m_buffer.size() < pos )
        return false;

      pos = m_buffer.find( "\00110=", pos-1 );
      if( pos == std::string::npos ) return false;
      pos += 4;
      pos = m_buffer.find( "\001", pos );
      if( pos == std::string::npos ) return false;
      pos += 1;

      str = m_buffer.substr( 0, pos );
      m_buffer.erase( 0, pos );
      return true;
    }
  }
  catch( MessageParseError& e )
  {
    m_buffer.erase( 0, pos + length );
    throw e;
  }

  readFromStream();
  return false;

  QF_STACK_POP
}


I hope this helps.

Regards,

Emir

-----Original Message-----
From: Oren Miller [mailto:or...@qu...]
Sent: Friday, December 10, 2004 10:04 PM
To: em...@co...; qui...@li...
Subject: Re: [Quickfix-users] Possible bug in the Parser when receiving
huge amounts through th e socket


QuickFIX Documentation:
http://www.quickfixengine.org/quickfix/doc/html/index.html
QuickFIX FAQ: http://www.quickfixengine.org/wikifix/index.php?QuickFixFAQ
QuickFIX Support: http://www.quickfixengine.org/services.html

I think I do need some more information about the version you are using.  I 
wrote a test and ran it against 1.9.4 and was not able to duplicate the 
behavior described.  The parser has changed over time so it may have been 
resolved somewhere along the way.  What version are you using?

--oren

----- Original Message ----- 
From: <em...@co...>
To: <qui...@li...>
Sent: Thursday, December 09, 2004 10:14 AM
Subject: [Quickfix-users] Possible bug in the Parser when receiving huge 
amounts through th e socket


> QuickFIX Documentation: 
> http://www.quickfixengine.org/quickfix/doc/html/index.html
> QuickFIX FAQ: http://www.quickfixengine.org/wikifix/index.php?QuickFixFAQ
> QuickFIX Support: http://www.quickfixengine.org/services.html
>
> Here is the scenario.
>
> At the moment the socket buffer in the parser is 4096. So, I have recently
> generated a lot of traffic so I constantly receive of 4096 bytes on the
> socket and have encountered the following problem.
>
> The end of the incoming buffer (m_buffer) of 4096 is as follows (some of 
> the
> values have been substituted to hide trading details):
>
>
8=FIX.4.2
9=291
35=8
34=156110
369=31450
52=20041209-15:35:32.687
49=BLA
50=
>
G
56=D33030N
60=20041209-15:35:32
59=0
55=GE
54=1
48=BLA000060467
107=BLF5
1
>
67=FUT
44=9740.00
41=0
40=2
39=0
38=10
37=20041209004077
151=10
150=0
20=0
9
>
717=67960
17=00687120041209093532
14=0
11=00067960
6=0
432=20041209
1=123456
> 78
10=097
8=F
>
> As we can see, when the parser gets to parsing this part of the buffer, it
> will extract the length of the valid message which is 294 (9=291) and in
> turn send of the message for processing into the application code, etc.
> After, it has processed the message the buffer then looks like:
>
> 8=F
>
> This is because the rest of the message is resting on the socket to be 
> read
> so a valid FIX message can be composed.
>
> As you can see there is a 8=F at the end which was received into the 
> buffer,
> and more data is left over on the socket to be read into the buffer. But, 
> at
> the moment what the parser will do is make 8=F into a fix message as this 
> is
> where the buffer is ending, and will try and extract length from it, and
> this generates a series of exceptions (FieldConvertError, 
> MessageParseError,
> FieldNotFound and InvalidMessage).
>
>
> Once the parser tries to parse this message, and throws its exceptions, it
> will get around to reading the socket again and receiving the following 
> into
> buffer:
>
>
IX.4.2
9=349
35=8
34=156111
369=31450
52=20041209-15:35:32.692
49=BLA
50=G
5
>
6=D33030N
60=20041209-15:35:32
59=0
55=GE
54=1
48=CME000060467
107=BLF5
167=
>
FUT
44=9740.00
41=0
40=2
39=2
38=10
37=20041209004077
337=0C0
32=10
31=9740.
>
00
151=0
150=2
20=0
9717=67960
17=00687220041209093532TN0002843
75=20041209

> 14=0
11=00067960
375=BLA030A1
6=0
432=20041209
1=12345678
10=178

>
> Once this is received the m_buffer is "8=F" and m_readBuffer is as above.
> When the parser appends m_readBuffer to m_buffer the message looks like:
>
>
8=FIX.4.2
9=349
35=8
34=156111
369=31450
52=20041209-15:35:32.692
49=BLA
50=
>
G
56=D33030N
60=20041209-15:35:32
59=0
55=GE
54=1
48=CME000060467
107=BLF5
1
>
67=FUT
44=9740.00
41=0
40=2
39=2
38=10
37=20041209004077
337=0C0
32=10
31=97
>
40.00
151=0
150=2
20=0
9717=67960
17=00687220041209093532TN0002843
75=200412
> 09
14=0
11=00067960
375=BLA030A1
6=0
432=20041209
1=12345678
10=178

>
> a COMPLETE message as it should be.
>
> Now surely, if this kind of scenario is encountered the parser should not
> even attempt to parse the message, as it is incomplete. I think that there
> needs to be an extra safeguard in the parser that needs to check presence 
> of
> field 10 (Standard Trailer) in order to verify that it is a complete
> message.
>
> If anyone has any ideas how to solve this, please be my guess as it has
> taken my over 2 days just to track this problem down.
>
> Thanks a lot
>
> Emir
>
> ---
> Outgoing mail is certified Virus Free.
> Checked by AVG anti-virus system (http://www.grisoft.com).
> Version: 6.0.766 / Virus Database: 513 - Release Date: 9/17/2004
>
>
>
> -------------------------------------------------------
> SF email is sponsored by - The IT Product Guide
> Read honest & candid reviews on hundreds of IT Products from real users.
> Discover which products truly live up to the hype. Start reading now.
> http://productguide.itmanagersjournal.com/
> _______________________________________________
> Quickfix-users mailing list
> Qui...@li...
> https://lists.sourceforge.net/lists/listinfo/quickfix-users
> 



-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now. 
http://productguide.itmanagersjournal.com/
_______________________________________________
Quickfix-users mailing list
Qui...@li...
https://lists.sourceforge.net/lists/listinfo/quickfix-users

---
Incoming mail is certified Virus Free.
Checked by AVG anti-virus system (http://www.grisoft.com).
Version: 6.0.766 / Virus Database: 513 - Release Date: 9/17/2004
 

---
Outgoing mail is certified Virus Free.
Checked by AVG anti-virus system (http://www.grisoft.com).
Version: 6.0.766 / Virus Database: 513 - Release Date: 9/17/2004
 

---
Outgoing mail is certified Virus Free.
Checked by AVG anti-virus system (http://www.grisoft.com).
Version: 6.0.766 / Virus Database: 513 - Release Date: 9/17/2004
 

Re: [Quickfix-users] Possible bug in the Parser when receiving huge amounts through th e socket

[Oren M. <or...@qu...>]

I think I do need some more information about the version you are using.  I 
wrote a test and ran it against 1.9.4 and was not able to duplicate the 
behavior described.  The parser has changed over time so it may have been 
resolved somewhere along the way.  What version are you using?

--oren

----- Original Message ----- 
From: <em...@co...>
To: <qui...@li...>
Sent: Thursday, December 09, 2004 10:14 AM
Subject: [Quickfix-users] Possible bug in the Parser when receiving huge 
amounts through th e socket


> QuickFIX Documentation: 
> http://www.quickfixengine.org/quickfix/doc/html/index.html
> QuickFIX FAQ: http://www.quickfixengine.org/wikifix/index.php?QuickFixFAQ
> QuickFIX Support: http://www.quickfixengine.org/services.html
>
> Here is the scenario.
>
> At the moment the socket buffer in the parser is 4096. So, I have recently
> generated a lot of traffic so I constantly receive of 4096 bytes on the
> socket and have encountered the following problem.
>
> The end of the incoming buffer (m_buffer) of 4096 is as follows (some of 
> the
> values have been substituted to hide trading details):
>
> 8=FIX.4.2
9=291
35=8
34=156110
369=31450
52=20041209-15:35:32.687
49=BLA
50=
> G
56=D33030N
60=20041209-15:35:32
59=0
55=GE
54=1
48=BLA000060467
107=BLF5
1
> 67=FUT
44=9740.00
41=0
40=2
39=0
38=10
37=20041209004077
151=10
150=0
20=0
9
> 717=67960
17=00687120041209093532
14=0
11=00067960
6=0
432=20041209
1=123456
> 78
10=097
8=F
>
> As we can see, when the parser gets to parsing this part of the buffer, it
> will extract the length of the valid message which is 294 (9=291) and in
> turn send of the message for processing into the application code, etc.
> After, it has processed the message the buffer then looks like:
>
> 8=F
>
> This is because the rest of the message is resting on the socket to be 
> read
> so a valid FIX message can be composed.
>
> As you can see there is a 8=F at the end which was received into the 
> buffer,
> and more data is left over on the socket to be read into the buffer. But, 
> at
> the moment what the parser will do is make 8=F into a fix message as this 
> is
> where the buffer is ending, and will try and extract length from it, and
> this generates a series of exceptions (FieldConvertError, 
> MessageParseError,
> FieldNotFound and InvalidMessage).
>
>
> Once the parser tries to parse this message, and throws its exceptions, it
> will get around to reading the socket again and receiving the following 
> into
> buffer:
>
> IX.4.2
9=349
35=8
34=156111
369=31450
52=20041209-15:35:32.692
49=BLA
50=G
5
> 6=D33030N
60=20041209-15:35:32
59=0
55=GE
54=1
48=CME000060467
107=BLF5
167=
> FUT
44=9740.00
41=0
40=2
39=2
38=10
37=20041209004077
337=0C0
32=10
31=9740.
> 00
151=0
150=2
20=0
9717=67960
17=00687220041209093532TN0002843
75=20041209

> 14=0
11=00067960
375=BLA030A1
6=0
432=20041209
1=12345678
10=178

>
> Once this is received the m_buffer is "8=F" and m_readBuffer is as above.
> When the parser appends m_readBuffer to m_buffer the message looks like:
>
> 8=FIX.4.2
9=349
35=8
34=156111
369=31450
52=20041209-15:35:32.692
49=BLA
50=
> G
56=D33030N
60=20041209-15:35:32
59=0
55=GE
54=1
48=CME000060467
107=BLF5
1
> 67=FUT
44=9740.00
41=0
40=2
39=2
38=10
37=20041209004077
337=0C0
32=10
31=97
> 40.00
151=0
150=2
20=0
9717=67960
17=00687220041209093532TN0002843
75=200412
> 09
14=0
11=00067960
375=BLA030A1
6=0
432=20041209
1=12345678
10=178

>
> a COMPLETE message as it should be.
>
> Now surely, if this kind of scenario is encountered the parser should not
> even attempt to parse the message, as it is incomplete. I think that there
> needs to be an extra safeguard in the parser that needs to check presence 
> of
> field 10 (Standard Trailer) in order to verify that it is a complete
> message.
>
> If anyone has any ideas how to solve this, please be my guess as it has
> taken my over 2 days just to track this problem down.
>
> Thanks a lot
>
> Emir
>
> ---
> Outgoing mail is certified Virus Free.
> Checked by AVG anti-virus system (http://www.grisoft.com).
> Version: 6.0.766 / Virus Database: 513 - Release Date: 9/17/2004
>
>
>
> -------------------------------------------------------
> SF email is sponsored by - The IT Product Guide
> Read honest & candid reviews on hundreds of IT Products from real users.
> Discover which products truly live up to the hype. Start reading now.
> http://productguide.itmanagersjournal.com/
> _______________________________________________
> Quickfix-users mailing list
> Qui...@li...
> https://lists.sourceforge.net/lists/listinfo/quickfix-users
> 

Re: [Quickfix-users] Possible bug in the Parser when receiving huge amounts through th e socket

[Oren M. <or...@qu...>]

We've added a report into the bugtracker.  We'll confirm this and come up 
with a resolution.

--oren

----- Original Message ----- 
From: <em...@co...>
To: <qui...@li...>
Sent: Thursday, December 09, 2004 10:14 AM
Subject: [Quickfix-users] Possible bug in the Parser when receiving huge 
amounts through th e socket


> QuickFIX Documentation: 
> http://www.quickfixengine.org/quickfix/doc/html/index.html
> QuickFIX FAQ: http://www.quickfixengine.org/wikifix/index.php?QuickFixFAQ
> QuickFIX Support: http://www.quickfixengine.org/services.html
>
> Here is the scenario.
>
> At the moment the socket buffer in the parser is 4096. So, I have recently
> generated a lot of traffic so I constantly receive of 4096 bytes on the
> socket and have encountered the following problem.
>
> The end of the incoming buffer (m_buffer) of 4096 is as follows (some of 
> the
> values have been substituted to hide trading details):
>
> 8=FIX.4.2
9=291
35=8
34=156110
369=31450
52=20041209-15:35:32.687
49=BLA
50=
> G
56=D33030N
60=20041209-15:35:32
59=0
55=GE
54=1
48=BLA000060467
107=BLF5
1
> 67=FUT
44=9740.00
41=0
40=2
39=0
38=10
37=20041209004077
151=10
150=0
20=0
9
> 717=67960
17=00687120041209093532
14=0
11=00067960
6=0
432=20041209
1=123456
> 78
10=097
8=F
>
> As we can see, when the parser gets to parsing this part of the buffer, it
> will extract the length of the valid message which is 294 (9=291) and in
> turn send of the message for processing into the application code, etc.
> After, it has processed the message the buffer then looks like:
>
> 8=F
>
> This is because the rest of the message is resting on the socket to be 
> read
> so a valid FIX message can be composed.
>
> As you can see there is a 8=F at the end which was received into the 
> buffer,
> and more data is left over on the socket to be read into the buffer. But, 
> at
> the moment what the parser will do is make 8=F into a fix message as this 
> is
> where the buffer is ending, and will try and extract length from it, and
> this generates a series of exceptions (FieldConvertError, 
> MessageParseError,
> FieldNotFound and InvalidMessage).
>
>
> Once the parser tries to parse this message, and throws its exceptions, it
> will get around to reading the socket again and receiving the following 
> into
> buffer:
>
> IX.4.2
9=349
35=8
34=156111
369=31450
52=20041209-15:35:32.692
49=BLA
50=G
5
> 6=D33030N
60=20041209-15:35:32
59=0
55=GE
54=1
48=CME000060467
107=BLF5
167=
> FUT
44=9740.00
41=0
40=2
39=2
38=10
37=20041209004077
337=0C0
32=10
31=9740.
> 00
151=0
150=2
20=0
9717=67960
17=00687220041209093532TN0002843
75=20041209

> 14=0
11=00067960
375=BLA030A1
6=0
432=20041209
1=12345678
10=178

>
> Once this is received the m_buffer is "8=F" and m_readBuffer is as above.
> When the parser appends m_readBuffer to m_buffer the message looks like:
>
> 8=FIX.4.2
9=349
35=8
34=156111
369=31450
52=20041209-15:35:32.692
49=BLA
50=
> G
56=D33030N
60=20041209-15:35:32
59=0
55=GE
54=1
48=CME000060467
107=BLF5
1
> 67=FUT
44=9740.00
41=0
40=2
39=2
38=10
37=20041209004077
337=0C0
32=10
31=97
> 40.00
151=0
150=2
20=0
9717=67960
17=00687220041209093532TN0002843
75=200412
> 09
14=0
11=00067960
375=BLA030A1
6=0
432=20041209
1=12345678
10=178

>
> a COMPLETE message as it should be.
>
> Now surely, if this kind of scenario is encountered the parser should not
> even attempt to parse the message, as it is incomplete. I think that there
> needs to be an extra safeguard in the parser that needs to check presence 
> of
> field 10 (Standard Trailer) in order to verify that it is a complete
> message.
>
> If anyone has any ideas how to solve this, please be my guess as it has
> taken my over 2 days just to track this problem down.
>
> Thanks a lot
>
> Emir
>
> ---
> Outgoing mail is certified Virus Free.
> Checked by AVG anti-virus system (http://www.grisoft.com).
> Version: 6.0.766 / Virus Database: 513 - Release Date: 9/17/2004
>
>
>
> -------------------------------------------------------
> SF email is sponsored by - The IT Product Guide
> Read honest & candid reviews on hundreds of IT Products from real users.
> Discover which products truly live up to the hype. Start reading now.
> http://productguide.itmanagersjournal.com/
> _______________________________________________
> Quickfix-users mailing list
> Qui...@li...
> https://lists.sourceforge.net/lists/listinfo/quickfix-users
> 

Re: [Quickfix-users] Possible bug in the Parser when receiving hu ge amounts through th e socket

[Kaczmarek, T. <tka...@Cr...>]

On Thu, 2004-12-09 at 16:14 +0000, em...@co... wrote:
> QuickFIX Documentation:
http://www.quickfixengine.org/quickfix/doc/html/index.html
> QuickFIX FAQ: http://www.quickfixengine.org/wikifix/index.php?QuickFixFAQ
> QuickFIX Support: http://www.quickfixengine.org/services.html
> 
> Here is the scenario. 
> 
> At the moment the socket buffer in the parser is 4096. So, I have recently
> generated a lot of traffic so I constantly receive of 4096 bytes on the
> socket and have encountered the following problem.
> 
> The end of the incoming buffer (m_buffer) of 4096 is as follows (some of
the
> values have been substituted to hide trading details):
> 
>
8=FIX.4.2
9=291
35=8
34=156110
369=31450
52=20041209-15:35:32.687
49=BLA
50=
>
G
56=D33030N
60=20041209-15:35:32
59=0
55=GE
54=1
48=BLA000060467
107=BLF5
1
>
67=FUT
44=9740.00
41=0
40=2
39=0
38=10
37=20041209004077
151=10
150=0
20=0
9
>
717=67960
17=00687120041209093532
14=0
11=00067960
6=0
432=20041209
1=123456
> 78
10=097
8=F
> 
> As we can see, when the parser gets to parsing this part of the buffer, it
> will extract the length of the valid message which is 294 (9=291) and in
> turn send of the message for processing into the application code, etc.
> After, it has processed the message the buffer then looks like:
> 
> 8=F
> 
> This is because the rest of the message is resting on the socket to be
read
> so a valid FIX message can be composed.
> 
> As you can see there is a 8=F at the end which was received into the
buffer,
> and more data is left over on the socket to be read into the buffer. But,
at
> the moment what the parser will do is make 8=F into a fix message as this
is
> where the buffer is ending, and will try and extract length from it, and
> this generates a series of exceptions (FieldConvertError,
MessageParseError,
> FieldNotFound and InvalidMessage).
> 
> 
> Once the parser tries to parse this message, and throws its exceptions, it
> will get around to reading the socket again and receiving the following
into
> buffer:
> 
>
IX.4.2
9=349
35=8
34=156111
369=31450
52=20041209-15:35:32.692
49=BLA
50=G
5
>
6=D33030N
60=20041209-15:35:32
59=0
55=GE
54=1
48=CME000060467
107=BLF5
167=
>
FUT
44=9740.00
41=0
40=2
39=2
38=10
37=20041209004077
337=0C0
32=10
31=9740.
>
00
151=0
150=2
20=0
9717=67960
17=00687220041209093532TN0002843
75=20041209

> 14=0
11=00067960
375=BLA030A1
6=0
432=20041209
1=12345678
10=178

> 
> Once this is received the m_buffer is "8=F" and m_readBuffer is as above.
> When the parser appends m_readBuffer to m_buffer the message looks like:
> 
>
8=FIX.4.2
9=349
35=8
34=156111
369=31450
52=20041209-15:35:32.692
49=BLA
50=
>
G
56=D33030N
60=20041209-15:35:32
59=0
55=GE
54=1
48=CME000060467
107=BLF5
1
>
67=FUT
44=9740.00
41=0
40=2
39=2
38=10
37=20041209004077
337=0C0
32=10
31=97
>
40.00
151=0
150=2
20=0
9717=67960
17=00687220041209093532TN0002843
75=200412
> 09
14=0
11=00067960
375=BLA030A1
6=0
432=20041209
1=12345678
10=178

> 
> a COMPLETE message as it should be.
> 
> Now surely, if this kind of scenario is encountered the parser should not
> even attempt to parse the message, as it is incomplete. I think that there
> needs to be an extra safeguard in the parser that needs to check presence
of
> field 10 (Standard Trailer) in order to verify that it is a complete
> message.
> 
> If anyone has any ideas how to solve this, please be my guess as it has
> taken my over 2 days just to track this problem down.
> 
> Thanks a lot
> 
> Emir
> 
> ---
Would it be too much too ask what version you are running and what you
are running it on?

Ted
DISCLAIMER

This e-mail, and any attachments thereto, is intended only for use by the
addressee(s) named herein and may contain legally privileged and/or
confidential information. If you are not the intended recipient of this
e-mail, you are hereby notified that any dissemination, distribution or
copying of this e-mail, and any attachments thereto, is strictly prohibited.
If you have received this e-mail in error, please immediately notify me and
permanently delete the original and any copy of any e-mail and any printout
thereof.

E-mail transmission cannot be guaranteed to be secure or error-free. The
sender therefore does not accept liability for any errors or omissions in
the contents of this message which arise as a result of e-mail transmission.
REGARDING PRIVACY AND CONFIDENTIALITY
Crown Financial Group may, at its discretion, monitor and review the content
of all e-mail communications.

[Quickfix-users] Possible bug in the Parser when receiving huge amounts through th e socket

[<em...@co...>]

Here is the scenario. 

At the moment the socket buffer in the parser is 4096. So, I have recently
generated a lot of traffic so I constantly receive of 4096 bytes on the
socket and have encountered the following problem.

The end of the incoming buffer (m_buffer) of 4096 is as follows (some of the
values have been substituted to hide trading details):

8=FIX.4.2
9=291
35=8
34=156110
369=31450
52=20041209-15:35:32.687
49=BLA
50=
G
56=D33030N
60=20041209-15:35:32
59=0
55=GE
54=1
48=BLA000060467
107=BLF5
1
67=FUT
44=9740.00
41=0
40=2
39=0
38=10
37=20041209004077
151=10
150=0
20=0
9
717=67960
17=00687120041209093532
14=0
11=00067960
6=0
432=20041209
1=123456
78
10=097
8=F

As we can see, when the parser gets to parsing this part of the buffer, it
will extract the length of the valid message which is 294 (9=291) and in
turn send of the message for processing into the application code, etc.
After, it has processed the message the buffer then looks like:

8=F

This is because the rest of the message is resting on the socket to be read
so a valid FIX message can be composed.

As you can see there is a 8=F at the end which was received into the buffer,
and more data is left over on the socket to be read into the buffer. But, at
the moment what the parser will do is make 8=F into a fix message as this is
where the buffer is ending, and will try and extract length from it, and
this generates a series of exceptions (FieldConvertError, MessageParseError,
FieldNotFound and InvalidMessage).


Once the parser tries to parse this message, and throws its exceptions, it
will get around to reading the socket again and receiving the following into
buffer:

IX.4.2
9=349
35=8
34=156111
369=31450
52=20041209-15:35:32.692
49=BLA
50=G
5
6=D33030N
60=20041209-15:35:32
59=0
55=GE
54=1
48=CME000060467
107=BLF5
167=
FUT
44=9740.00
41=0
40=2
39=2
38=10
37=20041209004077
337=0C0
32=10
31=9740.
00
151=0
150=2
20=0
9717=67960
17=00687220041209093532TN0002843
75=20041209

14=0
11=00067960
375=BLA030A1
6=0
432=20041209
1=12345678
10=178


Once this is received the m_buffer is "8=F" and m_readBuffer is as above.
When the parser appends m_readBuffer to m_buffer the message looks like:

8=FIX.4.2
9=349
35=8
34=156111
369=31450
52=20041209-15:35:32.692
49=BLA
50=
G
56=D33030N
60=20041209-15:35:32
59=0
55=GE
54=1
48=CME000060467
107=BLF5
1
67=FUT
44=9740.00
41=0
40=2
39=2
38=10
37=20041209004077
337=0C0
32=10
31=97
40.00
151=0
150=2
20=0
9717=67960
17=00687220041209093532TN0002843
75=200412
09
14=0
11=00067960
375=BLA030A1
6=0
432=20041209
1=12345678
10=178


a COMPLETE message as it should be.

Now surely, if this kind of scenario is encountered the parser should not
even attempt to parse the message, as it is incomplete. I think that there
needs to be an extra safeguard in the parser that needs to check presence of
field 10 (Standard Trailer) in order to verify that it is a complete
message.

If anyone has any ideas how to solve this, please be my guess as it has
taken my over 2 days just to track this problem down.

Thanks a lot

Emir

---
Outgoing mail is certified Virus Free.
Checked by AVG anti-virus system (http://www.grisoft.com).
Version: 6.0.766 / Virus Database: 513 - Release Date: 9/17/2004