RE: [Quickfix-users] Possible bug in the Parser when receiving hu ge amounts through th e socket

[<em...@co...>]

This is my post from earlier. I guess you didnt see it Oren. Here it goes
anyway....




Oren,

AM_INIT_AUTOMAKE(quickfix, 1.5.0), but have also take some code from
AM_INIT_AUTOMAKE(quickfix, 1.9.2) so I can have an upto date version of the
engine. But,  most of it is 1.5.0 with changes made to a few files in order
to have an upto date track of the bug fixes.

Also, my test platform is Windows Server 2000 SP4 (quad 550Mhz processor
machine, 1Gb memory), and have actually also made some changes to how data
is received on the socket now, and how the socket is created. The socket is
now created as an overlapped socket explicitly through
::WSASocket( PF_INET, SOCK_STREAM, IPPROTO_TCP, NULL, 0, WSA_FLAG_OVERLAPPED
);
So, this creates an overlapped socket and allows me to perform asynchronous
calls on the socket. 

I have started of a new thread which creates a completion port, and calls
'WSARecv' and then calls 'GetQueuedCompletionStatus' depending on the value
of the return of 'WSARecv'. The reason I have gone for this approach is that
when calling 'recv' on the socket and there is 5,000 bytes on the socket,
and you give it a sufficiently large buffer (lets says 10,000 bytes), it is
not guaranteed that you are going to receive all 5,000 bytes in that single
receive. This is explained in the MSDN when calling 'ioctlsocket', or in the
code 'socket_fionread( m_socket, bytes )'. The below is copied direct from
MSDN:
----------------------------------------------------------------------------
-
FIONREAD 
Use to determine the amount of data pending in the network's input buffer
that can be read from socket s. The argp parameter points to an unsigned
long value in which ioctlsocket stores the result. FIONREAD returns the
amount of data that can be read in a single call to the recv function, which
may not be the same as the total amount of data queued on the socket. If s
is message oriented (for example, type SOCK_DGRAM), FIONREAD still returns
the amount of pending data in the network buffer, however, the amount that
can actually be read in a single call to the recv function is limited to the
data size written in the send or sendto function call. 
----------------------------------------------------------------------------
-

So, we can see that 'ioctlsocket' will let us know how much we can actually
read from the socket, even though there may be more data pending on the
socket. So, I came to a scenario where I would have read a buffer of lets
says 15-20 messages, but the end of the buffer would have looked like
'8=FI', and on the next 'recv' call I would have got 'X.4.2' ....etc.

I think that it is important to check for the existence of the standard
trailer in the buffer before trying to extract the length, as it seems at
least on the Microsoft platform you are not guaranteed to receive all the
data from socket even if you give it a large enough buffer. I have only been
able to use a Microsoft platform as this is what we work on, and would be
very interested to know on what platform you carried out your test.

I do not think that this a problem specifically with any version, because I
check the very latest version of the parser on the internet
(http://cvs.sourceforge.net/viewcvs.py/quickfix/quickfix/src/C%2B%2B/Parser.
cpp?rev=1.11&view=markup) and 


bool Parser::readFixMessage( std::string& str )
throw( MessageParseError, RecvFailed )
{ QF_STACK_PUSH(Parser::readFixMessage)

  readFromStream();

  std::string::size_type pos = 0;
  if( m_buffer.length() < 2 ) return false;
  pos = m_buffer.find( "8=" );
  if( pos == std::string::npos ) return false;
  m_buffer.erase( 0, pos );

	/*
	*	NEED TO CHECK FOR STANDARD TRAILER HERE TO COMPENSATE FOR
THE SCENARIO DESCRIBED ABOVE
	*/

  int length = 0;

  try
  {
    if( extractLength(length, pos, m_buffer) )
    {
      pos += length;
      if( m_buffer.size() < pos )
        return false;
-----------------------------------------------------


That is the change I had made and have not seen the problem again.

I hope this helps in some way, and once again I say that I have only seen
this when creating huge amounts of traffic (ie. filling an order for 100
lots and getting back 100 partial fills, etc), and on Microsoft platform.


Regards,


Emir Kamber

-----Original Message-----
From: Oren Miller [mailto:or...@qu...]
Sent: Tuesday, December 14, 2004 2:57 PM
To: em...@co...
Cc: qui...@li...
Subject: Re: [Quickfix-users] Possible bug in the Parser when receiving
hu ge amounts through th e socket


QuickFIX Documentation:
http://www.quickfixengine.org/quickfix/doc/html/index.html
QuickFIX FAQ: http://www.quickfixengine.org/wikifix/index.php?QuickFixFAQ
QuickFIX Support: http://www.quickfixengine.org/services.html

Emir,

Can you tell me the version that is listed in your configure.in at the  
top, should be in this format AM_INIT_AUTOMAKE(quickfix,  
major.minor.revision)

Here we can at least tell the release yours is based on.  My goal is to  
write a test for that version which duplicates the issue in a  
repeatable manner, and then run the test against the latest version to  
see if the problem still exists.  If not I will apply your patch and  
see if that does the trick.  Thanks.

--oren

On Dec 13, 2004, at 5:16 AM, em...@co... wrote:

> QuickFIX Documentation:  
> http://www.quickfixengine.org/quickfix/doc/html/index.html
> QuickFIX FAQ:  
> http://www.quickfixengine.org/wikifix/index.php?QuickFixFAQ
> QuickFIX Support: http://www.quickfixengine.org/services.html
>
>
> I am not quite suire of the version I am using but this is the code  
> that was
> giving me trouble. I am not sure of the version becase I have  
> customised the
> code heavily, so I have lately taken code from your updated versions  
> and big
> fixes and just added, instead of taking a whole new release, sorry  
> about
> this ;)...
>
>
>
> The following was giving me trouble. I think the problem is that the
> 'extractLength' below should not be called before 'pos = m_buffer.find(
> "\00110=", pos-1 );', this is because you cannot check for the length  
> unless
> the 'standard trailer' is part of the message. This is the problem I  
> was
> faced with where the only thing I would have in the buffer when trying  
> to
> extract the length is '8=FI', and obviously at this point the  
> extractLength
> would throw an exception. So, I have just modified it to check for the
> standard message trailer before attempting to extract length and have  
> not
> seen this problem.
>
> Here is the code below.
>
> bool Parser::readFixMessage( std::string& str )
> throw( MessageParseError&, RecvFailed& )
> { QF_STACK_PUSH(Parser::readFixMessage)
>
>   readFromStream();
>
>   std::string::size_type pos = 0;
>
>   if( m_buffer.length() < 2 )
> 	  return false;
>   pos = m_buffer.find( "8=" );
>   if( pos == std::string::npos )
> 	  return false;
>
>
>   int pos_end_trailer_ = m_buffer.find( "\00110=");	\\	I HAVE ADDED
> THE FOLLOWING 3 LINES
>   if(pos_end_trailer_ == std::string::npos)		\\ 	AND HAVE
> SEEMED TO SOLVED THE PRBLEM
> 	  return false;						\\
>
>   if(pos)
> 	  m_buffer.erase( 0, pos );
>
>   int length = 0;
>
>   try
>   {
>     if( extractLength(length, pos, m_buffer) )
>     {
>       pos += length;
>       if( m_buffer.size() < pos )
>         return false;
>
>       pos = m_buffer.find( "\00110=", pos-1 );
>       if( pos == std::string::npos ) return false;
>       pos += 4;
>       pos = m_buffer.find( "\001", pos );
>       if( pos == std::string::npos ) return false;
>       pos += 1;
>
>       str = m_buffer.substr( 0, pos );
>       m_buffer.erase( 0, pos );
>       return true;
>     }
>   }
>   catch( MessageParseError& e )
>   {
>     m_buffer.erase( 0, pos + length );
>     throw e;
>   }
>
>   readFromStream();
>   return false;
>
>   QF_STACK_POP
> }
>
>
> I hope this helps.
>
> Regards,
>
> Emir
>
> -----Original Message-----
> From: Oren Miller [mailto:or...@qu...]
> Sent: Friday, December 10, 2004 10:04 PM
> To: em...@co...; qui...@li...
> Subject: Re: [Quickfix-users] Possible bug in the Parser when receiving
> huge amounts through th e socket
>
>
> QuickFIX Documentation:
> http://www.quickfixengine.org/quickfix/doc/html/index.html
> QuickFIX FAQ:  
> http://www.quickfixengine.org/wikifix/index.php?QuickFixFAQ
> QuickFIX Support: http://www.quickfixengine.org/services.html
>
> I think I do need some more information about the version you are  
> using.  I
> wrote a test and ran it against 1.9.4 and was not able to duplicate the
> behavior described.  The parser has changed over time so it may have  
> been
> resolved somewhere along the way.  What version are you using?
>
> --oren
>
> ----- Original Message -----
> From: <em...@co...>
> To: <qui...@li...>
> Sent: Thursday, December 09, 2004 10:14 AM
> Subject: [Quickfix-users] Possible bug in the Parser when receiving  
> huge
> amounts through th e socket
>
>
>> QuickFIX Documentation:
>> http://www.quickfixengine.org/quickfix/doc/html/index.html
>> QuickFIX FAQ:  
>> http://www.quickfixengine.org/wikifix/index.php?QuickFixFAQ
>> QuickFIX Support: http://www.quickfixengine.org/services.html
>>
>> Here is the scenario.
>>
>> At the moment the socket buffer in the parser is 4096. So, I have  
>> recently
>> generated a lot of traffic so I constantly receive of 4096 bytes on  
>> the
>> socket and have encountered the following problem.
>>
>> The end of the incoming buffer (m_buffer) of 4096 is as follows (some  
>> of
>> the
>> values have been substituted to hide trading details):
>>
>>
> 8=FIX.4.2
9=291
35=8
34=156110
369=31450
52=20041209-15:35: 
> 32.687
49=BLA
50=
>>
> G
56=D33030N
60=20041209-15:35: 
> 32
59=0
55=GE
54=1
48=BLA000060467
107=BLF5
1
>>
> 67=FUT
44=9740.00
41=0
40=2
39=0
38=10
37=20041209004077
151=10
150=0
2 
> 0=0
9
>>
> 717=67960
17=00687120041209093532
14=0
11=00067960
6=0
432=20041209
1=1 
> 23456
>> 78
10=097
8=F
>>
>> As we can see, when the parser gets to parsing this part of the  
>> buffer, it
>> will extract the length of the valid message which is 294 (9=291) and  
>> in
>> turn send of the message for processing into the application code,  
>> etc.
>> After, it has processed the message the buffer then looks like:
>>
>> 8=F
>>
>> This is because the rest of the message is resting on the socket to be
>> read
>> so a valid FIX message can be composed.
>>
>> As you can see there is a 8=F at the end which was received into the
>> buffer,
>> and more data is left over on the socket to be read into the buffer.  
>> But,
>> at
>> the moment what the parser will do is make 8=F into a fix message as  
>> this
>> is
>> where the buffer is ending, and will try and extract length from it,  
>> and
>> this generates a series of exceptions (FieldConvertError,
>> MessageParseError,
>> FieldNotFound and InvalidMessage).
>>
>>
>> Once the parser tries to parse this message, and throws its  
>> exceptions, it
>> will get around to reading the socket again and receiving the  
>> following
>> into
>> buffer:
>>
>>
> IX.4.2
9=349
35=8
34=156111
369=31450
52=20041209-15:35: 
> 32.692
49=BLA
50=G
5
>>
> 6=D33030N
60=20041209-15:35: 
> 32
59=0
55=GE
54=1
48=CME000060467
107=BLF5
167=
>>
> FUT
44=9740.00
41=0
40=2
39=2
38=10
37=20041209004077
337=0C0
32=10
31= 
> 9740.
>>
> 00
151=0
150=2
20=0
9717=67960
17=00687220041209093532TN0002843
75=2004 
> 1209

>> 14=0
11=00067960
375=BLA030A1
6=0
432=20041209
1=12345678
10=178

>>
>> Once this is received the m_buffer is "8=F" and m_readBuffer is as  
>> above.
>> When the parser appends m_readBuffer to m_buffer the message looks  
>> like:
>>
>>
> 8=FIX.4.2
9=349
35=8
34=156111
369=31450
52=20041209-15:35: 
> 32.692
49=BLA
50=
>>
> G
56=D33030N
60=20041209-15:35: 
> 32
59=0
55=GE
54=1
48=CME000060467
107=BLF5
1
>>
> 67=FUT
44=9740.00
41=0
40=2
39=2
38=10
37=20041209004077
337=0C0
32=10
 
> 31=97
>>
> 40.00
151=0
150=2
20=0
9717=67960
17=00687220041209093532TN0002843
75=2 
> 00412
>> 09
14=0
11=00067960
375=BLA030A1
6=0
432=20041209
1=12345678
10=178

>>
>> a COMPLETE message as it should be.
>>
>> Now surely, if this kind of scenario is encountered the parser should  
>> not
>> even attempt to parse the message, as it is incomplete. I think that  
>> there
>> needs to be an extra safeguard in the parser that needs to check  
>> presence
>> of
>> field 10 (Standard Trailer) in order to verify that it is a complete
>> message.
>>
>> If anyone has any ideas how to solve this, please be my guess as it  
>> has
>> taken my over 2 days just to track this problem down.
>>
>> Thanks a lot
>>
>> Emir
>>
>> ---
>> Outgoing mail is certified Virus Free.
>> Checked by AVG anti-virus system (http://www.grisoft.com).
>> Version: 6.0.766 / Virus Database: 513 - Release Date: 9/17/2004
>>
>>
>>
>> -------------------------------------------------------
>> SF email is sponsored by - The IT Product Guide
>> Read honest & candid reviews on hundreds of IT Products from real  
>> users.
>> Discover which products truly live up to the hype. Start reading now.
>> http://productguide.itmanagersjournal.com/
>> _______________________________________________
>> Quickfix-users mailing list
>> Qui...@li...
>> https://lists.sourceforge.net/lists/listinfo/quickfix-users
>>
>
>
>
> -------------------------------------------------------
> SF email is sponsored by - The IT Product Guide
> Read honest & candid reviews on hundreds of IT Products from real  
> users.
> Discover which products truly live up to the hype. Start reading now.
> http://productguide.itmanagersjournal.com/
> _______________________________________________
> Quickfix-users mailing list
> Qui...@li...
> https://lists.sourceforge.net/lists/listinfo/quickfix-users
>
> ---
> Incoming mail is certified Virus Free.
> Checked by AVG anti-virus system (http://www.grisoft.com).
> Version: 6.0.766 / Virus Database: 513 - Release Date: 9/17/2004
>
>
> ---
> Outgoing mail is certified Virus Free.
> Checked by AVG anti-virus system (http://www.grisoft.com).
> Version: 6.0.766 / Virus Database: 513 - Release Date: 9/17/2004
>
>
> ---
> Outgoing mail is certified Virus Free.
> Checked by AVG anti-virus system (http://www.grisoft.com).
> Version: 6.0.766 / Virus Database: 513 - Release Date: 9/17/2004
>
>
>
> -------------------------------------------------------
> SF email is sponsored by - The IT Product Guide
> Read honest & candid reviews on hundreds of IT Products from real  
> users.
> Discover which products truly live up to the hype. Start reading now.
> http://productguide.itmanagersjournal.com/
> _______________________________________________
> Quickfix-users mailing list
> Qui...@li...
> https://lists.sourceforge.net/lists/listinfo/quickfix-users
>



-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now. 
http://productguide.itmanagersjournal.com/
_______________________________________________
Quickfix-users mailing list
Qui...@li...
https://lists.sourceforge.net/lists/listinfo/quickfix-users

---
Incoming mail is certified Virus Free.
Checked by AVG anti-virus system (http://www.grisoft.com).
Version: 6.0.766 / Virus Database: 513 - Release Date: 9/17/2004
 

---
Outgoing mail is certified Virus Free.
Checked by AVG anti-virus system (http://www.grisoft.com).
Version: 6.0.766 / Virus Database: 513 - Release Date: 9/17/2004