Re: [Quickfix-users] Possible bug in the Parser when receiving hu ge amounts through th e socket

[Oren M. <or...@qu...>]

No, I got it, I'm just designing a test to expose the problem.  Should be 
able to duplicate in a unit test to the Parser by feeding in the data in the 
same increments that caused your problem.

--oren

----- Original Message ----- 
From: <em...@co...>
To: <or...@qu...>
Cc: <qui...@li...>
Sent: Wednesday, December 15, 2004 3:36 AM
Subject: RE: [Quickfix-users] Possible bug in the Parser when receiving hu 
ge amounts through th e socket


> This is my post from earlier. I guess you didnt see it Oren. Here it goes
> anyway....
>
>
>
>
> Oren,
>
> AM_INIT_AUTOMAKE(quickfix, 1.5.0), but have also take some code from
> AM_INIT_AUTOMAKE(quickfix, 1.9.2) so I can have an upto date version of 
> the
> engine. But,  most of it is 1.5.0 with changes made to a few files in 
> order
> to have an upto date track of the bug fixes.
>
> Also, my test platform is Windows Server 2000 SP4 (quad 550Mhz processor
> machine, 1Gb memory), and have actually also made some changes to how data
> is received on the socket now, and how the socket is created. The socket 
> is
> now created as an overlapped socket explicitly through
> ::WSASocket( PF_INET, SOCK_STREAM, IPPROTO_TCP, NULL, 0, 
> WSA_FLAG_OVERLAPPED
> );
> So, this creates an overlapped socket and allows me to perform 
> asynchronous
> calls on the socket.
>
> I have started of a new thread which creates a completion port, and calls
> 'WSARecv' and then calls 'GetQueuedCompletionStatus' depending on the 
> value
> of the return of 'WSARecv'. The reason I have gone for this approach is 
> that
> when calling 'recv' on the socket and there is 5,000 bytes on the socket,
> and you give it a sufficiently large buffer (lets says 10,000 bytes), it 
> is
> not guaranteed that you are going to receive all 5,000 bytes in that 
> single
> receive. This is explained in the MSDN when calling 'ioctlsocket', or in 
> the
> code 'socket_fionread( m_socket, bytes )'. The below is copied direct from
> MSDN:
> ----------------------------------------------------------------------------
> -
> FIONREAD
> Use to determine the amount of data pending in the network's input buffer
> that can be read from socket s. The argp parameter points to an unsigned
> long value in which ioctlsocket stores the result. FIONREAD returns the
> amount of data that can be read in a single call to the recv function, 
> which
> may not be the same as the total amount of data queued on the socket. If s
> is message oriented (for example, type SOCK_DGRAM), FIONREAD still returns
> the amount of pending data in the network buffer, however, the amount that
> can actually be read in a single call to the recv function is limited to 
> the
> data size written in the send or sendto function call.
> ----------------------------------------------------------------------------
> -
>
> So, we can see that 'ioctlsocket' will let us know how much we can 
> actually
> read from the socket, even though there may be more data pending on the
> socket. So, I came to a scenario where I would have read a buffer of lets
> says 15-20 messages, but the end of the buffer would have looked like
> '8=FI', and on the next 'recv' call I would have got 'X.4.2' ....etc.
>
> I think that it is important to check for the existence of the standard
> trailer in the buffer before trying to extract the length, as it seems at
> least on the Microsoft platform you are not guaranteed to receive all the
> data from socket even if you give it a large enough buffer. I have only 
> been
> able to use a Microsoft platform as this is what we work on, and would be
> very interested to know on what platform you carried out your test.
>
> I do not think that this a problem specifically with any version, because 
> I
> check the very latest version of the parser on the internet
> (http://cvs.sourceforge.net/viewcvs.py/quickfix/quickfix/src/C%2B%2B/Parser.
> cpp?rev=1.11&view=markup) and
>
>
> bool Parser::readFixMessage( std::string& str )
> throw( MessageParseError, RecvFailed )
> { QF_STACK_PUSH(Parser::readFixMessage)
>
>  readFromStream();
>
>  std::string::size_type pos = 0;
>  if( m_buffer.length() < 2 ) return false;
>  pos = m_buffer.find( "8=" );
>  if( pos == std::string::npos ) return false;
>  m_buffer.erase( 0, pos );
>
> /*
> * NEED TO CHECK FOR STANDARD TRAILER HERE TO COMPENSATE FOR
> THE SCENARIO DESCRIBED ABOVE
> */
>
>  int length = 0;
>
>  try
>  {
>    if( extractLength(length, pos, m_buffer) )
>    {
>      pos += length;
>      if( m_buffer.size() < pos )
>        return false;
> -----------------------------------------------------
>
>
> That is the change I had made and have not seen the problem again.
>
> I hope this helps in some way, and once again I say that I have only seen
> this when creating huge amounts of traffic (ie. filling an order for 100
> lots and getting back 100 partial fills, etc), and on Microsoft platform.
>
>
> Regards,
>
>
> Emir Kamber
>
> -----Original Message-----
> From: Oren Miller [mailto:or...@qu...]
> Sent: Tuesday, December 14, 2004 2:57 PM
> To: em...@co...
> Cc: qui...@li...
> Subject: Re: [Quickfix-users] Possible bug in the Parser when receiving
> hu ge amounts through th e socket
>
>
> QuickFIX Documentation:
> http://www.quickfixengine.org/quickfix/doc/html/index.html
> QuickFIX FAQ: http://www.quickfixengine.org/wikifix/index.php?QuickFixFAQ
> QuickFIX Support: http://www.quickfixengine.org/services.html
>
> Emir,
>
> Can you tell me the version that is listed in your configure.in at the
> top, should be in this format AM_INIT_AUTOMAKE(quickfix,
> major.minor.revision)
>
> Here we can at least tell the release yours is based on.  My goal is to
> write a test for that version which duplicates the issue in a
> repeatable manner, and then run the test against the latest version to
> see if the problem still exists.  If not I will apply your patch and
> see if that does the trick.  Thanks.
>
> --oren
>
> On Dec 13, 2004, at 5:16 AM, em...@co... wrote:
>
>> QuickFIX Documentation:
>> http://www.quickfixengine.org/quickfix/doc/html/index.html
>> QuickFIX FAQ:
>> http://www.quickfixengine.org/wikifix/index.php?QuickFixFAQ
>> QuickFIX Support: http://www.quickfixengine.org/services.html
>>
>>
>> I am not quite suire of the version I am using but this is the code
>> that was
>> giving me trouble. I am not sure of the version becase I have
>> customised the
>> code heavily, so I have lately taken code from your updated versions
>> and big
>> fixes and just added, instead of taking a whole new release, sorry
>> about
>> this ;)...
>>
>>
>>
>> The following was giving me trouble. I think the problem is that the
>> 'extractLength' below should not be called before 'pos = m_buffer.find(
>> "\00110=", pos-1 );', this is because you cannot check for the length
>> unless
>> the 'standard trailer' is part of the message. This is the problem I
>> was
>> faced with where the only thing I would have in the buffer when trying
>> to
>> extract the length is '8=FI', and obviously at this point the
>> extractLength
>> would throw an exception. So, I have just modified it to check for the
>> standard message trailer before attempting to extract length and have
>> not
>> seen this problem.
>>
>> Here is the code below.
>>
>> bool Parser::readFixMessage( std::string& str )
>> throw( MessageParseError&, RecvFailed& )
>> { QF_STACK_PUSH(Parser::readFixMessage)
>>
>>   readFromStream();
>>
>>   std::string::size_type pos = 0;
>>
>>   if( m_buffer.length() < 2 )
>>   return false;
>>   pos = m_buffer.find( "8=" );
>>   if( pos == std::string::npos )
>>   return false;
>>
>>
>>   int pos_end_trailer_ = m_buffer.find( "\00110="); \\ I HAVE ADDED
>> THE FOLLOWING 3 LINES
>>   if(pos_end_trailer_ == std::string::npos) \\ AND HAVE
>> SEEMED TO SOLVED THE PRBLEM
>>   return false; \\
>>
>>   if(pos)
>>   m_buffer.erase( 0, pos );
>>
>>   int length = 0;
>>
>>   try
>>   {
>>     if( extractLength(length, pos, m_buffer) )
>>     {
>>       pos += length;
>>       if( m_buffer.size() < pos )
>>         return false;
>>
>>       pos = m_buffer.find( "\00110=", pos-1 );
>>       if( pos == std::string::npos ) return false;
>>       pos += 4;
>>       pos = m_buffer.find( "\001", pos );
>>       if( pos == std::string::npos ) return false;
>>       pos += 1;
>>
>>       str = m_buffer.substr( 0, pos );
>>       m_buffer.erase( 0, pos );
>>       return true;
>>     }
>>   }
>>   catch( MessageParseError& e )
>>   {
>>     m_buffer.erase( 0, pos + length );
>>     throw e;
>>   }
>>
>>   readFromStream();
>>   return false;
>>
>>   QF_STACK_POP
>> }
>>
>>
>> I hope this helps.
>>
>> Regards,
>>
>> Emir
>>
>> -----Original Message-----
>> From: Oren Miller [mailto:or...@qu...]
>> Sent: Friday, December 10, 2004 10:04 PM
>> To: em...@co...; qui...@li...
>> Subject: Re: [Quickfix-users] Possible bug in the Parser when receiving
>> huge amounts through th e socket
>>
>>
>> QuickFIX Documentation:
>> http://www.quickfixengine.org/quickfix/doc/html/index.html
>> QuickFIX FAQ:
>> http://www.quickfixengine.org/wikifix/index.php?QuickFixFAQ
>> QuickFIX Support: http://www.quickfixengine.org/services.html
>>
>> I think I do need some more information about the version you are
>> using.  I
>> wrote a test and ran it against 1.9.4 and was not able to duplicate the
>> behavior described.  The parser has changed over time so it may have
>> been
>> resolved somewhere along the way.  What version are you using?
>>
>> --oren
>>
>> ----- Original Message -----
>> From: <em...@co...>
>> To: <qui...@li...>
>> Sent: Thursday, December 09, 2004 10:14 AM
>> Subject: [Quickfix-users] Possible bug in the Parser when receiving
>> huge
>> amounts through th e socket
>>
>>
>>> QuickFIX Documentation:
>>> http://www.quickfixengine.org/quickfix/doc/html/index.html
>>> QuickFIX FAQ:
>>> http://www.quickfixengine.org/wikifix/index.php?QuickFixFAQ
>>> QuickFIX Support: http://www.quickfixengine.org/services.html
>>>
>>> Here is the scenario.
>>>
>>> At the moment the socket buffer in the parser is 4096. So, I have
>>> recently
>>> generated a lot of traffic so I constantly receive of 4096 bytes on
>>> the
>>> socket and have encountered the following problem.
>>>
>>> The end of the incoming buffer (m_buffer) of 4096 is as follows (some
>>> of
>>> the
>>> values have been substituted to hide trading details):
>>>
>>>
>> 8=FIX.4.2
9=291
35=8
34=156110
369=31450
52=20041209-15:35:
>> 32.687
49=BLA
50=
>>>
>> G
56=D33030N
60=20041209-15:35:
>> 32
59=0
55=GE
54=1
48=BLA000060467
107=BLF5
1
>>>
>> 67=FUT
44=9740.00
41=0
40=2
39=0
38=10
37=20041209004077
151=10
150=0
2
>> 0=0
9
>>>
>> 717=67960
17=00687120041209093532
14=0
11=00067960
6=0
432=20041209
1=1
>> 23456
>>> 78
10=097
8=F
>>>
>>> As we can see, when the parser gets to parsing this part of the
>>> buffer, it
>>> will extract the length of the valid message which is 294 (9=291) and
>>> in
>>> turn send of the message for processing into the application code,
>>> etc.
>>> After, it has processed the message the buffer then looks like:
>>>
>>> 8=F
>>>
>>> This is because the rest of the message is resting on the socket to be
>>> read
>>> so a valid FIX message can be composed.
>>>
>>> As you can see there is a 8=F at the end which was received into the
>>> buffer,
>>> and more data is left over on the socket to be read into the buffer.
>>> But,
>>> at
>>> the moment what the parser will do is make 8=F into a fix message as
>>> this
>>> is
>>> where the buffer is ending, and will try and extract length from it,
>>> and
>>> this generates a series of exceptions (FieldConvertError,
>>> MessageParseError,
>>> FieldNotFound and InvalidMessage).
>>>
>>>
>>> Once the parser tries to parse this message, and throws its
>>> exceptions, it
>>> will get around to reading the socket again and receiving the
>>> following
>>> into
>>> buffer:
>>>
>>>
>> IX.4.2
9=349
35=8
34=156111
369=31450
52=20041209-15:35:
>> 32.692
49=BLA
50=G
5
>>>
>> 6=D33030N
60=20041209-15:35:
>> 32
59=0
55=GE
54=1
48=CME000060467
107=BLF5
167=
>>>
>> FUT
44=9740.00
41=0
40=2
39=2
38=10
37=20041209004077
337=0C0
32=10
31=
>> 9740.
>>>
>> 00
151=0
150=2
20=0
9717=67960
17=00687220041209093532TN0002843
75=2004
>> 1209

>>> 14=0
11=00067960
375=BLA030A1
6=0
432=20041209
1=12345678
10=178

>>>
>>> Once this is received the m_buffer is "8=F" and m_readBuffer is as
>>> above.
>>> When the parser appends m_readBuffer to m_buffer the message looks
>>> like:
>>>
>>>
>> 8=FIX.4.2
9=349
35=8
34=156111
369=31450
52=20041209-15:35:
>> 32.692
49=BLA
50=
>>>
>> G
56=D33030N
60=20041209-15:35:
>> 32
59=0
55=GE
54=1
48=CME000060467
107=BLF5
1
>>>
>> 67=FUT
44=9740.00
41=0
40=2
39=2
38=10
37=20041209004077
337=0C0
32=10

>> 31=97
>>>
>> 40.00
151=0
150=2
20=0
9717=67960
17=00687220041209093532TN0002843
75=2
>> 00412
>>> 09
14=0
11=00067960
375=BLA030A1
6=0
432=20041209
1=12345678
10=178

>>>
>>> a COMPLETE message as it should be.
>>>
>>> Now surely, if this kind of scenario is encountered the parser should
>>> not
>>> even attempt to parse the message, as it is incomplete. I think that
>>> there
>>> needs to be an extra safeguard in the parser that needs to check
>>> presence
>>> of
>>> field 10 (Standard Trailer) in order to verify that it is a complete
>>> message.
>>>
>>> If anyone has any ideas how to solve this, please be my guess as it
>>> has
>>> taken my over 2 days just to track this problem down.
>>>
>>> Thanks a lot
>>>
>>> Emir
>>>
>>> ---
>>> Outgoing mail is certified Virus Free.
>>> Checked by AVG anti-virus system (http://www.grisoft.com).
>>> Version: 6.0.766 / Virus Database: 513 - Release Date: 9/17/2004
>>>
>>>
>>>
>>> -------------------------------------------------------
>>> SF email is sponsored by - The IT Product Guide
>>> Read honest & candid reviews on hundreds of IT Products from real
>>> users.
>>> Discover which products truly live up to the hype. Start reading now.
>>> http://productguide.itmanagersjournal.com/
>>> _______________________________________________
>>> Quickfix-users mailing list
>>> Qui...@li...
>>> https://lists.sourceforge.net/lists/listinfo/quickfix-users
>>>
>>
>>
>>
>> -------------------------------------------------------
>> SF email is sponsored by - The IT Product Guide
>> Read honest & candid reviews on hundreds of IT Products from real
>> users.
>> Discover which products truly live up to the hype. Start reading now.
>> http://productguide.itmanagersjournal.com/
>> _______________________________________________
>> Quickfix-users mailing list
>> Qui...@li...
>> https://lists.sourceforge.net/lists/listinfo/quickfix-users
>>
>> ---
>> Incoming mail is certified Virus Free.
>> Checked by AVG anti-virus system (http://www.grisoft.com).
>> Version: 6.0.766 / Virus Database: 513 - Release Date: 9/17/2004
>>
>>
>> ---
>> Outgoing mail is certified Virus Free.
>> Checked by AVG anti-virus system (http://www.grisoft.com).
>> Version: 6.0.766 / Virus Database: 513 - Release Date: 9/17/2004
>>
>>
>> ---
>> Outgoing mail is certified Virus Free.
>> Checked by AVG anti-virus system (http://www.grisoft.com).
>> Version: 6.0.766 / Virus Database: 513 - Release Date: 9/17/2004
>>
>>
>>
>> -------------------------------------------------------
>> SF email is sponsored by - The IT Product Guide
>> Read honest & candid reviews on hundreds of IT Products from real
>> users.
>> Discover which products truly live up to the hype. Start reading now.
>> http://productguide.itmanagersjournal.com/
>> _______________________________________________
>> Quickfix-users mailing list
>> Qui...@li...
>> https://lists.sourceforge.net/lists/listinfo/quickfix-users
>>
>
>
>
> -------------------------------------------------------
> SF email is sponsored by - The IT Product Guide
> Read honest & candid reviews on hundreds of IT Products from real users.
> Discover which products truly live up to the hype. Start reading now.
> http://productguide.itmanagersjournal.com/
> _______________________________________________
> Quickfix-users mailing list
> Qui...@li...
> https://lists.sourceforge.net/lists/listinfo/quickfix-users
>
> ---
> Incoming mail is certified Virus Free.
> Checked by AVG anti-virus system (http://www.grisoft.com).
> Version: 6.0.766 / Virus Database: 513 - Release Date: 9/17/2004
>
>
> ---
> Outgoing mail is certified Virus Free.
> Checked by AVG anti-virus system (http://www.grisoft.com).
> Version: 6.0.766 / Virus Database: 513 - Release Date: 9/17/2004
>
>