Re: [Quickfix-users] Possible bug in the Parser when receiving hu ge amounts through th e socket

SourceForge Headquarters 1320 Columbia Street Suite 310 San Diego, CA 92101 +1 (858) 422-6466

Emir,

Can you tell me the version that is listed in your configure.in at the =20=

top, should be in this format AM_INIT_AUTOMAKE(quickfix, =20
major.minor.revision)

Here we can at least tell the release yours is based on.  My goal is to =20=

write a test for that version which duplicates the issue in a =20
repeatable manner, and then run the test against the latest version to =20=

see if the problem still exists.  If not I will apply your patch and =20
see if that does the trick.  Thanks.

--oren

On Dec 13, 2004, at 5:16 AM, em...@co... wrote:

> QuickFIX Documentation: =20
> http://www.quickfixengine.org/quickfix/doc/html/index.html
> QuickFIX FAQ: =20
> http://www.quickfixengine.org/wikifix/index.php?QuickFixFAQ
> QuickFIX Support: http://www.quickfixengine.org/services.html
>
>
> I am not quite suire of the version I am using but this is the code =20=

> that was
> giving me trouble. I am not sure of the version becase I have =20
> customised the
> code heavily, so I have lately taken code from your updated versions =20=

> and big
> fixes and just added, instead of taking a whole new release, sorry =20
> about
> this ;)...
>
>
>
> The following was giving me trouble. I think the problem is that the
> 'extractLength' below should not be called before 'pos =3D =
m_buffer.find(
> "\00110=3D", pos-1 );', this is because you cannot check for the =
length =20
> unless
> the 'standard trailer' is part of the message. This is the problem I =20=

> was
> faced with where the only thing I would have in the buffer when trying =
=20
> to
> extract the length is '8=3DFI', and obviously at this point the =20
> extractLength
> would throw an exception. So, I have just modified it to check for the
> standard message trailer before attempting to extract length and have =20=

> not
> seen this problem.
>
> Here is the code below.
>
> bool Parser::readFixMessage( std::string& str )
> throw( MessageParseError&, RecvFailed& )
> { QF_STACK_PUSH(Parser::readFixMessage)
>
>   readFromStream();
>
>   std::string::size_type pos =3D 0;
>
>   if( m_buffer.length() < 2 )
> 	  return false;
>   pos =3D m_buffer.find( "8=3D" );
>   if( pos =3D=3D std::string::npos )
> 	  return false;
>
>
>   int pos_end_trailer_ =3D m_buffer.find( "\00110=3D");	\\	=
I HAVE ADDED
> THE FOLLOWING 3 LINES
>   if(pos_end_trailer_ =3D=3D std::string::npos)		\\ 	=
AND HAVE
> SEEMED TO SOLVED THE PRBLEM
> 	  return false;						\\
>
>   if(pos)
> 	  m_buffer.erase( 0, pos );
>
>   int length =3D 0;
>
>   try
>   {
>     if( extractLength(length, pos, m_buffer) )
>     {
>       pos +=3D length;
>       if( m_buffer.size() < pos )
>         return false;
>
>       pos =3D m_buffer.find( "\00110=3D", pos-1 );
>       if( pos =3D=3D std::string::npos ) return false;
>       pos +=3D 4;
>       pos =3D m_buffer.find( "\001", pos );
>       if( pos =3D=3D std::string::npos ) return false;
>       pos +=3D 1;
>
>       str =3D m_buffer.substr( 0, pos );
>       m_buffer.erase( 0, pos );
>       return true;
>     }
>   }
>   catch( MessageParseError& e )
>   {
>     m_buffer.erase( 0, pos + length );
>     throw e;
>   }
>
>   readFromStream();
>   return false;
>
>   QF_STACK_POP
> }
>
>
> I hope this helps.
>
> Regards,
>
> Emir
>
> -----Original Message-----
> From: Oren Miller [mailto:or...@qu...]
> Sent: Friday, December 10, 2004 10:04 PM
> To: em...@co...; qui...@li...
> Subject: Re: [Quickfix-users] Possible bug in the Parser when =
receiving
> huge amounts through th e socket
>
>
> QuickFIX Documentation:
> http://www.quickfixengine.org/quickfix/doc/html/index.html
> QuickFIX FAQ: =20
> http://www.quickfixengine.org/wikifix/index.php?QuickFixFAQ
> QuickFIX Support: http://www.quickfixengine.org/services.html
>
> I think I do need some more information about the version you are =20
> using.  I
> wrote a test and ran it against 1.9.4 and was not able to duplicate =
the
> behavior described.  The parser has changed over time so it may have =20=

> been
> resolved somewhere along the way.  What version are you using?
>
> --oren
>
> ----- Original Message -----
> From: <em...@co...>
> To: <qui...@li...>
> Sent: Thursday, December 09, 2004 10:14 AM
> Subject: [Quickfix-users] Possible bug in the Parser when receiving =20=

> huge
> amounts through th e socket
>
>
>> QuickFIX Documentation:
>> http://www.quickfixengine.org/quickfix/doc/html/index.html
>> QuickFIX FAQ: =20
>> http://www.quickfixengine.org/wikifix/index.php?QuickFixFAQ
>> QuickFIX Support: http://www.quickfixengine.org/services.html
>>
>> Here is the scenario.
>>
>> At the moment the socket buffer in the parser is 4096. So, I have =20
>> recently
>> generated a lot of traffic so I constantly receive of 4096 bytes on =20=

>> the
>> socket and have encountered the following problem.
>>
>> The end of the incoming buffer (m_buffer) of 4096 is as follows (some =
=20
>> of
>> the
>> values have been substituted to hide trading details):
>>
>>
> 8=3DFIX.4.2=019=3D291=0135=3D8=0134=3D156110=01369=3D31450=0152=3D200412=
09-15:35:=20
> 32.687=0149=3DBLA=0150=3D
>>
> G=0156=3DD33030N=0160=3D20041209-15:35:=20
> 32=0159=3D0=0155=3DGE=0154=3D1=0148=3DBLA000060467=01107=3DBLF5=011
>>
> =
67=3DFUT=0144=3D9740.00=0141=3D0=0140=3D2=0139=3D0=0138=3D10=0137=3D200412=
09004077=01151=3D10=01150=3D0=012=20
> 0=3D0=019
>>
> =
717=3D67960=0117=3D00687120041209093532=0114=3D0=0111=3D00067960=016=3D0=01=
432=3D20041209=011=3D1=20
> 23456
>> 78=0110=3D097=018=3DF
>>
>> As we can see, when the parser gets to parsing this part of the =20
>> buffer, it
>> will extract the length of the valid message which is 294 (9=3D291) =
and =20
>> in
>> turn send of the message for processing into the application code, =20=

>> etc.
>> After, it has processed the message the buffer then looks like:
>>
>> 8=3DF
>>
>> This is because the rest of the message is resting on the socket to =
be
>> read
>> so a valid FIX message can be composed.
>>
>> As you can see there is a 8=3DF at the end which was received into =
the
>> buffer,
>> and more data is left over on the socket to be read into the buffer. =20=

>> But,
>> at
>> the moment what the parser will do is make 8=3DF into a fix message =
as =20
>> this
>> is
>> where the buffer is ending, and will try and extract length from it, =20=

>> and
>> this generates a series of exceptions (FieldConvertError,
>> MessageParseError,
>> FieldNotFound and InvalidMessage).
>>
>>
>> Once the parser tries to parse this message, and throws its =20
>> exceptions, it
>> will get around to reading the socket again and receiving the =20
>> following
>> into
>> buffer:
>>
>>
> IX.4.2=019=3D349=0135=3D8=0134=3D156111=01369=3D31450=0152=3D20041209-15=
:35:=20
> 32.692=0149=3DBLA=0150=3DG=015
>>
> 6=3DD33030N=0160=3D20041209-15:35:=20
> 32=0159=3D0=0155=3DGE=0154=3D1=0148=3DCME000060467=01107=3DBLF5=01167=3D=

>>
> =
FUT=0144=3D9740.00=0141=3D0=0140=3D2=0139=3D2=0138=3D10=0137=3D20041209004=
077=01337=3D0C0=0132=3D10=0131=3D=20
> 9740.
>>
> =
00=01151=3D0=01150=3D2=0120=3D0=019717=3D67960=0117=3D00687220041209093532=
TN0002843=0175=3D2004=20
> 1209=01
>> 14=3D0=0111=3D00067960=01375=3DBLA030A1=016=3D0=01432=3D20041209=011=3D=
12345678=0110=3D178=01
>>
>> Once this is received the m_buffer is "8=3DF" and m_readBuffer is as =20=

>> above.
>> When the parser appends m_readBuffer to m_buffer the message looks =20=

>> like:
>>
>>
> 8=3DFIX.4.2=019=3D349=0135=3D8=0134=3D156111=01369=3D31450=0152=3D200412=
09-15:35:=20
> 32.692=0149=3DBLA=0150=3D
>>
> G=0156=3DD33030N=0160=3D20041209-15:35:=20
> 32=0159=3D0=0155=3DGE=0154=3D1=0148=3DCME000060467=01107=3DBLF5=011
>>
> =
67=3DFUT=0144=3D9740.00=0141=3D0=0140=3D2=0139=3D2=0138=3D10=0137=3D200412=
09004077=01337=3D0C0=0132=3D10=01=20
> 31=3D97
>>
> =
40.00=01151=3D0=01150=3D2=0120=3D0=019717=3D67960=0117=3D00687220041209093=
532TN0002843=0175=3D2=20
> 00412
>> 09=0114=3D0=0111=3D00067960=01375=3DBLA030A1=016=3D0=01432=3D20041209=01=
1=3D12345678=0110=3D178=01
>>
>> a COMPLETE message as it should be.
>>
>> Now surely, if this kind of scenario is encountered the parser should =
=20
>> not
>> even attempt to parse the message, as it is incomplete. I think that =20=

>> there
>> needs to be an extra safeguard in the parser that needs to check =20
>> presence
>> of
>> field 10 (Standard Trailer) in order to verify that it is a complete
>> message.
>>
>> If anyone has any ideas how to solve this, please be my guess as it =20=

>> has
>> taken my over 2 days just to track this problem down.
>>
>> Thanks a lot
>>
>> Emir
>>
>> ---
>> Outgoing mail is certified Virus Free.
>> Checked by AVG anti-virus system (http://www.grisoft.com).
>> Version: 6.0.766 / Virus Database: 513 - Release Date: 9/17/2004
>>
>>
>>
>> -------------------------------------------------------
>> SF email is sponsored by - The IT Product Guide
>> Read honest & candid reviews on hundreds of IT Products from real =20
>> users.
>> Discover which products truly live up to the hype. Start reading now.
>> http://productguide.itmanagersjournal.com/
>> _______________________________________________
>> Quickfix-users mailing list
>> Qui...@li...
>> https://lists.sourceforge.net/lists/listinfo/quickfix-users
>>
>
>
>
> -------------------------------------------------------
> SF email is sponsored by - The IT Product Guide
> Read honest & candid reviews on hundreds of IT Products from real =20
> users.
> Discover which products truly live up to the hype. Start reading now.
> http://productguide.itmanagersjournal.com/
> _______________________________________________
> Quickfix-users mailing list
> Qui...@li...
> https://lists.sourceforge.net/lists/listinfo/quickfix-users
>
> ---
> Incoming mail is certified Virus Free.
> Checked by AVG anti-virus system (http://www.grisoft.com).
> Version: 6.0.766 / Virus Database: 513 - Release Date: 9/17/2004
>
>
> ---
> Outgoing mail is certified Virus Free.
> Checked by AVG anti-virus system (http://www.grisoft.com).
> Version: 6.0.766 / Virus Database: 513 - Release Date: 9/17/2004
>
>
> ---
> Outgoing mail is certified Virus Free.
> Checked by AVG anti-virus system (http://www.grisoft.com).
> Version: 6.0.766 / Virus Database: 513 - Release Date: 9/17/2004
>
>
>
> -------------------------------------------------------
> SF email is sponsored by - The IT Product Guide
> Read honest & candid reviews on hundreds of IT Products from real =20
> users.
> Discover which products truly live up to the hype. Start reading now.
> http://productguide.itmanagersjournal.com/
> _______________________________________________
> Quickfix-users mailing list
> Qui...@li...
> https://lists.sourceforge.net/lists/listinfo/quickfix-users
>