quickfix-developers

[Quickfix-developers] Field with embedded null character

[Yihu F. <Yih...@re...>]

Hi,

=20

I understand that technically a field of string type which includes some
null character may be valid, i.e. Tag=3DABC<null><soh>. As stated in FIX
4.2 specification, "String: Alpha-numeric free format strings, can
include any character or punctuation except the delimiter".

=20

However, in QuickFIX message constructor, it takes string as the
parameter. It will throw an invalid message exception because it thinks
the string is truncated at the null character and cannot find the Tag
10.

=20

Is it true that such message is considered invalid in real world
scenario?

=20

Thanks.

=20

-Yihu

=20

=20



-----------------------------------------------------------------
        Visit our Internet site at http://www.reuters.com

Get closer to the financial markets with Reuters Messaging - for more
information and to register, visit http://www.reuters.com/messaging

Any views expressed in this message are those of  the  individual
sender,  except  where  the sender specifically states them to be
the views of Reuters Ltd.

Re: [Quickfix-developers] Field with embedded null character

[Caleb E. <cal...@gm...>]

On Wed, 27 Oct 2004 21:23:04 -0400, Yihu Fang <yih...@re...> wrote:
> However, in QuickFIX message constructor, it takes string as the parameter.
> It will throw an invalid message exception because it thinks the string is
> truncated at the null character and cannot find the Tag 10. 

Make sure you construct the string you pass to the Message constructor
properly.  The std::string constructor can take a const char* with or
without a length.  If you don't pass the length, it'll stop at the
first embedded \0 and you'll end up with a truncated message.

That said, it does seem that QuickFIX has a bug in the message
serialization code when dealing with fields having embedded \0 bytes:

#include <quickfix/Message.h>
#include <iostream>

using namespace FIX; using namespace std;

int main () {
  Message m;
  m.setField (1000, string ("one\0", 4));
  cout << m.toString () << endl;
  cout << m.getField (1000) << endl;
}

Gives the output (where ^A = ASCII SOH and \0 is an ASCII NULL)

9=10^A1000= one^A10=057^A
one\0

Note extra space and lack of \0 for serialized message form.

-- 
Caleb Epstein
cal...@gm...

Re: [Quickfix-developers] Field with embedded null character

[Caleb E. <cal...@gm...>]

On Thu, 28 Oct 2004 09:05:47 -0400, Caleb Epstein
<cal...@gm...> wrote:

> That said, it does seem that QuickFIX has a bug in the message
> serialization code when dealing with fields having embedded \0 bytes:

Ironically, the bug turns out to be in code that I submitted.  In
Field.h there is this optimization:

    char buf[64];

    if( 13 + m_string.length() < sizeof(buf) )
    {
      m_length = sprintf( buf, "%d=%*.*s\001", m_field,
                          (int)m_string.length(), (int)m_string.length(),
                          m_string.data() );
      m_data.assign( buf, m_length );
    }

At least on Linux, sprintf refuses to write anything after a \0 byte
to the output buffer, so the field gets padded with spaces on the
left.  Here's a re-worked version that is not as simple but is still
faster than the string + string + ... implementation that is used when
the value is too long to fit in buf:

    char buf[64];

    if( 13 + m_string.length() < sizeof(buf) )
    {
      int fidlen = sprintf (buf, "%d=", m_field);
      m_length = fidlen + m_string.length() + 1;
      memcpy (buf + fidlen, m_string.data(), m_string.length());
      buf[m_length - 1] = '\001';
      m_data.assign (buf, m_length);
    }

-- 
Caleb Epstein
cal...@gm...

Re: [Quickfix-developers] Field with embedded null character

[Oren M. <or...@qu...>]

Thanks.  I've already added a unit test for this case, so hopefully it 
will pass once I apply this.

--oren

On Oct 28, 2004, at 11:02 AM, Caleb Epstein wrote:

> On Thu, 28 Oct 2004 09:05:47 -0400, Caleb Epstein
> <cal...@gm...> wrote:
>
>> That said, it does seem that QuickFIX has a bug in the message
>> serialization code when dealing with fields having embedded \0 bytes:
>
> Ironically, the bug turns out to be in code that I submitted.  In
> Field.h there is this optimization:
>
>     char buf[64];
>
>     if( 13 + m_string.length() < sizeof(buf) )
>     {
>       m_length = sprintf( buf, "%d=%*.*s\001", m_field,
>                           (int)m_string.length(), 
> (int)m_string.length(),
>                           m_string.data() );
>       m_data.assign( buf, m_length );
>     }
>
> At least on Linux, sprintf refuses to write anything after a \0 byte
> to the output buffer, so the field gets padded with spaces on the
> left.  Here's a re-worked version that is not as simple but is still
> faster than the string + string + ... implementation that is used when
> the value is too long to fit in buf:
>
>     char buf[64];
>
>     if( 13 + m_string.length() < sizeof(buf) )
>     {
>       int fidlen = sprintf (buf, "%d=", m_field);
>       m_length = fidlen + m_string.length() + 1;
>       memcpy (buf + fidlen, m_string.data(), m_string.length());
>       buf[m_length - 1] = '\001';
>       m_data.assign (buf, m_length);
>     }
>
> -- 
> Caleb Epstein
> cal...@gm...
>