Re: [Quickfix-developers] C# Memory growing fast

SourceForge Headquarters 225 Broadway Suite 1600 San Diego, CA 92101 +1 (858) 422-6466

Sorry just got a good read of your original post, and your problem is
that you're storing the messages, QF stores a lot of information about
a message not just what the message is in ascii but also way to
effectively grab the specific data you need out, so yes if you store
the entire fix message through your programs normal processing your
going to notice some ineffeciences, you shouldn't be store the entire
message, you should just be storing what you need need pulled from the
message outside of the message.

On Tue, Mar 3, 2009 at 13:43, Bristotti, Fernando
<fer...@cr...> wrote:
> Malinka,
>
> The problem occurs also when I process the message with C++ code only.
>
> Dispose() has been called on ALL messages.
>
> Just to be clear, my problem is high memory cosuming, not leak of memory.
>
> Thanks,
> Fernando Bristotti
>
> -----Original Message-----
> From: Malinka Rellikwodahs [mailto:ael...@gm...]
> Sent: Tuesday, March 03, 2009 3:24 PM
> To: Ariel Piñeiro
> Cc: Bristotti, Fernando; qui...@li...
> Subject: Re: [Quickfix-developers] C# Memory growing fast
>
> In order to solve youre memory leaks you need to make sure that on ALL messages when you are done processing them you call .dispose().
> Because of the way managed C++ and C# integrate in QF there's no way for the the C# garbage collector to know when it needs to reclaim the messages because it doesn't know what size they are, if you look through the mailing list archives you can see this discussed before
>
> On Tue, Mar 3, 2009 at 12:50, Ariel Piñeiro <ari...@gm...> wrote:
>> QuickFIX Documentation:
>> http://www.quickfixengine.org/quickfix/doc/html/index.html
>> QuickFIX Support: http://www.quickfixengine.org/services.html
>>
>>
>> have the same problem and I do not know how to fix it!!!!!!!!!!!!!!!
>>
>> Memory leaks problem.
>>
>> 2009/3/3 Bristotti, Fernando <fer...@cr...>
>>>
>>> - Mostrar texto citado -
>>> QuickFIX Documentation:
>>> http://www.quickfixengine.org/quickfix/doc/html/index.html
>>> QuickFIX Support: http://www.quickfixengine.org/services.html
>>>
>>>
>>> Shane,
>>>
>>> The problem is in the way C++ allocates memory for keeping field order ( struct message_order ):
>>>
>>> quickfix\src\C++\MessageSorters.cpp:
>>>
>>> void message_order::setOrder( int size, const int order[] ) {
>>> QF_STACK_PUSH(message_order::operator=)
>>>
>>>   if(size < 1) return;
>>>   m_largest = m_delim = order[0];
>>>
>>>   int* fields = new int[ size + 1 ];
>>>   fields[ 1 ] = m_delim;
>>>   // collect all fields and find the largest field number
>>>   int i;
>>>   for ( i = 2; i <= size; ++i )
>>>   {
>>>     int field = order[i-1];
>>>     m_largest = m_largest > field ? m_largest : field;
>>>     fields[ i ] = field;
>>>   }
>>>
>>>   // populate array with field number as key and position as value
>>>   m_groupOrder = new int[ m_largest + 1 ];
>>>   memset( m_groupOrder, 0, ( m_largest + 1 ) * sizeof( int ) );
>>>   for ( i = 1; i <= size; ++i )
>>>     m_groupOrder[ fields[ i ] ] = i;
>>>   delete [] fields;
>>>
>>>   QF_STACK_POP
>>> }
>>>
>>> Let's suppose that you have a group with only two custom fields ( 10000, 10001 ) in a give fix message. To me, it doesn't make any sence to allocate a int[] vector size of the largest field (10001 in this case) for each entry of this group. Memory will grow really fast!!!!!
>>>
>>> Fernando Bristotti
>>> ________________________________
>>> From: Shane Trotter [mailto:str...@co...]
>>> Sent: Tuesday, March 03, 2009 12:15 PM
>>> To: Bristotti, Fernando
>>> Cc: qui...@li...
>>> Subject: Re: [Quickfix-developers] C# Memory growing fast
>>>
>>> I believe with the .NET version you must dispose each message after you have used it.  Is it a requirement in your application to maintain a reference to all 35,000 trade messages?
>>>
>>> --
>>> Shane Trotter
>>> Connamara Systems, LLC
>>>
>>> On Tue, Mar 3, 2009 at 9:06 AM, Bristotti, Fernando <fer...@cr...> wrote:
>>>>
>>>> QuickFIX Documentation:
>>>> http://www.quickfixengine.org/quickfix/doc/html/index.html
>>>> QuickFIX Support: http://www.quickfixengine.org/services.html
>>>>
>>>>
>>>> Hi,
>>>>
>>>> I am currently using QuickFIX 1.12.4. Memory grows fast when QuickFIX process a market data snapshot message, for instance the day I've received a snapshot with about 35000 trades (attached), memory grows to about 1.0G and my application crashes.
>>>>
>>>> Taking a look on QuickFIX source code I could see that struct message_order ( MessageSorters.cpp ) allocates for each group a vector int[] size of the largest field in the group even if the group contains only two fields. The largest field in NoMDEntry group for BM&F market data is a custom field = 6032, thus for processing the message above QuickFIX allocates 6032 x ( 35000 trades ) x ( 4 bytes ).
>>>>
>>>> I have changed the code in order to allocate a vector int[] size of the number of fields in the group, any suggestion?
>>>>
>>>> Thanks,
>>>> Fernando
>>>>
>>>> ====================================================================
>>>> ========== Please access the attached hyperlink for an important
>>>> electronic communications disclaimer:
>>>>
>>>> http://www.credit-suisse.com/legal/en/disclaimer_email_ib.html
>>>> ====================================================================
>>>> ==========
>>>>
>>>> --------------------------------------------------------------------
>>>> ---------- Open Source Business Conference (OSBC), March 24-25,
>>>> 2009, San Francisco, CA -OSBC tackles the biggest issue in open
>>>> source: Open Sourcing the Enterprise -Strategies to boost innovation
>>>> and cut costs with open source participation -Receive a $600
>>>> discount off the registration fee with the source code: SFAD
>>>> http://p.sf.net/sfu/XcvMzF8H
>>>> _______________________________________________
>>>> Quickfix-developers mailing list
>>>> Qui...@li...
>>>> https://lists.sourceforge.net/lists/listinfo/quickfix-developers
>>>
>>> =====================================================================
>>> ========= Please access the attached hyperlink for an important
>>> electronic communications disclaimer:
>>>
>>> http://www.credit-suisse.com/legal/en/disclaimer_email_ib.html
>>> =====================================================================
>>> =========
>>>
>>> ---------------------------------------------------------------------
>>> --------- Open Source Business Conference (OSBC), March 24-25, 2009,
>>> San Francisco, CA -OSBC tackles the biggest issue in open source:
>>> Open Sourcing the Enterprise -Strategies to boost innovation and cut
>>> costs with open source participation -Receive a $600 discount off the
>>> registration fee with the source code: SFAD
>>> http://p.sf.net/sfu/XcvMzF8H
>>> _______________________________________________
>>> Quickfix-developers mailing list
>>> Qui...@li...
>>> https://lists.sourceforge.net/lists/listinfo/quickfix-developers
>>
>>
>> ----------------------------------------------------------------------
>> -------- Open Source Business Conference (OSBC), March 24-25, 2009,
>> San Francisco, CA -OSBC tackles the biggest issue in open source: Open
>> Sourcing the Enterprise -Strategies to boost innovation and cut costs
>> with open source participation -Receive a $600 discount off the
>> registration fee with the source code: SFAD
>> http://p.sf.net/sfu/XcvMzF8H
>> _______________________________________________
>> Quickfix-developers mailing list
>> Qui...@li...
>> https://lists.sourceforge.net/lists/listinfo/quickfix-developers
>>
>
>
> ==============================================================================
> Please access the attached hyperlink for an important electronic communications disclaimer:
>
> http://www.credit-suisse.com/legal/en/disclaimer_email_ib.html
> ==============================================================================
>
>