Re: [opendds-devel] Subscriber memory leak when publisher crashes

[Steve T. <tot...@oc...>]

Hello Mario,

Thank you for your interest in OpenDDS.  I forwarded your request to my
colleagues at OCI and we have some comments and questions for you,
below.

On 3/19/2010 4:26 AM, Mario Danelli wrote:

> in test we are executing in our laboratories about OpenDDS use we
> encounter a problem.
>
> We are executing this test:
>
>      * Java subscriber with default QOS
>      * C++ publisher with default QOS
>      * OpenDDS v2.1.1
>      * Repository started with "-NOBITS" command-line arguments for delay
>        problem encountered during test hosted in two different hosts
>        (instead of the same for publisher and subscriber processes)

Your description of the "delay problem" is not very specific.  I may or
may not be related to the problem you describe, below.  It is hard to
tell without more information.  Could you provide more details?

I was wondering if it might have something to do with host name
resolution, since you do not see the delay when publisher and subscriber
are on the same host.  Perhaps one of the host names is not resolvable
from the host on which the DCPSInfoRepo process is run.

Or, it may just be the delay in the participants getting their Built-in
Topic association established.  We resolved the known race conditions
with built-in topics some time ago, so it should work correctly 
regardless of ordering.  However, we may have missed something.

Also, you can use the StatusCondition for 'subscriptions_matched' in the
publishing application to wait for associations to be established
before starting to write (if that is an issue for you).  And, you can
use DataWriter::wait_for_acknowledgments() to wait for published data
to be received before tearing down in the publishing application (if
that is an issue).

Now, on to the rest of your issue...

> When the subscriber starts "n" MB are allocated to the subscriber memory.
> When the publisher starts other ~6.3 MB are allocated to the subscriber
> memory.
> If the publisher close correctly (execute the publisher clean-up code)
> the ~6.3 MB are deallocated from the subscriber memory.
> If the publisher crashes (simulated with CTRL+C), and so none publisher
> clean-up code is executed, the ~6.3 MB are NOT deallocated from the
> subscriber memory.

> The problem is encountered both in tests in the same host or in two
> different hosts.
>
> Instead the ~6.3 MB are deallocated from the publisher memory if is the
> subscriber to crash.
>
> There are some command-line arguments or QOS policy to use to solve this
> problem?

It may be that the subscription still has an association with the
publication (since the publishing process did not shutdown
gracefully).  You might  be able to use LIVELINESS or DEADLINE QoS to
detect the status condition change (either via a listener or a wait
set) when the writer becomes NOT_ALIVE (or is no longer writing) and
take whatever action you need to in the subscribing application.

Note that the writer becoming NOT_ALIVE only unregisters that specific
instance, which explicitly does *not* do a dispose() on it.  But, the
application could dispose() the instance on the status condition change,
if it desires.

If a dispose() is not explicitly called, then the subscription will
retain the instance data as the writer may reappear (re-associate) and
increase the generation count.  This may be the cause of one of the
memory effects you are seeing.

In addition, the LIFESPAN and READER_DATA_LIFECYCLE QoS policies may
address some of the issues with how long the data is kept in the
application.  Also, memory usage can be tuned, to some extent, using
the HISTORY, RESOURCE_LIMITS, and WRITER_DATA_LIFECYCLE policy values.
OpenDDS preallocates a pool of storage using the buffer sizes indicated
by those policy values.

Finally, you could use a signal handler in your subscribing application
to make sure all of the cleanup code is called.  Otherwise, the
DCPSInfoRepo may not realize the publishing process is gone.

> We realized that starting a second subscriber for the same topic make
> deallocate the memory not previously deallocated from the first subscriber.

It may be that registering the new subscription causes the DCPSInfoRepo
to re-evaluate its associations and determine that the original
subscription's association with the publication should be torn down.
The DCPSInfoRepo would try to reach the publishing process to establish
the new association, but since the publishing process has crashed, an
exception would be raised, so the DCPSInfoRepo would remove the
publication and remove the association with the original subscribing
process, which then cleans up the memory.

> Thanks in advance.

You are welcome.  I hope we have been able to provide some useful
guidance to you.  If you will be using OpenDDS in your project, you
may want to consider establishing a support agreement with OCI.  Feel
free to contact me directly, or see http://www.opendds.org/support,
for more information.

Steve
-- 
----------------------------------------------------------------
  Steve Totten, Principal Software Engineer and Partner
  Object Computing, Inc. (OCI), St. Louis, MO, USA
  http://www.ociweb.com/  http://www.theaceorb.com/
----------------------------------------------------------------