Re: [opendds-devel] Subscriber memory leak when publisher crashes

[Steve T. <tot...@oc...>]

Hi Mario,

Mario Danelli wrote:
>
> we are still looking for an automatic way (with a parameter or a
> command-line) to release the allocated resources (automatically
> allocated from OpenDDS when publisher and subscriber are matched) in the
> Subscriber memory when a Publisher is unreachable (as happens in the
> Publisher memory when is the Subscriber to be unreachable) because seems
> what we have tried until now (also using the QoS parameters you
> specified and listed in the developer guide) didn't work.

OK.  For reference, here is the original problem description:

>> On 3/19/2010 4:26 AM, Mario Danelli wrote:
>>
>>> in test we are executing in our laboratories about OpenDDS use we
>>> encounter a problem.
>>>
>>> We are executing this test:
>>>
>>>        * Java subscriber with default QOS
>>>        * C++ publisher with default QOS
>>>        * OpenDDS v2.1.1
>>>        * Repository started with "-NOBITS" command-line arguments for delay
>>>          problem encountered during test hosted in two different hosts
>>>          (instead of the same for publisher and subscriber processes)
>>>
>>> When the subscriber starts "n" MB are allocated to the subscriber memory.
>>> When the publisher starts other ~6.3 MB are allocated to the subscriber
>>> memory.
>>> If the publisher close correctly (execute the publisher clean-up code)
>>> the ~6.3 MB are deallocated from the subscriber memory.
>>> If the publisher crashes (simulated with CTRL+C), and so none publisher
>>> clean-up code is executed, the ~6.3 MB are NOT deallocated from the
>>> subscriber memory.
>>
>>> The problem is encountered both in tests in the same host or in two
>>> different hosts.
>>>
>>> Instead the ~6.3 MB are deallocated from the publisher memory if is the
>>> subscriber to crash.
>>>
>>> There are some command-line arguments or QOS policy to use to solve this
>>> problem?
>>>
>>> We realized that starting a second subscriber for the same topic make
>>> deallocate the memory not previously deallocated from the first subscriber.

At first, it would seem that LIVELINESS QoS would address the issue,
since it would allow the subscribing application to be informed that
the writer had gone away, via a change in the LIVELINESS_CHANGED_STATUS
communication status.  But, a colleague of mine explained why this is
not enough.  I paraphrase his explanation here:

> The liveliness status change will bump the generation count, but the
> samples (allocated resources) are not removed until an instance is
> 'disposed'.  Disposal does not occur naturally as the service assumes
> that an instance may be incarnated at a later time (the generation
> count keeps track of this) and the resources (samples) are still
> relevant.  The application is responsible for indicating when the
> resources are no longer valid.  Unfortunately this means that a
> crashed publication will not be able to dispose an instance - leaking
> the resources on the receive side.

You can manage the amount of resource consumed with HISTORY and
RESOURCE_LIMITS QoS, but the liveliness status change will not trigger
them to be cleaned up.

However, you should be able to cause the resources to be automatically
purged after some configurable delay interval after the liveliness
status change.  You can do this using the READER_DATA_LIFECYCLE QoS
policy.  The value of this policy is a structure with two data members:

struct ReaderDataLifecycleQosPolicy {
   Duration_t autopurge_nowriter_samples_delay;
   Duration_t autopurge_disposed_samples_delay;
};

In the case of the writer abruptly going away due to publishing
application crashing, the instance state will change to
NOT_ALIVE_NO_WRITERS.  Then, after a delay specified by the
autopurge_nowriter_samples_delay member of this policy value, the
reader will purge the resources allocated for the samples and other
information for that instance.  The default delay is infinite, which
means the samples are never purged.

The OpenDDS Developer's Guide, section 3.2.20, for an explanation of
the READER_DATA_LIFECYCLE QoS policy.  See
$DDS_ROOT/tests/DCPS/ReaderDataLifecycle/ for an example of using this
policy.

Another option might be to add a feature to the DCPSInfoRepo to allow it
to detect (or be informed) when a publication has gone away and have it
initiate a tear-down of associations between the defunct publication
and its matching subscriptions.  (Something like that is happening when
your second subscriber is started.)  However, adding such a feature
would require some out-of-band communications between applications and
the DCPSInfoRepo (such as pings or periodic status updates).  It would
be better to handle this resources clean-up as part of the normal
writer/reader communications.

Please give the READER_DATA_LIFECYCLE QoS policy a try and let us know
if it addresses the resource clean-up issue in your application.

Thank you for trying out OpenDDS!

Steve
-- 
----------------------------------------------------------------
  Steve Totten, Principal Software Engineer and Partner
  Object Computing, Inc. (OCI), St. Louis, MO, USA
  http://www.ociweb.com/  http://www.theaceorb.com/
----------------------------------------------------------------