#701 Memory leak in command with list argument

[Johan Forsberg]

I've written a simple PyTango device that acts as a Tango Logging API endpoint in order to harvest log messages from other devices and store them in Elasticsearch. This means it has one command, "Log", which takes a list of strings as argument. This command gets run a lot, easily hundreds of times per second when doing debug logging from many devices.

Unfortunately, this has exposed a memory leak somewhere in the server, which causes the usage of the server process to grow at a rate of a megabyte every few seconds. I've seen it grow to 2 GB before I had to kill it.

It's easy to reproduce with a simple server with only one command taking a list of anything as an argument, returning nothing and doing nothing, and then running this command in a fast loop. The problem appears regardless of type in the list, but it does not occur with a scalar argument, so my guess it that there is some issue specifically related to deallocation of lists.

I've tried both the "old" server api and the high level API, with python 2.7 and python 3 and the result is the same AFAICT. I know there are some memory problems with python 2 related to allocation of many small objects, but I believe those have been solved in python 3. I have not tried if the bug is present in the C++ API too.

[Johan Forsberg]

An example program triggering the bug.

[Emmanuel Taurel]

Hi,

I have done the same test for a C++ device class (one command with DevVarStringArray
as input arg, returning nothing and doing nothing) on Ubuntu. After 10000 calls, valgrind
reports 0 definitely lost bytes. It seems the problem is in the PyTango layer

Regards

Emmanuel

[NexeyaSGara]

I have seen also a memory leeak simply by testing a TaurusDevicePanel on sys/tg_test/1. My python process keeps increasing memory usage.
So I think it's related to what you found.
And it's also a simple way to trigger it.
Windows 7, python 2.7, PyTango 8.1.6 and Taurus 3.4.0

[Jan Kotanski]

Hi,

I get a similar memory leak for my test PyTango server (attached bellow).

When I make calls for my function with a string list, i.e.

dp = PyTango.DeviceProxy("p09/mcs/r228")
for i in range(1000000):
    dp.command_inout("createConfiguration",["beamstop", "slit1", "slit2"])

I get leak of 80MB per 1000000 calls.

For a command without parameters. i.e.

dp = PyTango.DeviceProxy("p09/mcs/r228")
for i in range(1000000):
    dp.command_inout("open")

I do not get this leak.

I use debian wheezy with

python-pytango 8.1.1-1~bpo70+1
tango-common 8.1.2c+dfsg-4~bpo70+1
tango-db 8.1.2c+dfsg-4~bpo70+1

Bests,
Jan

[Jan Kotanski]

Hi,

Indeed, I could get this memory leak only in PyTango (not in pure C++ Tango).
Moreover, I've noticed that the leaked memory grows linearly with numbers of Strings in the list as well with lengths of these Strings, i.e.

dp = PyTango.DeviceProxy("p09/mcs/r228")
for i in range(1000000):
    dp.command_inout("createConfiguration",["beamstop"*1000, "slit1"*1000, "slit2"*1000]*100)

quickly kills my computer. So for me it seems that PyTango leaves a copy of the string list somewhere in memory without deleting it.

Bests,
Jan

[Jan Kotanski]

Confirm for others DevVar*Array types.

[Jan Kotanski]

Hi,

I've just noticed that to workaround this leak it is enough to compile PyTango with DISABLE_PYTANGO_NUMPY flag on. Actually, it is enough to remove a part of the code in extract_array(...) from server/command.cpp starting from #ifndef DISABLE_PYTANGO_NUMPY till #else, i.e. to call instead of this code a line which is bellow:

 py_result = to_py_list(tmp_ptr);

But, of course the real bug is somewhere in "removed" lines which convert a TangoArrayType object to a numpy array.

Bests,
Jan

[Jan Kotanski]

Hi,

Looking at the code it seems that

delete static_cast<TANGO_const2type(tangoTypeConst)*>(ptr_);

statement (from command.cpp) is not performed because
in TANGO_DO_ON_ATTRIBUTE_DATA_TYPE_ID macro (from tgutils.h) there is no
a proper case e.g. DEVVAR_STRINGARRAY case (but only DEV_STRING).

So if in dev_var_x_array_deleter__(PyObject* obj) (from command.cpp )

one exchanges

TANGO_DO_ON_ATTRIBUTE_DATA_TYPE_ID

to

TANGO_DO_ON_DEVICE_DATA_TYPE_ID

(with second parameter "empty") the bug should be fixed.

Bests,
Jan

[Jan Kotanski]

Hi Tiago,

could you please apply the attached patch (or something instead of it).

Bests,
Jan

[Tiago Coutinho]

Hi Jan,

I can reproduce the bug only for string array.
Thanks for the patch. I will apply it. Sorry for the late reply but I am having a very busy week.

[Jan Kotanski]

Hi Tiago,

Great. Thanks. Yes, I can image.

It is easy to catch the bug using the debuging mode, i.e. compiling PyTango with

OPT="-g -O3" python setup.py build

then the memory leak is catch by assert statement (bug #704).

Thus, e.g. in my example when I change the paramter type of my createConfiguration() example from PyTango.DevVarStringArray to PyTango.DevVarLongArray
after

dp = PyTango.DeviceProxy("p09/mcs/r228")
dp.command_inout("createConfiguration",[12,31,23]*10000)

I get

type <type 'numpy.ndarray'="">
python: /home/jkotan/sources/PyTango-trunk/src/boost/cpp/server/command.cpp:220: void dev_var_x_array_deleter__(PyObject*) [with long int type = 11l; PyObject = _object]: Assertion `false' failed.</type>

which means that in a non-debuging mode deleter is not executed.

Bests,
Jan

[Tiago Coutinho]

I confirm memory leak also for other data types.

Fixed. Will become visible in next PyTango 8.1.7

Thanks for reporting.

[Georg Brandl]

Hi Tiago, thanks for the fix. Is the newest revision available somewhere to test? The last commit in your repository on github appears to be from Apr 17...

[Tiago Coutinho]

Hi Georg,
I just pushed the changes to github. Feel free to try it out.
Let me know if you have any problems

[Georg Brandl]

Hm, I'm afraid I still see a memory leak with the master from github. I will try to get a minimal example. Maybe the reason is that we are using DevVarDoubleStringArray?

[Georg Brandl]

OK, that appears to have been #724. With that fixed too, I don't see leaks anymore. Thanks!