#33 sr_poll - memory leak problem

[Jun Hu]

sr_poll process memory keeps growing with time.
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
12419 sarra 20 0 19.1g 14g 2268 S 0 31.1 614:07.82 sr_poll
27294 sarra 20 0 6420m 6.1g 2364 S 4 13.0 240:45.76 sr_poll
12415 sarra 20 0 155m 57m 1940 S 0 0.1 39:35.04 sr_poll

[psilva]

working around it with daily cronjobs to restart.

[Murray Rennie]

How about I try to use http://www.lshift.net/blog/2008/11/14/tracing-python-memory-leaks/ (if I get it to work) to figure this out. not gaurenteeing anything. Plus I will be learning how to set this up.

[Murray Rennie]

Does sr_poll.py ever run on it's own? I've been having some issues with running it on it's own.
for one the config doc says "filename" is an option but I found that if it's not set nothing seems to get posted to the broker. And if VIP and interface aren't set it doesn't seem to get to post either.

And I'm now seeing an error stating that

2016-06-21 20:07:32,055 [ERROR] Type: <class 'attributeerror'="">, Value: 'sr_poll' object has no attribute 'info', ...
I've been putting etatements in to see what's what.
As far as I know I could be doing something wrong. But this error seems to prevent me from going past the first entry in the list of files to be transfered. I can see how this would look like a leak if the array is never stepped though and cleared.</class>

[Murray Rennie]

And it looks like I was using an older copy of Sarra on my machine. the ERROR above has long since been taken care of.

But could the memory leak be because the internal array of files is never decreased in size as long as nothing is deleted on the site being polled?

[psilva]

next week, 2.17.03a5 will be installed on pxatx. Will look to see if the problem is still there.

[psilva]

was installed, problem is still there. On the other hand, it is sure that the problem is not
in sr_poll itself because the mirroring case for HPC hall-A-B has no memory problem at all.
so it is definitely in the NWS plugin somewhere.

Don't have a good approach to this.
some ideas:

https://mflerackers.wordpress.com/2012/04/12/fixing-and-avoiding-memory-leaks-in-python/
http://pclib.github.io/safari/program/python-cookbook/Text/ch08s23.html

Thinking that the self.parent = parent pattern, likely is a good candidate to use weakref,
but how to measure... kind of strange to just blindly apply changes.

[psilva]

another link, more explicit:

https://stackoverflow.com/questions/10791588/getting-container-parent-object-from-within-python/10791613#10791613

[psilva]

from Jun:

I’m not sure of your conclusion.
Here’s my reason:
- other poll like NDE have the memory leak problem too but not as much as serious then NWS (for ex. now NDE = 0.5% and NWS = 3.5%).
- NDE has much les quantity of the files then NWS.
- The policy poll don’t have the saving files list like NWS (for ex: -rw-rw-rw- 1 sarra px 27882 Jun 11 15:38 ls_2canada_bulletins_normal_priority)

Is it possible that sr_poll don’t closed properly the file descriptor of ls_?

very good point. Don't know. there is still a bit inside sr_poll (list comparison logic) that could easily be the source.

[psilva]

now: https://github.com/MetPX/sarracenia/issues/47