[Jython-bugs] [issue1761] threading memory leak

[Timothy R. <re...@bu...>]

New submission from Timothy Reaves <bv...@sn...>:

Actually the version I am using is 2.5.2, but the Versions list does not offer that as a selection.  I upgraded to 2.5.2 from 2.5.2rc4 to see if this issue is resolved.

This issue may be 1660 - that issue is marked closed, but, if it is not that issue, it is another, very similar one.

I use threading for work in my code.  It's code copied from one of the Python doc pages.

class Worker(Thread):
	def __init__(self, tasks):
		Thread.__init__(self)
		self.tasks = tasks
		self.daemon = True
		self.start()

	def run(self):
		while True:
			func, args, kargs = self.tasks.get()
			try:
				func(*args, **kargs)
			except Exception, exception:
				logging.error(exception, extra={"sbs_app":"Worker"})
			self.tasks.task_done()

class ThreadPool:
	def __init__(self, num_threads):
		self.tasks = Queue(num_threads)
		for _ in range(num_threads):
			Worker(self.tasks)

	def add_task(self, func, *args, **kargs):
		self.tasks.put((func, args, kargs), True)

	def wait_completion(self):
		self.tasks.join()


In this case, Jython is being executed by ODI, an Oracle product (I don't really think that's relevant, but...).  I run this code with 32 threads, and then use the wait_for_completion().  All tasks run and finish fine.  However, none of the 32 threads ever goes away.

I run jconsole, and when I kick off my process, I see the thread count increase, typically by 35 threads (some are ODI's).  When the process has finished, and ODI finishes it's stuff, the delta is always exactly 32; it just keeps growing.  Eventually, the VM runs out of memory, or the ulimit is hit.

In jconsole, all of the threads are still visible, and show:
Name: Thread
State: WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@57cac7
Total blocked: 38  Total waited: 140

Stack trace: 
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
org.python.modules._threading.Condition.Condition_wait(Condition.java:87)
org.python.modules._threading.Condition$Condition_wait_exposer.__call__(Unknown Source)
org.python.core.PyObject.__call__(PyObject.java:375)
Queue$py.get$12(/u01/app/oracle/product/ODI/oracledi_J251/lib/scripting/Lib/Queue.py:179)
Queue$py.call_function(/u01/app/oracle/product/ODI/oracledi_J251/lib/scripting/Lib/Queue.py)
org.python.core.PyTableCode.call(PyTableCode.java:165)
org.python.core.PyBaseCode.call(PyBaseCode.java:301)
org.python.core.PyBaseCode.call(PyBaseCode.java:127)
org.python.core.PyFunction.__call__(PyFunction.java:317)
org.python.core.PyMethod.__call__(PyMethod.java:109)
sbs.threadsupport$py.run$3(/opt/csw/lib/python/site-packages/ImageProcessingPipeline-2.3.1-py2.6.egg/sbs/threadsupport.py:44)
sbs.threadsupport$py.call_function(/opt/csw/lib/python/site-packages/ImageProcessingPipeline-2.3.1-py2.6.egg/sbs/threadsupport.py)
org.python.core.PyTableCode.call(PyTableCode.java:165)
org.python.core.PyBaseCode.call(PyBaseCode.java:134)
org.python.core.PyFunction.__call__(PyFunction.java:317)
org.python.core.PyMethod.__call__(PyMethod.java:109)
threading$py._Thread__bootstrap$33(/u01/app/oracle/product/ODI/oracledi_J251/lib/scripting/Lib/threading.py:223)
threading$py.call_function(/u01/app/oracle/product/ODI/oracledi_J251/lib/scripting/Lib/threading.py)
org.python.core.PyTableCode.call(PyTableCode.java:165)
org.python.core.PyBaseCode.call(PyBaseCode.java:301)
org.python.core.PyBaseCode.call(PyBaseCode.java:194)
org.python.core.PyFunction.__call__(PyFunction.java:387)
org.python.core.PyMethod.instancemethod___call__(PyMethod.java:220)
org.python.core.PyMethod.__call__(PyMethod.java:211)
org.python.core.PyMethod.__call__(PyMethod.java:201)
org.python.core.PyMethod.__call__(PyMethod.java:196)
org.python.core.FunctionThread.run(FunctionThread.java:21)


Line 223 in threading.py is the except clause of the attempt at deleting self, when an exception has occurred. So for some reason, the threads do not seem able to self-destruct.

Any help getting this taken care will be greatly appreciated.

----------
components: Core
messages: 6553
nosy: treaves
severity: major
status: open
title: threading memory leak
versions: Deferred

_______________________________________
Jython tracker <re...@bu...>
<http://bugs.jython.org/issue1761>
_______________________________________