Re: [Jython-users] What's the best way to dispose a PythonInterpreter?

[Jim B. <jim...@py...>]

We are going to fix these resource leak issues to the extent possible for
the 2.7.0 release. Problems we have seen now or in the past are as follows:

*Thread locals*. In the case of PythonInterpreter, we only added them with
setLocals, as used by JSR 223, we never removed. Obvious opportunity for a
memory leak! See http://bugs.jython.org/issue2026, which is currently in
progress, include support for try-with-resources as now available in Java
7. So PythonInterpreter#close (which will be a new synonym from
Closeable/Autocloseable for the cleanup method) means cleaning all aspects
of this PythonInterpreter. So Indra, you will get what you want: "prefer
that I could just call cleanup() and remove all references and that would
be it"

There's a more subtle problem associated with thread locals in that they
can also leak ClassLoaders. ThreadState is a good example of where this has
been a problem in the past (http://bugs.jython.org/issue1327). We should
also apply an additional step of indirection, as described in
http://blog.crazybob.org/2006/07/hard-core-java-threadlocal.html

*Caches*. I believe we are mostly pass this as a problem, thanks to using
Google Guava collections and especially its support for caches built using
weakrefs.

*Mutable static fields* holding hard references. PySystemState is
problematic here. We partially addressed this with a shadowing mechanism in
http://hg.python.org/jython/rev/62d5016fe64e, but we need to do a clean
sweep and remove the remainder in favor of making these instance fields, as
well as make PySystemState#shadow a deprecated no-op. Note this will be a
breaking API change, but something we must do for 2.7.

*Subclassed threads*. We see this in http://bugs.jython.org/issue2127 (under
review) and recently fixed in http://hg.python.org/jython/rev/55f7eaddc954 Use
Runnable instead.

None of this is too difficult to implement, assuming we can break APIs, but
it can be hard to duplicate specific leak scenarios. However, I think we
know best practices here and we should just start applying them.

- Jim


On Sat, May 10, 2014 at 1:58 AM, Indra Talip <ind...@gm...> wrote:

> Steve,
> Sadly I don't think that is sufficient. The "sys" module (aka
> PySystemState) exists as a ThreadLocal that the jython modules written in
> java access through Py.getSystemState(). BTW this turns out to be a real
> pain in the neck when you use a thread pool to execute code for different
> PythonInterpreters (it requires me to track the PySystemState that should
> be used for each PythonInterpreter and call Py.setSystemState() with the
> correct PySystemState before invoking any of my python code).
>
> I don't think there is a way via Py.java to clear the thread state (happy
> to be corrected). For my use case I ended up not bothering to throw away
> the PythonInterpreter and attempt to (with some shenanigans related to
> import hooks) delete and reload modules that have changed. I also however
> would prefer that I could just call cleanup() and remove all references and
> that would be it.
>
> Even the (possibly overkill) approach of loading each PythonInterpreter
> into it's own ClassLoader probably won't work/will leak memory at the
> moment due to memory leaks related to subclassing Thread (c.f.
> https://bitbucket.org/jython/jython/pull-request/19/pysystemstateclosershutdowncloser-use
> and http://bugs.jython.org/issue2127).
>
> Does anyone else have any suggestions?
>
> Cheers
> Indra
>
> On 10 May 2014 17:01, Steve Yegge <ste...@gm...> wrote:
>
>> I create PythonInterpreter instances on demand when admins log into my
>> system.  When they leave I'd like to clean up their interpreter instance.
>>
>> Do I just call cleanup() and then remove any references to it?  What
>> exactly does PySystemState.callExitFunc do?  Is it just a customization
>> hook that I can safely assume is null if I didn't set it?
>>
>>
>> ------------------------------------------------------------------------------
>> Is your legacy SCM system holding you back? Join Perforce May 7 to find
>> out:
>> &#149; 3 signs your SCM is hindering your productivity
>> &#149; Requirements for releasing software faster
>> &#149; Expert tips and advice for migrating your SCM now
>> http://p.sf.net/sfu/perforce
>> _______________________________________________
>> Jython-users mailing list
>> Jyt...@li...
>> https://lists.sourceforge.net/lists/listinfo/jython-users
>>
>>
>
>
> --
> Indra Talip
>
>
> ------------------------------------------------------------------------------
> Is your legacy SCM system holding you back? Join Perforce May 7 to find
> out:
> &#149; 3 signs your SCM is hindering your productivity
> &#149; Requirements for releasing software faster
> &#149; Expert tips and advice for migrating your SCM now
> http://p.sf.net/sfu/perforce
> _______________________________________________
> Jython-users mailing list
> Jyt...@li...
> https://lists.sourceforge.net/lists/listinfo/jython-users
>
>