[Jython-users] Fwd: What's the best way to dispose a PythonInterpreter?

[Jim B. <jim...@py...>]

reply to list

---------- Forwarded message ----------
From: Jim Baker <jim...@py...>
Date: Fri, Jun 20, 2014 at 11:23 AM
Subject: Re: [Jython-users] What's the best way to dispose a
PythonInterpreter?
To: Steve Yegge <ste...@gm...>
Cc: Indra Talip <ind...@gm...>


Just to be clear, by broken APIs I mean having to use getters/setters in
the Java API and not directly access via Java fields, at least with respect
to static fields.

Example: use PySystemState#setDefaultEncoding, instead of manipulating
defaultEncoding, because such fields are going to become private and
nonstatic. This is a good example where we were perhaps too conservative in
our API changes and implemented a complex "shadowing" mechanism to support
both old and new style usage together.

So nothing controversial in this refactoring, just good coding practice,
and the sort of thing that users of the Java API can adjust to in a few
minutes of work, or possibly just recompile. Users from Python will of
course not even see a difference.

- Jim


On Fri, Jun 20, 2014 at 1:11 AM, Steve Yegge <ste...@gm...> wrote:

> Thanks for the update, Jim.  I don't mind broken APIs to get this
> functionality.  Looking forward to it.
>
>
> On Thu, Jun 19, 2014 at 4:32 PM, Jim Baker <jim...@py...> wrote:
>
>> We are going to fix these resource leak issues to the extent possible for
>> the 2.7.0 release. Problems we have seen now or in the past are as follows:
>>
>> *Thread locals*. In the case of PythonInterpreter, we only added them
>> with setLocals, as used by JSR 223, we never removed. Obvious opportunity
>> for a memory leak! See http://bugs.jython.org/issue2026, which is
>> currently in progress, include support for try-with-resources as now
>> available in Java 7. So PythonInterpreter#close (which will be a new
>> synonym from Closeable/Autocloseable for the cleanup method) means cleaning
>> all aspects of this PythonInterpreter. So Indra, you will get what you
>> want: "prefer that I could just call cleanup() and remove all references
>> and that would be it"
>>
>> There's a more subtle problem associated with thread locals in that they
>> can also leak ClassLoaders. ThreadState is a good example of where this has
>> been a problem in the past (http://bugs.jython.org/issue1327). We should
>> also apply an additional step of indirection, as described in
>> http://blog.crazybob.org/2006/07/hard-core-java-threadlocal.html
>>
>> *Caches*. I believe we are mostly pass this as a problem, thanks to
>> using Google Guava collections and especially its support for caches built
>> using weakrefs.
>>
>> *Mutable static fields* holding hard references. PySystemState is
>> problematic here. We partially addressed this with a shadowing mechanism in
>> http://hg.python.org/jython/rev/62d5016fe64e, but we need to do a clean
>> sweep and remove the remainder in favor of making these instance fields, as
>> well as make PySystemState#shadow a deprecated no-op. Note this will be a
>> breaking API change, but something we must do for 2.7.
>>
>> *Subclassed threads*. We see this in http://bugs.jython.org/issue2127 (under
>> review) and recently fixed in
>> http://hg.python.org/jython/rev/55f7eaddc954 Use Runnable instead.
>>
>> None of this is too difficult to implement, assuming we can break APIs,
>> but it can be hard to duplicate specific leak scenarios. However, I think
>> we know best practices here and we should just start applying them.
>>
>> - Jim
>>
>>
>> On Sat, May 10, 2014 at 1:58 AM, Indra Talip <ind...@gm...>
>> wrote:
>>
>>> Steve,
>>> Sadly I don't think that is sufficient. The "sys" module (aka
>>> PySystemState) exists as a ThreadLocal that the jython modules written in
>>> java access through Py.getSystemState(). BTW this turns out to be a real
>>> pain in the neck when you use a thread pool to execute code for different
>>> PythonInterpreters (it requires me to track the PySystemState that should
>>> be used for each PythonInterpreter and call Py.setSystemState() with the
>>> correct PySystemState before invoking any of my python code).
>>>
>>> I don't think there is a way via Py.java to clear the thread state
>>> (happy to be corrected). For my use case I ended up not bothering to throw
>>> away the PythonInterpreter and attempt to (with some shenanigans related to
>>> import hooks) delete and reload modules that have changed. I also however
>>> would prefer that I could just call cleanup() and remove all references and
>>> that would be it.
>>>
>>> Even the (possibly overkill) approach of loading each PythonInterpreter
>>> into it's own ClassLoader probably won't work/will leak memory at the
>>> moment due to memory leaks related to subclassing Thread (c.f.
>>> https://bitbucket.org/jython/jython/pull-request/19/pysystemstateclosershutdowncloser-use
>>> and http://bugs.jython.org/issue2127).
>>>
>>> Does anyone else have any suggestions?
>>>
>>> Cheers
>>> Indra
>>>
>>> On 10 May 2014 17:01, Steve Yegge <ste...@gm...> wrote:
>>>
>>>> I create PythonInterpreter instances on demand when admins log into my
>>>> system.  When they leave I'd like to clean up their interpreter instance.
>>>>
>>>> Do I just call cleanup() and then remove any references to it?  What
>>>> exactly does PySystemState.callExitFunc do?  Is it just a customization
>>>> hook that I can safely assume is null if I didn't set it?
>>>>
>>>>
>>>> ------------------------------------------------------------------------------
>>>> Is your legacy SCM system holding you back? Join Perforce May 7 to find
>>>> out:
>>>> &#149; 3 signs your SCM is hindering your productivity
>>>> &#149; Requirements for releasing software faster
>>>> &#149; Expert tips and advice for migrating your SCM now
>>>> http://p.sf.net/sfu/perforce
>>>> _______________________________________________
>>>> Jython-users mailing list
>>>> Jyt...@li...
>>>> https://lists.sourceforge.net/lists/listinfo/jython-users
>>>>
>>>>
>>>
>>>
>>> --
>>> Indra Talip
>>>
>>>
>>> ------------------------------------------------------------------------------
>>> Is your legacy SCM system holding you back? Join Perforce May 7 to find
>>> out:
>>> &#149; 3 signs your SCM is hindering your productivity
>>> &#149; Requirements for releasing software faster
>>> &#149; Expert tips and advice for migrating your SCM now
>>> http://p.sf.net/sfu/perforce
>>> _______________________________________________
>>> Jython-users mailing list
>>> Jyt...@li...
>>> https://lists.sourceforge.net/lists/listinfo/jython-users
>>>
>>>
>>
>