Stefan,

Rather than monkeypatching a custom _weakref module from JyNI (solution #2), it would seem better if we developed a better API, much as I tried to do with the callback scheme in my previous email. This would then let you build an efficient mechanism for your solution #1 and without any problematic "tweaks".

But in doing so, we can also revisit the old scheme of the private static field references being an ArrayList in _weakref.GlobalRef, and the management of AbstractRef objects in it. There easily seem to be additional O(n) and O(n^2) factors in this code that we could eliminate, not to mention possibly also removing the synchronized access, while also providing support for your needs. In part, I'm pretty sure this is due to the fact that weakref was apparently part of Jython 2.1 (anyone want to verify this?), and we really haven't looked at its performance in subsequent releases, just correctness. (I even touched it at one point to change references from Vector to ArrayList, but I did so as part of a general refactoring of the Jython codebase at the end of 2008.)

In general, I'm a big fan of Google Guava collections, which we already use extensively in Jython's implementation; I'm pretty sure we can find a better mapping.

I'm on vacation today through the weekend, so I'm sure I'll have more analysis next week. Looking forward to a good conversation.

- Jim



On Fri, May 30, 2014 at 1:01 AM, Stefan Richthofer <Stefan.Richthofer@gmx.de> wrote:
Hello Jim,
thanks for the response. I also thought about using weakref's callback-mechanism. The problem here is that callbacks are processed when the referent is collected. However this is too late for my purpose. Without the adjustment, the PyCPeers (if in the role of referents) might be collected too early, i.e. when the native object is still available in fact (but kept alive by native pointers). So my adjustment must take place before the CPeer is collected, optimally directly when a weak reference is created. I also thought about reviving the weakref (i.e. GlobalRef) on AbstractReference-level once the callback is received. Problem here is that other callbacks might be registered for the CPeer. These would be called by the RefReaperThread equally to my own callback, receive wrong alerts and do all sorts of cleanup too early. I don't see a good way to tell the RefReaperThread which callbacks to call and which not. So I would prefere solution #1. I am still keen on understanding why #2 fails. Any clue?
 
- Stefan
 
Gesendet: Freitag, 30. Mai 2014 um 00:59 Uhr
Von: "Jim Baker" <jim.baker@python.org>
An: "Stefan Richthofer" <Stefan.Richthofer@gmx.de>
Cc: "Jython Developers" <jython-dev@lists.sourceforge.net>
Betreff: Re: [Jython-dev] weak reference support in JyNI
Stefan,
 
Glad to hear of the progress on JyNI. Getting weakrefs working is going to be essential to JyNI supporting the CPython memory model in conjunction with Java's.
 
We likely need another solution, one that's closer to approach #0, but without the poll. Basically it should be possible for using code to register callbacks, with RefReaperThread calling each of the registered callbacks for each GlobalRef object. A callback could enqueue (onto a LinkedBlockingQueue for example) or do some other processing. This should be general enough for your management purposes.
 
Note that the current implementation causes a recurring issue we have seen in how threads can interact with class loaders: http://bugs.jython.org/issue2127 This simply means we can fix this problem in revisiting this support!
 
Interestingly, Jiwon Seo requested a somewhat similar notification mechanism here, but with respect to generated bytecode, https://bitbucket.org/jython/jython/pull-request/41/adding-bytecode-notification-mechanism This makes sense to me - we are now trying to build more sophisticated integrations with Jython. (Hopefully we will get that merged in soon, sorry that has taken so long, Jiwon!)
 
- Jim
 
 
On Thu, May 29, 2014 at 7:17 AM, Stefan Richthofer <Stefan.Richthofer@gmx.de> wrote:
Hello,

I am currently working out the support of garbage collection and weak references in JyNI. For weak references, I would use the _weakref module that is already included in Jython, but I need to tweak it a bit, because JyNI needs to know, whether a PyCPeer (i.e. a subclass of PyObject that JyNI uses to wrap native objects in some cases) is weakly referenced. The reason for this is - roughly speaking - that in case of weak references, the native object should keep the peer alive (i.e. use a global ref in JNI), while normal references should use a peer that keeps the native object alive and not vise versa (i.e. use a weak global ref in JNI). So there will be potentially 2 versions of PyCPeer having the keep-alive relation
PyCPeer ---keeps alive---> native object ---keeps alive---> PyCPeer
Normal references should refere to the leftmost one while weak references should refere to the rightmost one (the python level equality function will report them as equal).

Of course the default implementation of _weakref is not aware of this. So I made up roughly two (three) competing ideas how to do it.

0) The official way to get those weak references that point to a specific object is WeakrefModule.getweakrefs(...)
However I would have to poll that method, which is an unprecise and unefficient solution. (So I don't count it as solution)
Once a new weak reference to a PyCPeer would show up, I would replace its referent by the right version. Since GlobalRef extends java.lang.ref.WeakReference, which does not allow to modify the referent, this would involve creating a new GlobalRef and replacing the old one. This is rather tweaky since that is stored in a private field (i.e. I would use reflection with setAccessible(true); or let native code do the operation).

1) Use org.python.modules._weakref.GlobalRef.references to keep track.
I would write a List implementation that wraps some other list as backend but reports modifications to a listener. I would use the original org.python.modules._weakref.GlobalRef.references as backend and insert my custom list into the org.python.modules._weakref.GlobalRef.references-field. Since it is a private field, this would be tweaky again, but could be done the same way as mentioned above. The rest would be like in "solution" 0) but the polling would be avoided.

2) Write an adjusted version of the _weakref-module (i.e. that checks for PyCPeer BEFORE creating GlobalRefs - so it works completely without fumbling with private fields and avoids useless recreation of GlobalRefs).
I tried this first, because I consider it the least tweaky solution, but ran into problems I can't resolve. For testing and a proof of concept, I took the _weakref source code, put it into a new package JyNI._weakref and only modified the doc-string to be able to distinguish the modules.

Then I adjusted the JyNI-initializer as follows:

public void initialize(Properties preProperties, Properties postProperties, String[] argv, ClassLoader classLoader, ExtensiblePyObjectAdapter adapter)
{
                //Customize the _weakref module to JyNI-needs:
                String blti = (String) postProperties.get("python.modules.builtin");
                postProperties.put("python.modules.builtin", blti == null ? "_weakref:JyNI._weakref.WeakrefModule" : blti+",_weakref:JyNI._weakref.WeakrefModule");

                //init PySystemState:
                PySystemState initState = PySystemState.doInitialize(preProperties, postProperties, argv, classLoader, adapter);

//further initialization stuff...


This would overwrite the _weakref-builtin with my custom version. If I do

import _weakref
print _weakref.__doc__

it works nicely and indeed prints the doc-string of my custom module.
But if I do

from _weakref import ref
print ref
bla = "bla"
test = ref(bla)
print test

original Jython outputs:

<type 'weakref'>
<weakref at 0x2; to 'str' at 0x3>

(i.e. it works fine), while the output using my initializer is:

<type 'JyNI._weakref.ReferenceType'>
Traceback (most recent call last):
  File "/home/stefan/eclipseWorkspace/JyNI/JyNI-Demo/src/JyNIWeakRefTest.py", line 41, in <module>
    test = ref(bla)
TypeError: JyNI._weakref.ReferenceType(): expected 2-3 args; got 1


So obviously, my tweak fails to make up the right type for ref and yields no appropriate mro. However I have no clue how this comes, since Jython should initialize the custom module just the same way it would do with or.python.modules._weakref.WeakrefModule - it is absolutely identical apart from the doc-string and the package path (but that is adjusted in the properties).

Can someone tell me why approach 2) fails and maybe how to fix it? Or should I work out approach 1) instead? Or do you even have a better idea how to reach my goal?

Thanks in advance.

-Stefan

------------------------------------------------------------------------------
Time is money. Stop wasting it! Get your web API in 5 minutes.
www.restlet.com/download
http://p.sf.net/sfu/restlet
_______________________________________________
Jython-dev mailing list
Jython-dev@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/jython-dev
------------------------------------------------------------------------------ Time is money. Stop wasting it! Get your web API in 5 minutes. www.restlet.com/download http://p.sf.net/sfu/restlet_______________________________________________ Jython-dev mailing list Jython-dev@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/jython-dev