jython-dev

[Jython-dev] Problems with weak references?

[<Ric...@i2...>]

Hi,

I've been playing with the new python.options.internalTablesImpl option.
I'm noticing some oddities when I set this to "weak".

Every once in a while I get the following error when I start up the
Jython interpreter:

Jython 2.0rc1 on java1.3.0_01 (JIT: null)
error importing site
Traceback (innermost last):
  File "/home/rseddon/pnp/3rdparty/jython/Lib/site.py", line 68, in ?
  File "/home/rseddon/pnp/3rdparty/jython/Lib/javaos.py", line 6, in ?
NameError: OSError

This doesn't happen very often, and I haven't found any way to
reproduce it consistently.

Is it possible that the table holding this is being garbage collected?
I never see this error when I'm not using the weak table implementation.

Also, it seems that the PyBeanEventProperty class is has a memory
leak. Previously I had made a local patch to it, replacing it's
HashMap's with WeakHasMaps. I verified that garbage collection was
occurring by means of debugging print statements in the "finalize" method
of some of my Java base classes. These same statements are never
executed when using the new weak reference implementation.

I can't see anything wrong with the new code, but it seems
fairly certain that it isn't working When I switch back to my
"WeakHashMap" patch I see that they are executed.

Of course, it's difficult to _prove_ anything here, since I don't have
much control over when garbage collection occurs. But things do look
a bit suspicious. Has anyone else been trying out the weak reference
implementation?

Rich

Re: [Jython-dev] Problems with weak references?

[<bc...@wo...>]

[Rich]

>I've been playing with the new python.options.internalTablesImpl option.
>I'm noticing some oddities when I set this to "weak".
>
>Every once in a while I get the following error when I start up the
>Jython interpreter:
>
>Jython 2.0rc1 on java1.3.0_01 (JIT: null)
>error importing site
>Traceback (innermost last):
>  File "/home/rseddon/pnp/3rdparty/jython/Lib/site.py", line 68, in ?
>  File "/home/rseddon/pnp/3rdparty/jython/Lib/javaos.py", line 6, in ?
>NameError: OSError
>
>This doesn't happen very often, and I haven't found any way to
>reproduce it consistently.

Luckily <wink> it happens fairly often on my 300Mhz win2k & jdk1.3.

The situation where it happens for me, seems to be because
PyJavaClass(__builtin__) gets booted from the internal tables.
When that happens, all the assignments done in Py.setBuiltinExceptions()
is lost. The booting is done during PySystemState.initPackages, which is
before the default PySystemState instance is create. When that instance
is created, it will keep the __builtin__ dict alive.

Storing a reference of the PyJavaClass(__builtin__) in Py appears to
close the hole.

    private static PyJavaClass v;

    public static void setBuiltinExceptions() {
        v = PyJavaClass.lookup(org.python.core.__builtin__.class);
        PyObject dict = v.__getattr__("__dict__");


Since the the builtin exceptions are stored in PyJavaClass.__dict__,
these bindings are not recreated when the PyJavaClass(__builtin__) is
recreated.

I think that could explain it. Do you agree Samuele?

regards,
finn

[Jython-dev] Mirror possible?

[Mats W. <ma...@la...>]

Is it possible to mirror the last release or two
someplace other than Source Forge?  I can't download
from there unless I get extremely lucky. I don't
know if I have worse paths than anyone else to the
admittedly inadequate sourceforge download server... I
have not been able to download the release candidate file,
which has failed a half-dozen times now.  The average
transfer rate - before things just give up - hovers
around 0.2k/sec, which would put it at a 2.5 hour
download, if it actually completed!  (eventually it
just gives up with no messages)

Mats

Re: [Jython-dev] Problems with weak references?

[Samuele P. <pe...@in...>]

Hi.

[Finn]
>Luckily <wink> it happens fairly often on my 300Mhz win2k & jdk1.3.
>
>The situation where it happens for me, seems to be because
>PyJavaClass(__builtin__) gets booted from the internal tables.
>When that happens, all the assignments done in Py.setBuiltinExceptions()
>is lost. The booting is done during PySystemState.initPackages, which is
>before the default PySystemState instance is create. When that instance
>is created, it will keep the __builtin__ dict alive.
>
>Storing a reference of the PyJavaClass(__builtin__) in Py appears to
>close the hole.
>
>    private static PyJavaClass v;
>
>    public static void setBuiltinExceptions() {
>       v = PyJavaClass.lookup(org.python.core.__builtin__.class);
>        PyObject dict = v.__getattr__("__dict__");
>
>
>Since the the builtin exceptions are stored in PyJavaClass.__dict__,
>these bindings are not recreated when the PyJavaClass(__builtin__) is
>recreated.
>
>I think that could explain it. Do you agree Samuele?
Yes, if an instance of PyJavaClass is kept alive in the non-weak case just by
the internal tables, it could
be garbage collected when weak internal tables are turned on.
The fix seems fine up to the field name v <wink>.
We should check if something similar is happening for other classes too. I
regret not having done such a check before.

[Rich]
> Also, it seems that the PyBeanEventProperty class is has a memory
> leak. Previously I had made a local patch to it, replacing it's
> HashMap's with WeakHasMaps. I verified that garbage collection was
> occurring by means of debugging print statements in the "finalize" method
> of some of my Java base classes. These same statements are never
> executed when using the new weak reference implementation.
>
> I can't see anything wrong with the new code, but it seems
> fairly certain that it isn't working When I switch back to my
> "WeakHashMap" patch I see that they are executed.
>
> Of course, it's difficult to _prove_ anything here, since I don't have
> much control over when garbage collection occurs. But things do look
> a bit suspicious. Has anyone else been trying out the weak reference
> implementation?
Are the adapters that do not get garbage collected (this should work also
without the weak option,
under java2 the adapters table is always implemented as a WeakHashTable)
or the adaptersClasses?

At least for adapters the following example shows that (at least in principle)
they are garbage collected:
Note: itd.py is the module that I used to debug the new internal tables,
you can use it to track the problem (that would be nice <wink>).
(I already tried that before submitting the new table design).
I have experience that also classes (not tried for adaptersClasses) in the main
table get properly discarded.

Jython 2.0beta2 on java1.3.0 (JIT: null)
Type "copyright", "credits" or "license" for more information.
>>> from java import awt,lang
>>> import itd
>>> itd.ac() # lists the "listeners" for which an adapter-class exists and is
hold in the tbl
[]
>>> itd.ad() # lists the adpter instances (truly the for each one the
"listener"
 name is shown)
[]
>>> b=awt.Button() # this alone does not create an adapter
>>> # we force the creation of an adapter
>>> b.actionPerformed
<CompoundCallable with 0 callables>
>>> itd.ac(),itd.ad()
(['java.awt.event.ActionListener'], ['java.awt.event.ActionListener'])
>>> del b
>>> lang.System.gc() # we force gc
>>> itd.ac(),itd.ad() # the adapter should have been gced
(['java.awt.event.ActionListener'], [])
Let me know.

regards, Samuele Pedroni.

<itd.py>
# Copyright 2000 Samuele Pedroni
from java.lang import Class
from org.python.core import PyJavaClass as JC

t=JC.getInternalTables()

def lazy():
 r = []
 t._beginLazyCanonical()
 while 1:
  x=t._next()
  if not x: break
  r.append(x.name)
 return r

def canon():
 r = []
 t._beginCanonical()
 while 1:
  x=t._next()
  if not x: break
  r.append(Class.getName(x))
 return r

def ac():
 r = []
 t._beginOverAdapterClasses()
 while 1:
  x=t._next()
  if not x: break
  r.append(Class.getName(x))
 return r

def ad():
 r = []
 t._beginOverAdapters()
 while 1:
  x=t._next()
  if not x: break
  r.append(Class.getName(x))
 return r

Re: [Jython-dev] Problems with weak references?

[Samuele P. <pe...@in...>]

Hi.

First of all it seems (but maybe I'm missing something) that java.awt leaks
memory too or at least
to discard things at non-predictable points:
>>> C:\WINDOWS>\jython\mjyth
Jython 2.0beta2 on java1.3.0 (JIT: null)
Type "copyright", "credits" or "license" for more information.
>>> from java.lang import System,ref
>>> from java import awt
>>> f=awt.Frame()
>>> b=awt.Button()
>>> f.add(b);f.pack();f.show()
java.awt.Button[button0,0,0,0x0,invalid,label=]
>>> 0 # clears _
0
>>> w=ref.WeakReference(b)
>>> del b
>>> f.remove(w.get())
>>> System.gc(); System.gc()
>>> w.get() # leak? why?
java.awt.Button[button0,4,28,136x23,label=]
>>> 0
0
>>> f.dispose()
>>> del f
>>> System.gc()
>>> w.get()
>>>
f.remove() + System.gc() seems not enough in order to discard b?
This is related to the tables because as long as b is around, its adapters will
be too.

Indipendently of this: the table can leak memory in some cases (it should be
possible to fix this but after 2.0):
Jython 2.0beta2 on java1.3.0 (JIT: null)
Type "copyright", "credits" or "license" for more information.
>>> from java import awt
>>> import itd
>>> from java.lang import System
>>> b=awt.Button(); b.actionPerformed = lambda e: b
>>> itd.ad()
['java.awt.event.ActionListener']
>>> del b
>>> System.gc()
>>> itd.ad()
[]
>>> # no leaks with this code but
>>> b=awt.Button(); b.actionPerformed = lambda e,me=b: None
>>> itd.ad()
['java.awt.event.ActionListener']
>>> del b
>>> System.gc()
>>> itd.ad()
['java.awt.event.ActionListener']
>>> # so we get a memory leak
In this case the listener function hold a ref to the source object, to solve
this the table should (carefully) use weak-refs for values
too. In the meantime this kind of code can be avoided given that events come
with source info.

regards, Samuele Pedroni.

Re: [Jython-dev] Problems with weak references?

[Samuele P. <pe...@in...>]

Hi.

[me]
>....
> In this case the listener function hold a ref to the source object, to solve
> this the table should (carefully) use weak-refs for values
> too. In the meantime this kind of code can be avoided given that events come
> with source info.
>
In a private discussion with Rich and Finn the following type of code has been
recalled:

class Button1(java.awt.Button):
  def __init__(self):
    self.actionPerformed = self.action

  def action(self):
    pass

b=Button1()

More complex components behaviour is customized this way - very often  I
imagine. It's a natural style that I used too, and I just forgot about it.
Clearly setting
self.actionPerformed to a bound method (contains a ref to self) prevent gc as
in the example I previously presented.
Asking to avoid this kind of coding is asking too much. So I have committed a
patch along my proposal, that allows b to be collected.

Important things:
- WeakHashMap use for adapters was and now the new patch is always enabled
under java2 (no need to set internalTablesImpl for that - this option is really
only related to class unloading, not adapters gc).
- b would remain stuck in memory (no gc) also under JPython 1.1 and this will
continue to happen also with jython under java 1.1 - I see no way to avoid
that.

regards, Samuele Pedroni.