Jordi,

First, do you have a reproducible test case? Just to start with something, I wrote a small test case that should get to the heart of your problem. In the following test case, a generator is constructed, used just once, then allowed to eventually garbage collect:

def f():
    for i in xrange(2, 100):
        yield i

j = 0
for i in xrange(100000000):
    j += next(f())
print j

(Code samples here are against Jython 2.7)

For smaller numbers in the generator constructing loop, this test case works fine. But on my laptop, with 100000000 iterations or so, I will get a very busy CPU that won't attach to JMX and eventually 
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space. So that sounds like your problem.

Unfortunately it's also clear that PEP 342 does require the finalization semantics. Because it's about *this* resource, we obviously cannot use a PhantomReference to work around.

One issue to be aware of with respect to finalization: f.close() will be called by a GC thread. In this case, that's no problem, because the support for generators is thread safe, so long as it's not being called by simultaneous threads.

So here's what I would do: I would explicitly close the generator as soon as I know this is feasible. This is similar to other adaptations needed in Jython to move from a deterministic ref-counting GC as seen in CPython to what is necessary on the JVM. Think of closing files. I like contextlib.closing for this purpose:

from contextlib import closing

def f():
    for i in xrange(2, 100):
        yield i

j = 0
for i in xrange(100000000):
    with closing(f()) as g:
        j += next(g)
print j

Hope this helps!

- Jim





On Tue, Aug 13, 2013 at 8:31 AM, Jordi Domingo <jordi.domingo@blueliv.com> wrote:
Hi all,

We are hitting an OOM due to slow finalize in PyGenerator. A heap dump showed that 1G of memory was occupied by finalizers of this class.

As we are using Jython heavily, object destruction is too slow compared with object creation.

Is it possible to not override finalize in PyGenerator to improve GC speed?

Thanks a lot, regards,
Jordi

P.D. We are using jython 2.5.3, but this part of code hasnt changed in any of the newer releases.

------------------------------------------------------------------------------
Get 100% visibility into Java/.NET code with AppDynamics Lite!
It's a free troubleshooting tool designed for production.
Get down to code-level detail for bottlenecks, with <2% overhead.
Download for free and get started troubleshooting in minutes.
http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk
_______________________________________________
Jython-users mailing list
Jython-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/jython-users