Re: [Jython-users] JPython/Jython for large data sets

[Ben H. <be...@in...>]

Hi Anton,

Anton Gluck wrote:

> I am working on a project for which a data analyzing algorithm was written
> in Java, and wrote a JPython 1.5.2 script to test the algorithm. When
> analyzing large data sets (flat text files) of more than 20, 30 MB we
> originally faced two problems: This was slow, and we ran out of memory
> even if the JVM is allocated the full 256 MB RAM. After rewriting the
> JPython code the out-of-memory problem went away, but presumably the limit
> was just pushed a little further.

I read an interesting article recently about the differences bewteen virtual
machine versions in memory usage behavior. Earlier, non-hotspot VMs throw Out
Of Memory much earlier and easier than hotspot. Hotspot appears to utilize the
JVM process allocated memory more efficiently, so that more of it 90%+ can be
allocated to objects without throwing OutOfMemory, compared to 60-70% on older
VMs. This article was on a public Sun webserver, which I got to from the bug
parade while investigating garbage collection.

So, one piece of advice would be upgrade your VM version if possible.

Ben

--
Ben Hutchison
Software Engineer-Market Predictor
Webmind Australia
http://www.webmind.com/productspredictor.html