Jython

Alex Tichtchenko wrote:
Hello,

I apologize if it is a FAQ -- didn't find anything in the mailing list
archives.

Anybody has any hints on how to control memory allocator/garbage collector
behaviour ? Consider the following bit of code:

#md5 fodder to make it a sensible bit of code :))
import md5

m = md5.new()
f = open('foo', 'rb')
b = f.read(2048)
while b != '':
     m.update(b)
     b = f.read(2048)
print m.hexdigest()

The interesting part here is 'b = f.read()' in the loop. I tried different
approaches, but invariably this bit of code tends to run out of memory
fairly quickly on the moderate size files, say 130Mb.
Actually, the interesting part is the m.update( b ).

 From MD5Object.update(...):

         data += arg.toString();

The md5 object is accumulating the entire string, then the digest method 
calculates the md5sum of it.

This is inherently unscalable.  :-(  It should be filed as a bug.

If you get rid of the m.update(), or use a non-collecting implementation, you 
shouldn't have any problem. (Except, of course, that you said the md5 stuff 
wasn't really your application - I assume whatever you are really wanting to 
do is doing something similar.)

Any hints, ways to kickstart garbage collection cycles, anything -- highly
appreciated.

     rt = Runtime.getRuntime()
     rt.gc()

suggests the interpreter should do a garbage collection.  This won't help you 
here, because m.data == the whole file read to date.

You can also examine total & free memory to watch if something is consuming 
memory:

         print rt.totalMemory(), rt.freeMemory()

kb

Kevin J. Butler wrote:

A very poorly formatted message (Darn the copy & paste in Mozilla mail!).
Sorry about that, here's a better formatting job:

Alex Tichtchenko wrote:
 >
 > Anybody has any hints on how to control memory allocator/garbage collector
 > behaviour ? Consider the following bit of code:
 >
 > #md5 fodder to make it a sensible bit of code :))
 > import md5
 >
 > m = md5.new()
 > f = open('foo', 'rb')
 > b = f.read(2048)
 > while b != '':
 >     m.update(b)
 >     b = f.read(2048)
 > print m.hexdigest()
 >
 > The interesting part here is 'b = f.read()' in the loop. I tried different
 > approaches, but invariably this bit of code tends to run out of memory
 > fairly quickly on the moderate size files, say 130Mb.

Actually, the interesting part is the m.update( b ).

 From MD5Object.update(...):

         data += arg.toString();

The md5 object is accumulating the entire string, then the digest method 
calculates the md5sum of it.

This is inherently unscalable.  :-(  It should be filed as a bug.

If you get rid of the m.update(), or use a non-collecting implementation, you 
shouldn't have any problem. (Except, of course, that you said the md5 stuff 
wasn't really your application - I assume whatever you are really wanting to 
do is doing something similar.)

 > Any hints, ways to kickstart garbage collection cycles, anything -- highly
 > appreciated.

     rt = Runtime.getRuntime()
     rt.gc()

suggests the interpreter should do a garbage collection.  This won't help you 
here, because m.data == the whole file read to date.

You can also examine total & free memory to watch if something is consuming 
memory:

         print rt.totalMemory(), rt.freeMemory()

kb

Wow!

Looks that my md5 sample was really the one in a million :)). I'll check
with gc() and see how this will improve the situation. Still, what is the
semantics of del, besides removing the name from the namespace and
decrementing the counter ? Does it do anything usefull ? Is there any way to
find out default settings for gc() ?

Btw, just out of curiousity, I gave the sample I provided a quick run on my
laptop: 128M ram, good amount of disk free, file size 136,394,407 bytes. The
swap file size jumped up (with no other apps running) to 220Mb and it is
been running for 45 minutes now swapping in and out badly. Still running %)

Thank you,
Alex

----- Original Message -----
From: "Kevin J. Butler" <kevinbutler1@...>
To: "Alex Tichtchenko" <flash@...>
Cc: <jython-users@...>
Sent: Wednesday, June 12, 2002 5:03 PM
Subject: Re: [Jython-users] Memory hogs


> Alex Tichtchenko wrote:
> Hello,
>
> I apologize if it is a FAQ -- didn't find anything in the mailing list
> archives.
>
> Anybody has any hints on how to control memory allocator/garbage collector
> behaviour ? Consider the following bit of code:
>
> #md5 fodder to make it a sensible bit of code :))
> import md5
>
> m = md5.new()
> f = open('foo', 'rb')
> b = f.read(2048)
> while b != '':
>      m.update(b)
>      b = f.read(2048)
> print m.hexdigest()
>
> The interesting part here is 'b = f.read()' in the loop. I tried different
> approaches, but invariably this bit of code tends to run out of memory
> fairly quickly on the moderate size files, say 130Mb.
> Actually, the interesting part is the m.update( b ).
>
>  From MD5Object.update(...):
>
>          data += arg.toString();
>
> The md5 object is accumulating the entire string, then the digest method
> calculates the md5sum of it.
>
> This is inherently unscalable.  :-(  It should be filed as a bug.
>
> If you get rid of the m.update(), or use a non-collecting implementation,
you
> shouldn't have any problem. (Except, of course, that you said the md5
stuff
> wasn't really your application - I assume whatever you are really wanting
to
> do is doing something similar.)
>
> Any hints, ways to kickstart garbage collection cycles, anything -- highly
> appreciated.
>
>      rt = Runtime.getRuntime()
>      rt.gc()
>
> suggests the interpreter should do a garbage collection.  This won't help
you
> here, because m.data == the whole file read to date.
>
> You can also examine total & free memory to watch if something is
consuming
> memory:
>
>          print rt.totalMemory(), rt.freeMemory()
>
> kb
>
>
> _______________________________________________________________
>
> Sponsored by:
> ThinkGeek at http://www.ThinkGeek.com/
> _______________________________________________
> Jython-users mailing list
> Jython-users@...
> https://lists.sourceforge.net/lists/listinfo/jython-users

Alex Tichtchenko wrote:

> Wow!
> 
> Looks that my md5 sample was really the one in a million :)). I'll check
> with gc() and see how this will improve the situation. Still, what is the
> semantics of del, besides removing the name from the namespace and
> decrementing the counter ?


It doesn't even do that.  There is no counter in Java.


 Does it do anything usefull ? Is there any way to
> find out default settings for gc() ?


You can change the amount of memory available for the Java heap with
the -Xmx flag to the java command.  Edit your jython script so it
says something like

java -Xmx128m  ...

That might help you out.  I think the default max heap is pretty small.
However, don't make it too big or the JVM and the OS thrash.


> 
> Btw, just out of curiousity, I gave the sample I provided a quick run on my
> laptop: 128M ram, good amount of disk free, file size 136,394,407 bytes. The
> swap file size jumped up (with no other apps running) to 220Mb and it is
> been running for 45 minutes now swapping in and out badly. Still running %)
> 
> Thank you,
> Alex
> 
> ----- Original Message -----
> From: "Kevin J. Butler" <kevinbutler1@...>
> To: "Alex Tichtchenko" <flash@...>
> Cc: <jython-users@...>
> Sent: Wednesday, June 12, 2002 5:03 PM
> Subject: Re: [Jython-users] Memory hogs
> 
> 
> 
>>Alex Tichtchenko wrote:
>>Hello,
>>
>>I apologize if it is a FAQ -- didn't find anything in the mailing list
>>archives.
>>
>>Anybody has any hints on how to control memory allocator/garbage collector
>>behaviour ? Consider the following bit of code:
>>
>>#md5 fodder to make it a sensible bit of code :))
>>import md5
>>
>>m = md5.new()
>>f = open('foo', 'rb')
>>b = f.read(2048)
>>while b != '':
>>     m.update(b)
>>     b = f.read(2048)
>>print m.hexdigest()
>>
>>The interesting part here is 'b = f.read()' in the loop. I tried different
>>approaches, but invariably this bit of code tends to run out of memory
>>fairly quickly on the moderate size files, say 130Mb.
>>Actually, the interesting part is the m.update( b ).
>>
>> From MD5Object.update(...):
>>
>>         data += arg.toString();
>>
>>The md5 object is accumulating the entire string, then the digest method
>>calculates the md5sum of it.
>>
>>This is inherently unscalable.  :-(  It should be filed as a bug.
>>
>>If you get rid of the m.update(), or use a non-collecting implementation,
>>
> you
> 
>>shouldn't have any problem. (Except, of course, that you said the md5
>>
> stuff
> 
>>wasn't really your application - I assume whatever you are really wanting
>>
> to
> 
>>do is doing something similar.)
>>
>>Any hints, ways to kickstart garbage collection cycles, anything -- highly
>>appreciated.
>>
>>     rt = Runtime.getRuntime()
>>     rt.gc()
>>
>>suggests the interpreter should do a garbage collection.  This won't help
>>
> you
> 
>>here, because m.data == the whole file read to date.
>>
>>You can also examine total & free memory to watch if something is
>>
> consuming
> 
>>memory:
>>
>>         print rt.totalMemory(), rt.freeMemory()
>>
>>kb
>>
>>
>>_______________________________________________________________
>>
>>Sponsored by:
>>ThinkGeek at http://www.ThinkGeek.com/
>>_______________________________________________
>>Jython-users mailing list
>>Jython-users@...
>>https://lists.sourceforge.net/lists/listinfo/jython-users
>>
>

Alex Tichtchenko wrote:
> Looks that my md5 sample was really the one in a million :)). I'll check
> with gc() and see how this will improve the situation. Still, what is the
> semantics of del, besides removing the name from the namespace and
> decrementing the counter ? Does it do anything usefull ? 

In CPython, it removes it from the namespace & decrements the counter.
In Jython, it removes it from the namespace (no counter - pure GC).

That is useful, but not in your case.  :-)  By replacing the value of 'b' each 
iteration, you are already allowing the old value to be garbage collected.

del is also useful to remove class members, dictionary & list entries, etc.

 > Is there any way to
> find out default settings for gc() ?

Some JVMs support some gc debug capability:
   java -verbose:gc

You can consult the JVM docs and java -X and java -h to get some info on 
GC/memory settings.

> Btw, just out of curiousity, I gave the sample I provided a quick run on my
> laptop: 128M ram, good amount of disk free, file size 136,394,407 bytes. The
> swap file size jumped up (with no other apps running) to 220Mb and it is
> been running for 45 minutes now swapping in and out badly. Still running %)

You may also need to set max heap size (-Xmx).

kb