Re: [Swig-user] swig + java + gc

[Brian C. <co...@ey...>]

We ran into this problem last week. The best solution we have found so  
far is tuning the garbage collector to work harder using the `- 
XX:GCTimeRatio=<nnn>` option (http://java.sun.com/docs/hotspot/gc5.0/gc_tuning_5.html#0.0.0.0.Ergonomics%20in%20the%20throughput%20collector%7Coutline 
). This with a combination of a better understanding from our users  
about memory issues seems to keep memory consumption in check.

However, this is a really undesirable solution for us as we distribute  
libraries to customers, and thus we don't have control over the  
command line options to the JVM. For the time being there doesn't seem  
to be another way around this problem. Our theory is that Java can  
create objects faster than the garbage collector can clean them up  
again. We think our next step is to see if speeding up the delete  
method will allow the garbage collector to keep pace with applications.

Is the problem rooted in the jni? Or swig's use of the jni? Or is it  
possible to overrun the JVM with straight java code?

-Brian

On Jun 9, 2008, at 1:41 PM, seth lytle wrote:

> i'm doing market data too. by "dispose of the objects myself" do you
> mean thru the finalizer, or directly (eg, by calling
> doubleArray.delete()) ? i don't think i have the option of doing it
> directly, since the user code could still have a reference. assuming
> that you're doing it directly, why did you decide to manage the memory
> yourself instead of using the gc / finalizer ? did you run into
> similar problems ?
>
> seth
>
>
> On Mon, Jun 9, 2008 at 2:55 PM, Lubomir Konstantinov <lu...@gm... 
> > wrote:
>>
>> Hi,
>>
>> I'm using swig in market data applications where the amount of data
>> flowing thru is, well...huge. The memory stays nice and tidy as long
>> as I make sure to dispose of the objects myself, without relying on
>> the gc to clean them up.
>>
>> Cheers,
>> Lubo
>>
>> On Mon, Jun 9, 2008 at 9:40 PM, seth lytle <set...@gm...>  
>> wrote:
>>> i'm using swig to wrap native lapack implementations (mkl and acml).
>>> i've fixed several issues (signal chaining and premature collection)
>>> and have a pretty good system. but i'm struggling to manage the
>>> memory.
>>>
>>> to test, i'm looping over a test suite. each iteration allocates and
>>> sets memory (using cpointer.i and carrays.i) and makes some lapack
>>> calls. no references to previous iterations are preserved.
>>>
>>> if i call System.gc() after each iteration, the vmsize of the app
>>> never seems to grow. but if i don't call the gc(), the vmsize grows
>>> without bound and appears to crash the machine (not sure why the
>>> oom-killer doesn't kill it). and if i call System.gc() every once  
>>> in a
>>> while, the size grows more slowly, but may or may not grow without
>>> bound. and once swig memory has been around for a while (eg hasn't
>>> been reclaimed in it's first 60 seconds), it will never be  
>>> reclaimed.
>>>
>>> it seems that the effectiveness of gc() at reclaiming the swig
>>> resources is dependent on how much java memory allocation has been
>>> done relative to the swig / native allocation. i've tried a number  
>>> of
>>> work-arounds.
>>>
>>> - "instrument" the user code. call gc() frequently in the same  
>>> thread
>>> as the user code. this works fine and by "timing" it right i can
>>> achieve very slow vmsize growth without slowing things down much.  
>>> but
>>> this needs to be transparent to the user (i'm writing a library  
>>> for a
>>> scripting environment), so it means i'd need to modify many of the
>>> entry or exit points and make sure the gc() was run if needed. this
>>> solution is effective but messy, and just seems wrong.
>>>
>>> - call gc() from another thread. i've tried this before and found
>>> operating points at which it broke down, but right now it seems work
>>> ok. vmsize grows but seems to stabilize.
>>>
>>> - allocate java memory and call gc() from another thread. more
>>> consistent but slower than just calling gc(). sensitive to java max
>>> heap size.
>>>
>>> i'm seeing a few crashes with the latter two techniques unless i use
>>> locking (in which case i might as well use the first method). i  
>>> assume
>>> that this is because i still need to clean up a bunch of premature
>>> collections. i'm also using openmp (well, mkl and acml are) - not  
>>> sure
>>> if this conflicts with using multiple threads in java (sure hope  
>>> not).
>>> if i'm still confused after doing some investigation, i'll post a
>>> separate note about openmp.
>>>
>>>
>>> wondering if others have seen similar behavior (or it's absence)  
>>> when
>>> allocating lots of memory on the swig side of the jni. i'm doing  
>>> 10k -
>>> 50 meg per iteration, and very little non-jni allocation.
>>>
>>>
>>>
>>> notes:
>>> two test machines:
>>> intel 32-bit, mkl, 1.5G, java 5 + 6, linux 2.6.24, ubuntu 8.04
>>> amd 64-bit, acml, 8G, java 5, linux 2.6.15, opensuse 10.0
>>> measuring vmsize using "cat /proc/stat" and using the vmsize entry
>>> interface files are attached
>>>
>>> -------------------------------------------------------------------------
>>> Check out the new SourceForge.net Marketplace.
>>> It's the best place to buy or sell services for
>>> just about anything Open Source.
>>> http://sourceforge.net/services/buy/index.php
>>> _______________________________________________
>>> Swig-user mailing list
>>> Swi...@li...
>>> https://lists.sourceforge.net/lists/listinfo/swig-user
>>>
>>>
>
> -------------------------------------------------------------------------
> Check out the new SourceForge.net Marketplace.
> It's the best place to buy or sell services for
> just about anything Open Source.
> http://sourceforge.net/services/buy/index.php
> _______________________________________________
> Swig-user mailing list
> Swi...@li...
> https://lists.sourceforge.net/lists/listinfo/swig-user