[Wrapper-user] Finding Memory Leaks (Was: Possible Enhancements)

[Leif M. <le...@ta...>]

GMX wrote:

>Leif,
>
>Thanks for the reply, I will certainly looking into this. One issue why I
>was asking for a more external watchdog was that in case the system runs out
>of memory it often has some sort of fit, killing tasks and other nasty
>stuff. I fear that the hook to catch uncought excpetions might be failing as
>well. If you had this in the outside world as part of the watchdog process,
>you would not be affected by this. Does this make sense?
>
Yes, it makes sense.  I am not currently aware of any way to discover 
that a JVM is having
memory exceptions from outside of the JVM.

I think that you should actually be fairly successful at catching out of 
memory errors using the
uncaughtException method in ThreadGroups.  The reason is that your 
application will usually
run out of memory deep inside some method.  When the out of memory error 
is thrown, the
exception gets thrown all the way up the call stack until it is caught 
by the ThreadGroup and
sent to the uncaughtException method.   Each method call on a stack 
takes up memory, so
as the Exception gets cascaded up to the top, memory will be freed up. 
 Should be enough
in most cases to make it possible to call WrapperManager.restart from 
within the
uncaughtException method.

That is basically all that I would be able to do if this was added to 
the Java side of the
Wrapper.

It could be possible to monitor the memory usage of the Java process.   
But because of
the way Java's memory management works, I am not sure how I could tell 
whether Java
is really out of memory.  Java typically allocates more memory than it 
actually needs and
then internally does its own memory management.

The Runtime.totalMemory call returns the amount of memory allocated from 
the system.
Then the Runtime.freeMemory call returns how much of that memory is 
still available.

When freeMemory approaches 0, Java will first invoke GC.  If that does 
not free up very
much memory, then java will allocate more memory from the system.  It 
will stop allocating
memory from the system when it reaches the value set by -mx property set 
when the JVM
was launched.  Note that this defaults to 64MB even if your machine has 
a lot of memory.
You can set this in the wrapper using the wrapper.java.maxmemory property.

I am open to other ideas that you may have, but right now, I am not sure 
how the Wrapper
could do a better job of handling this problem.

>Of course, we have to fix the problem itself, but that is another story :-)
>
As for tracking down your memory problem.  The first thing I would 
suggest is to turn off
the JIT compiler using the -Djava.compiler=NONE property when launching 
the JVM.
(Not sure if that still works with Java 1.4  It does with 1.3 though.   
If your memory leak goes
away then from experience, it is most likely a problem caused by 
synchronization
someplace.   I have had similar problems in the past where JIT compiled 
code behaves
badly when you are having synchronization errors.  Non-JIT compiled code 
will still have
errors.  But the memory leaks and crashes seem to go away.

If that isn't the problem, then you may have an object leak someplace in 
your code.  Add the
following to the wrapper.conf.
wrapper.java.additional.5=-Xrunhprof:depth=8
wrapper.shutdown.timeout=0

The first enables object profiling.  When the JVM is asked to exit, it 
will dump this info to a file.
This can take several minutes depending how large your app is.  To keep 
the Wrapper from
thinking the JVM is hung, you want to disable shutdown timeouts.  Make 
sure to remove both
of these properties when you are done.

When the JVM exits you will see the following in the console or log file.
---
Dumping Java heap ... allocation sites ... done.
---

Look in the directory where Wrapper.exe is located and you will fine a 
file: java.hprof.txt
This file could be quite large, so I hope you have a good editor.

Search for "SITES BEGIN".  You will see a section towards the end of the 
file that looks
like this:
---
SITES BEGIN (ordered by live bytes) Fri Oct 11 12:56:48 2002
          percent         live       alloc'ed  stack class
 rank   self  accum    bytes objs   bytes objs trace name
    1  6.80%  6.80%   544264 4779  544264 4779     0 [C
    2  2.78%  9.59%   222624  964  223280  966  6361 [C
    3  2.14% 11.73%   171248 1262  173504 1299     1 [C
    4  2.12% 13.85%   169368  106  169368  106 17124 [B
    5  2.08% 15.93%   166736  183  166736  183     0 [B
    6  1.71% 17.64%   136624  397  208088  796 17100 [C
    7  1.55% 19.18%   123664  435  123664  435 16788 [C
---
This section is ordered by how much memory is being used by objects which
were created at specific places in the code.  If you are lucky, the 
first row will
be a very large number.
The second to the last column is the Trace number.  This says where the 
objects
 were created.   In the above example this is "4779".
To find out where these [C (Character Arrays) where created, do another 
search
for "TRACE 4779:"
I get the following:
---
TRACE 4779:
    java.lang.StringBuffer.toString(StringBuffer.java:1233)
    
org.apache.catalina.startup.Catalina.createContextCommon(Catalina.java:614)
    
org.apache.catalina.startup.Catalina.createStartMapperDefaultContext(Catalina.java:521)
    
org.apache.catalina.startup.Catalina.createStartMapper(Catalina.java:391)
    org.apache.catalina.startup.Catalina.start(Catalina.java:722)
    org.apache.catalina.startup.Catalina.execute(Catalina.java:681)
    org.apache.catalina.startup.Catalina.process(Catalina.java:179)
    
sun.reflect.NativeMethodAccessorImpl.invoke0(NativeMethodAccessorImpl.java:Native 
method)
---
You may find that you need to increase the stack depth, but that takes 
more memory
and I usually find a depth of 8 to be sufficient.

Hope this helps you,
Cheers,
Leif