Menu
▾
▴
wrapper-user
|
From: GMX <lg...@gm...> - 2002-10-09 02:51:49
|
Hi, I would like to ask if it is possible to add some more functionality to the wrapper code. First, the ability to detect certain exceptions and as a result restart the machine. Mainly I am thinking about a OutOfMemoryException, which can occur if a system runs for a long time and somehow the code does not clear out its resources. Or a slight memory leak, which of course should be fixed in the first place but with that extra functionality in the wrapper it would mean to have another countermeasure against dreadful bugs. Second, and this is along the same line, monitoring the memory usage. Say you have set the maximum memory to 128Mb, it would then be good to set some sort of threshold to monitor the garbage collection and its effectiveness. In other words, when the memory increases above a certain limit and then stays there for a given amount of time, then restart the app. This is for the same reasons as above. Usually the garbage collector thread should free enough resources ones they are filled up, but when buggy code keeps eating resources, the garbage collector can only free a small percentage each time, even when running full garbage collections. As an example the minimum used memory limit that triggers this watchdog could be set to 112Mb and the time limit to 15 mins. If the memory usage does not drop below 112Mb within 15 mins from the time it got over that memory usage level, then restart the app. What I do not know without diving head on into the code is of this sort of thing is possible with the way wrapper is working and if that information can be reliably gathered using C or Java code. I hope I explained this thoroughly enough. Any help/hint/idea is highly appreciated. Keep up the good work! Kind regards, Lars |
|
From: Leif M. <le...@ta...> - 2002-10-10 03:50:01
|
Lars,
The features that you requested below are all things that you can
build into your
application. There is nothing there that should require any special code
on the Wrapper
level.
You can catch uncaught exceptions by extending the ThreadGroup class
and overriding
the uncaughtException(thread, throwable) method. You then launch the
threads of your
application within this thread group.
It would be possible to do this in the Wrapper by launching the
applications within a
custom WrapperThreadGroup. But I am wary of doing that as it would
likely cause problems
with some applications. A lot of server apps which are using the
Wrapper make heavy use
of thread groups. Most likely it would not cause any problems (??) But
I would not be able
to catch any exceptions from threads created by other thread groups, so
the feature would
not work consistently.
The second request about memory monitoring can be handled very
easily by your application.
(The best option would be to fix the leak :-) ) But to implement this.
Simply monitor the memory
using using the following:
Runtime r = Runtime.getRuntime();
if ( r.totalMemory() - r.freeMemory() > 128 * 1024 * 1024 )
{
WrapperManager.restart();
// Will not get here.
}
Just put that code in a daemon thread and let it run.
Cheers,
Leif
GMX wrote:
>Hi,
>
>I would like to ask if it is possible to add some more functionality to the
>wrapper code.
>
>First, the ability to detect certain exceptions and as a result restart the
>machine. Mainly I am thinking about a OutOfMemoryException, which can occur
>if a system runs for a long time and somehow the code does not clear out its
>resources. Or a slight memory leak, which of course should be fixed in the
>first place but with that extra functionality in the wrapper it would mean
>to have another countermeasure against dreadful bugs.
>
>Second, and this is along the same line, monitoring the memory usage. Say
>you have set the maximum memory to 128Mb, it would then be good to set some
>sort of threshold to monitor the garbage collection and its effectiveness.
>In other words, when the memory increases above a certain limit and then
>stays there for a given amount of time, then restart the app. This is for
>the same reasons as above. Usually the garbage collector thread should free
>enough resources ones they are filled up, but when buggy code keeps eating
>resources, the garbage collector can only free a small percentage each time,
>even when running full garbage collections. As an example the minimum used
>memory limit that triggers this watchdog could be set to 112Mb and the time
>limit to 15 mins. If the memory usage does not drop below 112Mb within 15
>mins from the time it got over that memory usage level, then restart the
>app.
>
>What I do not know without diving head on into the code is of this sort of
>thing is possible with the way wrapper is working and if that information
>can be reliably gathered using C or Java code.
>
>I hope I explained this thoroughly enough. Any help/hint/idea is highly
>appreciated. Keep up the good work!
>
>Kind regards,
>Lars
>
>
>
>-------------------------------------------------------
>This sf.net email is sponsored by:ThinkGeek
>Welcome to geek heaven.
>http://thinkgeek.com/sf
>_______________________________________________
>Wrapper-user mailing list
>Wra...@li...
>https://lists.sourceforge.net/lists/listinfo/wrapper-user
>
>
>
|
|
From: GMX <lg...@gm...> - 2002-10-11 02:47:18
|
Leif,
Thanks for the reply, I will certainly looking into this. One issue why I
was asking for a more external watchdog was that in case the system runs out
of memory it often has some sort of fit, killing tasks and other nasty
stuff. I fear that the hook to catch uncought excpetions might be failing as
well. If you had this in the outside world as part of the watchdog process,
you would not be affected by this. Does this make sense?
Of course, we have to fix the problem itself, but that is another story :-)
Thanks,
Lars
----- Original Message -----
From: "Leif Mortenson" <le...@ta...>
To: <wra...@li...>
Sent: Thursday, October 10, 2002 1:49 PM
Subject: Re: [Wrapper-user] Possible Enhancements
> Lars,
> The features that you requested below are all things that you can
> build into your
> application. There is nothing there that should require any special code
> on the Wrapper
> level.
>
> You can catch uncaught exceptions by extending the ThreadGroup class
> and overriding
> the uncaughtException(thread, throwable) method. You then launch the
> threads of your
> application within this thread group.
> It would be possible to do this in the Wrapper by launching the
> applications within a
> custom WrapperThreadGroup. But I am wary of doing that as it would
> likely cause problems
> with some applications. A lot of server apps which are using the
> Wrapper make heavy use
> of thread groups. Most likely it would not cause any problems (??) But
> I would not be able
> to catch any exceptions from threads created by other thread groups, so
> the feature would
> not work consistently.
>
> The second request about memory monitoring can be handled very
> easily by your application.
> (The best option would be to fix the leak :-) ) But to implement this.
> Simply monitor the memory
> using using the following:
>
> Runtime r = Runtime.getRuntime();
> if ( r.totalMemory() - r.freeMemory() > 128 * 1024 * 1024 )
> {
> WrapperManager.restart();
> // Will not get here.
> }
>
> Just put that code in a daemon thread and let it run.
>
> Cheers,
> Leif
>
>
> GMX wrote:
>
> >Hi,
> >
> >I would like to ask if it is possible to add some more functionality to
the
> >wrapper code.
> >
> >First, the ability to detect certain exceptions and as a result restart
the
> >machine. Mainly I am thinking about a OutOfMemoryException, which can
occur
> >if a system runs for a long time and somehow the code does not clear out
its
> >resources. Or a slight memory leak, which of course should be fixed in
the
> >first place but with that extra functionality in the wrapper it would
mean
> >to have another countermeasure against dreadful bugs.
> >
> >Second, and this is along the same line, monitoring the memory usage. Say
> >you have set the maximum memory to 128Mb, it would then be good to set
some
> >sort of threshold to monitor the garbage collection and its
effectiveness.
> >In other words, when the memory increases above a certain limit and then
> >stays there for a given amount of time, then restart the app. This is for
> >the same reasons as above. Usually the garbage collector thread should
free
> >enough resources ones they are filled up, but when buggy code keeps
eating
> >resources, the garbage collector can only free a small percentage each
time,
> >even when running full garbage collections. As an example the minimum
used
> >memory limit that triggers this watchdog could be set to 112Mb and the
time
> >limit to 15 mins. If the memory usage does not drop below 112Mb within 15
> >mins from the time it got over that memory usage level, then restart the
> >app.
> >
> >What I do not know without diving head on into the code is of this sort
of
> >thing is possible with the way wrapper is working and if that information
> >can be reliably gathered using C or Java code.
> >
> >I hope I explained this thoroughly enough. Any help/hint/idea is highly
> >appreciated. Keep up the good work!
> >
> >Kind regards,
> >Lars
> >
> >
> >
> >-------------------------------------------------------
> >This sf.net email is sponsored by:ThinkGeek
> >Welcome to geek heaven.
> >http://thinkgeek.com/sf
> >_______________________________________________
> >Wrapper-user mailing list
> >Wra...@li...
> >https://lists.sourceforge.net/lists/listinfo/wrapper-user
> >
> >
> >
>
>
>
>
> -------------------------------------------------------
> This sf.net email is sponsored by:ThinkGeek
> Welcome to geek heaven.
> http://thinkgeek.com/sf
> _______________________________________________
> Wrapper-user mailing list
> Wra...@li...
> https://lists.sourceforge.net/lists/listinfo/wrapper-user
>
>
|
|
From: Leif M. <le...@ta...> - 2002-10-11 04:06:56
|
GMX wrote:
>Leif,
>
>Thanks for the reply, I will certainly looking into this. One issue why I
>was asking for a more external watchdog was that in case the system runs out
>of memory it often has some sort of fit, killing tasks and other nasty
>stuff. I fear that the hook to catch uncought excpetions might be failing as
>well. If you had this in the outside world as part of the watchdog process,
>you would not be affected by this. Does this make sense?
>
Yes, it makes sense. I am not currently aware of any way to discover
that a JVM is having
memory exceptions from outside of the JVM.
I think that you should actually be fairly successful at catching out of
memory errors using the
uncaughtException method in ThreadGroups. The reason is that your
application will usually
run out of memory deep inside some method. When the out of memory error
is thrown, the
exception gets thrown all the way up the call stack until it is caught
by the ThreadGroup and
sent to the uncaughtException method. Each method call on a stack
takes up memory, so
as the Exception gets cascaded up to the top, memory will be freed up.
Should be enough
in most cases to make it possible to call WrapperManager.restart from
within the
uncaughtException method.
That is basically all that I would be able to do if this was added to
the Java side of the
Wrapper.
It could be possible to monitor the memory usage of the Java process.
But because of
the way Java's memory management works, I am not sure how I could tell
whether Java
is really out of memory. Java typically allocates more memory than it
actually needs and
then internally does its own memory management.
The Runtime.totalMemory call returns the amount of memory allocated from
the system.
Then the Runtime.freeMemory call returns how much of that memory is
still available.
When freeMemory approaches 0, Java will first invoke GC. If that does
not free up very
much memory, then java will allocate more memory from the system. It
will stop allocating
memory from the system when it reaches the value set by -mx property set
when the JVM
was launched. Note that this defaults to 64MB even if your machine has
a lot of memory.
You can set this in the wrapper using the wrapper.java.maxmemory property.
I am open to other ideas that you may have, but right now, I am not sure
how the Wrapper
could do a better job of handling this problem.
>Of course, we have to fix the problem itself, but that is another story :-)
>
As for tracking down your memory problem. The first thing I would
suggest is to turn off
the JIT compiler using the -Djava.compiler=NONE property when launching
the JVM.
(Not sure if that still works with Java 1.4 It does with 1.3 though.
If your memory leak goes
away then from experience, it is most likely a problem caused by
synchronization
someplace. I have had similar problems in the past where JIT compiled
code behaves
badly when you are having synchronization errors. Non-JIT compiled code
will still have
errors. But the memory leaks and crashes seem to go away.
If that isn't the problem, then you may have an object leak someplace in
your code. Add the
following to the wrapper.conf.
wrapper.java.additional.5=-Xrunhprof:depth=8
wrapper.shutdown.timeout=0
The first enables object profiling. When the JVM is asked to exit, it
will dump this info to a file.
This can take several minutes depending how large your app is. To keep
the Wrapper from
thinking the JVM is hung, you want to disable shutdown timeouts. Make
sure to remove both
of these properties when you are done.
When the JVM exits you will see the following in the console or log file.
---
Dumping Java heap ... allocation sites ... done.
---
Look in the directory where Wrapper.exe is located and you will fine a
file: java.hprof.txt
This file could be quite large, so I hope you have a good editor.
Search for "SITES BEGIN". You will see a section towards the end of the
file that looks
like this:
---
SITES BEGIN (ordered by live bytes) Fri Oct 11 12:56:48 2002
percent live alloc'ed stack class
rank self accum bytes objs bytes objs trace name
1 6.80% 6.80% 544264 4779 544264 4779 0 [C
2 2.78% 9.59% 222624 964 223280 966 6361 [C
3 2.14% 11.73% 171248 1262 173504 1299 1 [C
4 2.12% 13.85% 169368 106 169368 106 17124 [B
5 2.08% 15.93% 166736 183 166736 183 0 [B
6 1.71% 17.64% 136624 397 208088 796 17100 [C
7 1.55% 19.18% 123664 435 123664 435 16788 [C
---
This section is ordered by how much memory is being used by objects which
were created at specific places in the code. If you are lucky, the
first row will
be a very large number.
The second to the last column is the Trace number. This says where the
objects
were created. In the above example this is "4779".
To find out where these [C (Character Arrays) where created, do another
search
for "TRACE 4779:"
I get the following:
---
TRACE 4779:
java.lang.StringBuffer.toString(StringBuffer.java:1233)
org.apache.catalina.startup.Catalina.createContextCommon(Catalina.java:614)
org.apache.catalina.startup.Catalina.createStartMapperDefaultContext(Catalina.java:521)
org.apache.catalina.startup.Catalina.createStartMapper(Catalina.java:391)
org.apache.catalina.startup.Catalina.start(Catalina.java:722)
org.apache.catalina.startup.Catalina.execute(Catalina.java:681)
org.apache.catalina.startup.Catalina.process(Catalina.java:179)
sun.reflect.NativeMethodAccessorImpl.invoke0(NativeMethodAccessorImpl.java:Native
method)
---
You may find that you need to increase the stack depth, but that takes
more memory
and I usually find a depth of 8 to be sufficient.
Hope this helps you,
Cheers,
Leif
|
|
From: GMX <lg...@gm...> - 2002-10-15 00:06:36
|
Leif, Thank you for the explanation. I did indeed work with the profiler before, but somehow often the problems are more subtle or do not show themselves easily. Would be nice to have something like OpimizeIt to get this all visually and being able to drill into each interactively. But for now the built in profiler must suffice. Kind regards, Lars |