wrapper-user

[Wrapper-user] Out of memory error

[Ashish G. <as...@se...>]

Hi,

I seem to get the following error in my wrapper log and the wrapper 
doesnt seem to restart the JVM.

  [ERROR] ThreadPool - -Caught exception executing 
org.apache.tomcat.util.net.TcpWorkerThread@198fa95f, terminating thread 
<java.lang.OutOfMemoryError>java.lang.OutOfMemoryError

[ERROR] PoolTcpEndpoint - -Unexpected error 
<java.lang.OutOfMemoryError>java.lang.OutOfMemoryError

Any help regarding this would be helpful.
I am running wrapper 2.2.8, and tomcat version 4.1.15 and java 1.3.1

Thank you,

Ashish

Re: [Wrapper-user] Out of memory error

[Leif M. <le...@ta...>]

Ashish,
    The Wrapper does not do anything special in the event of 
OutOfMemoryErrors.
This is because in most cases, while a single thread may terminate, the 
JVM as
a whole will continue to function.   It is actually possible to catch 
out of memory
errors and continue just as can be done with any other error or exception.

    Java by default sets a maximum memory limit of 65MB.   The Wrapper 
preserves
this default.  If, as in this case, you find it necessary to make use of 
more memory then
you can do so by passing a -Xmx64m parameter to the JVM.  The Wrapper 
provides
a pair of parameters to make it easier to set the maximum and initial 
memory settings.
    wrapper.java.initmemory=16
    wrapper.java.maxmemory=64

    I usually run Tomcat myself with only 64MB of memory.  But have a 
few other apps
that run with a setting of 256.  Be aware that this is the maximum 
memory setting, the
JVM will not use that much memory unless it is necessary.  Giving the 
JVM more
room to work with can make garbage collection run a little more smoothly.

    The initial memory setting defines how much memory the JVM allocates 
at startup.
If you know that your application always takes 96MB of memory, then 
setting this to
a large initial value will make the app startup a little faster..

    Please see the documentation for more info on these properties.

Cheers,
Leif
   

Ashish Gawarikar wrote:

> Hi,
>
> I seem to get the following error in my wrapper log and the wrapper 
> doesnt seem to restart the JVM.
>
>  [ERROR] ThreadPool - -Caught exception executing 
> org.apache.tomcat.util.net.TcpWorkerThread@198fa95f, terminating 
> thread <java.lang.OutOfMemoryError>java.lang.OutOfMemoryError
>
> [ERROR] PoolTcpEndpoint - -Unexpected error 
> <java.lang.OutOfMemoryError>java.lang.OutOfMemoryError
>
> Any help regarding this would be helpful.
> I am running wrapper 2.2.8, and tomcat version 4.1.15 and java 1.3.1
>
> Thank you,
>
> Ashish
>
>
>
> -------------------------------------------------------
> This SF.NET email is sponsored by:
> SourceForge Enterprise Edition + IBM + LinuxWorld = Something 2 See!
> http://www.vasoftware.com
> _______________________________________________
> Wrapper-user mailing list
> Wra...@li...
> https://lists.sourceforge.net/lists/listinfo/wrapper-user
>

Re: [Wrapper-user] Out of memory error

[Ashish G. <as...@se...>]

Leif Mortenson wrote:

> Ashish,
>    The Wrapper does not do anything special in the event of 
> OutOfMemoryErrors.
> This is because in most cases, while a single thread may terminate, 
> the JVM as
> a whole will continue to function.   It is actually possible to catch 
> out of memory
> errors and continue just as can be done with any other error or 
> exception.
>
>    Java by default sets a maximum memory limit of 65MB.   The Wrapper 
> preserves
> this default.  If, as in this case, you find it necessary to make use 
> of more memory then
> you can do so by passing a -Xmx64m parameter to the JVM.  The Wrapper 
> provides
> a pair of parameters to make it easier to set the maximum and initial 
> memory settings.
>    wrapper.java.initmemory=16
>    wrapper.java.maxmemory=64
>
>    I usually run Tomcat myself with only 64MB of memory.  But have a 
> few other apps
> that run with a setting of 256.  Be aware that this is the maximum 
> memory setting, the
> JVM will not use that much memory unless it is necessary.  Giving the 
> JVM more
> room to work with can make garbage collection run a little more smoothly.
>
>    The initial memory setting defines how much memory the JVM 
> allocates at startup.
> If you know that your application always takes 96MB of memory, then 
> setting this to
> a large initial value will make the app startup a little faster..
>
>    Please see the documentation for more info on these properties.
>
> Cheers,
> Leif 

Leif,

Thanks for the info. I have used the settings (currently I have the max 
set to 128 MB).
But the real question (may be I missed to say that), is that the JVM 
stops responding,
and the tomcat responds with a page saying "horrible exception". My 
question in
regards to that was, if the JVM does stop responding, doesnt the wrapper 
detect that,
and restart the JVM?

Thanks,

Ashish

Re: [Wrapper-user] Out of memory error

[Leif M. <le...@ta...>]

Ashish Gawarikar wrote:

> Thanks for the info. I have used the settings (currently I have the 
> max set to 128 MB).
> But the real question (may be I missed to say that), is that the JVM 
> stops responding,
> and the tomcat responds with a page saying "horrible exception". My 
> question in
> regards to that was, if the JVM does stop responding, doesnt the 
> wrapper detect that,
> and restart the JVM? 

If this is easy to reproduce.  Could you please try running tomcat with 
the Wrapper's
debug output enabled.  My guess is that the JVM is actually fine but 
Tomcat has had a
critical error and it has stopped functioning.  If that is the case, 
then it would be a bug that
should be reported to the Tomcat team.

One thing that would be possible for me to do on the Wrapper side of 
things is to add a
property which would would have the Wrapper search for the string
"java.lang.OutOfMemoryError" in the stdout/stderr output of the JVM and 
then restart
the JVM.  It shouldn't cause that much of a performance hit, but I would 
still want it to be
optional.  Might as well allow users to enter their own trigger 
string(s) while I'm at it.
Needless to say, this would only work where output was sent to the JVM 
console.
These are actually problems caused by the application, but they can 
sometimes be
difficult to handle when the problem would need to be trapped within 3rd 
party code.

Cheers,
Leif

Re: [Wrapper-user] Out of memory error

[Ashish G. <as...@se...>]

Leif Mortenson wrote:

> Ashish Gawarikar wrote:
>
>> Thanks for the info. I have used the settings (currently I have the 
>> max set to 128 MB).
>> But the real question (may be I missed to say that), is that the JVM 
>> stops responding,
>> and the tomcat responds with a page saying "horrible exception". My 
>> question in
>> regards to that was, if the JVM does stop responding, doesnt the 
>> wrapper detect that,
>> and restart the JVM? 
>
>
> If this is easy to reproduce.  Could you please try running tomcat 
> with the Wrapper's
> debug output enabled.  My guess is that the JVM is actually fine but 
> Tomcat has had a
> critical error and it has stopped functioning.  If that is the case, 
> then it would be a bug that
> should be reported to the Tomcat team.

Its not very easy to reproduce. I ran tomcat with 100 users for 100 
iterations to do all the application
stuff, and sometimes in a blue moon this happens.
Our QA team is having a problem in reproducing this.

>
> One thing that would be possible for me to do on the Wrapper side of 
> things is to add a
> property which would would have the Wrapper search for the string
> "java.lang.OutOfMemoryError" in the stdout/stderr output of the JVM 
> and then restart
> the JVM.  It shouldn't cause that much of a performance hit, but I 
> would still want it to be
> optional.  Might as well allow users to enter their own trigger 
> string(s) while I'm at it.
> Needless to say, this would only work where output was sent to the JVM 
> console.
> These are actually problems caused by the application, but they can 
> sometimes be
> difficult to handle when the problem would need to be trapped within 
> 3rd party code.

This does sound like a very very good option. If something like this is 
present, it will
help us a lot :)

Thank you very much for the fast responses. I appreciate all the help.

Regards,

Ashish

>
> Cheers,
> Leif
>

Re: [Wrapper-user] Out of memory error

[Leif M. <le...@ta...>]

Ashish Gawarikar wrote:

> Its not very easy to reproduce. I ran tomcat with 100 users for 100 
> iterations to do all the application
> stuff, and sometimes in a blue moon this happens.
> Our QA team is having a problem in reproducing this. 

Try setting the max memory to 32 or 48MB rather than the default 64MB. 
 Should
make it happen more easily.

Another idea is to create a class EatMemory.doIt() that simply creates a 
LinkedList
and goes into an infinite loop of creating double arrays of length 
100000 and adding
them to the list.   You should get an OutOfMemoryError pretty quickly. 
 Question is
whether or not it happens in the right place to cause Tomcat to stop 
responding.

> This does sound like a very very good option. If something like this 
> is present, it will
> help us a lot :) 

Ok, I posted a feature request to track this feature:
http://sourceforge.net/tracker/index.php?func=detail&aid=676599&group_id=39428&atid=425190
Read it over and post any comments if you would like to see things done 
any differently.

Cheers,
Leif

Re: [Wrapper-user] Out of memory error

[Ashish G. <as...@se...>]

Leif Mortenson wrote:

>
>> This does sound like a very very good option. If something like this 
>> is present, it will
>> help us a lot :) 
>
>
> Ok, I posted a feature request to track this feature:
> http://sourceforge.net/tracker/index.php?func=detail&aid=676599&group_id=39428&atid=425190
> Read it over and post any comments if you would like to see things 
> done any differently.
>
> Cheers,
> Leif
>
>
>
Hi Leif,

I am having the following patch to do the feature, please let me know if 
this is okay (esp the methods wrapperRestartProcess() and 
wrapperStopProces()).

Thank you very much,

Ashish
------------------------

+    const char * prop;
+    const char * value;
+    char paramBuffer[128];
+    int i = 0;
+

     if (jvmOut != -1) {
         while (1) {
@@ -376,6 +381,34 @@
             if (bytesRead <= 0) {
                 break;
             }
+           do {
+                   sprintf(paramBuffer, "wrapper.filter.trigger.%d", i 
+ 1);
+                   prop = getStringProperty(properties, paramBuffer, NULL);
+                   if (prop) {
+                           if (strlen(prop) > 0) {
+                                   if (strstr(readBuf, prop))
+                                   {
+                                           sprintf(paramBuffer, 
"wrapper.filter.action.%d", i + 1);
+                                           value = 
getStringProperty(properties, paramBuffer, NULL);
+                                           if (value) {
+                                                   if (strlen(value) > 0) {
+                                                           if 
(strcasecmp(value,"SHUTDOWN") == 0)
+                                                           {
+                                                                   
wrapperStopRequested(1);
+                                                           }
+                                                           else if 
(strcasecmp(value,"RESTART") == 0)
+                                                           {
+                                                                   
wrapperRestartProcess();
+                                                                   
usleep(1000000); /* sleep 1 sec */
+                                                           }
+                                                   }
+                                           }
+                                   }
+                           }
+                   }
+                   ++i;
+           }while (prop);
+

Re: [Wrapper-user] Out of memory error

[Ashish G. <as...@se...>]

Ashish Gawarikar wrote:

> Leif Mortenson wrote:
>
>>
>>> This does sound like a very very good option. If something like this 
>>> is present, it will
>>> help us a lot :) 
>>
>>
>>
>> Ok, I posted a feature request to track this feature:
>> http://sourceforge.net/tracker/index.php?func=detail&aid=676599&group_id=39428&atid=425190
>> Read it over and post any comments if you would like to see things 
>> done any differently.
>>
>> Cheers,
>> Leif
>>
>>
>>
> Hi Leif,
>
> I am having the following patch to do the feature, please let me know 
> if this is okay (esp the methods wrapperRestartProcess() and 
> wrapperStopProces()).
>
> Thank you very much,
>
> Ashish
> ------------------------
>
> +    const char * prop;
> +    const char * value;
> +    char paramBuffer[128];
> +    int i = 0;
> +
>
>     if (jvmOut != -1) {
>         while (1) {
> @@ -376,6 +381,34 @@
>             if (bytesRead <= 0) {
>                 break;
>             }
> +           do {
> +                   sprintf(paramBuffer, "wrapper.filter.trigger.%d", 
> i + 1);
> +                   prop = getStringProperty(properties, paramBuffer, 
> NULL);
> +                   if (prop) {
> +                           if (strlen(prop) > 0) {
> +                                   if (strstr(readBuf, prop))
> +                                   {
> +                                           sprintf(paramBuffer, 
> "wrapper.filter.action.%d", i + 1);
> +                                           value = 
> getStringProperty(properties, paramBuffer, NULL);
> +                                           if (value) {
> +                                                   if (strlen(value) 
> > 0) {
> +                                                           if 
> (strcasecmp(value,"SHUTDOWN") == 0)
> +                                                           {
> +                                                                   
> wrapperStopRequested(1);
> +                                                           }
> +                                                           else if 
> (strcasecmp(value,"RESTART") == 0)
> +                                                           {
> +                                                                   
> wrapperRestartProcess();
> +                                                                   
> usleep(1000000); /* sleep 1 sec */
> +                                                           }
> +                                                   }
> +                                           }
> +                                   }
> +                           }
> +                   }
> +                   ++i;
> +           }while (prop);
> +
>
>
Sorry to mention that this is in the wrapper_unix.c, and the method is 
wrapperReadChildOutput().
Thanks,

Ashish

Re: [Wrapper-user] Out of memory error

[Leif M. <le...@ta...>]

Ashish,
    I got this all implemented and it will be in the 3.0.0 release.

http://sourceforge.net/tracker/index.php?func=detail&aid=676599&group_id=39428&atid=425190

Cheers,
Leif

Re: [Wrapper-user] Out of memory error

[Ashish G. <as...@se...>]

Leif Mortenson wrote:

> Ashish,
>    I got this all implemented and it will be in the 3.0.0 release.
>
> http://sourceforge.net/tracker/index.php?func=detail&aid=676599&group_id=39428&atid=425190
>
> Cheers,
> Leif
>
Thank you very much Leif!

Regards,

Ashish