wrapper-user — User issues for Wrapper. (Support questions, etc.)

Re: [Wrapper-user] Advice on stopping wrapper from triggering JVM restarts

[Casey J. <cas...@jo...>]

Ok, so I have been monitoring for a bit now and I can definitely say that I
have this memory leak and it is causing the issues I am experiencing.

Now the issue remains that I do not believe that I can upgrade glibc on my
system, given that all the threads I read say that will cause major
problems. Also, I need to use CentOS 7 if at all possible.

I noticed in the previously referenced thread that someone created a patch
directly in the wrapper to also fix this issue, does anyone know if this
fix made it into a release that I could use?

Thanks!

Re: [Wrapper-user] Getting transport error 202: bind failed: Address already in use error starting service

[Leif M. <lei...@ta...>]

David,
I am sorry you are having trouble.  The Wrapper ships with a 15 minute
timed license call src/conf/wrapper-license-timed.conf in the Wrapper
distribution.
You should be able to use this for quick tests without needing to obtain an
actual trial license.
This should be very painless and simple so I would of course like to help
you resolve the problem.

I will also send you a followup message through support.

Cheers,
Leif


On Mon, Dec 15, 2014 at 12:18 AM, David Hoffer <dho...@gm...> wrote:
>
> The need to debug my app has occurred again and I always have trouble
> getting this to work.  Last time I just used a separate trial license to
> get me by.  My situation is that we have two production apps running and
> occasionally I need to setup a separate instance of one of them to debug an
> issue.  I typically only need the debug instance for about 30 minutes but I
> can never get JSW to run the separate instance and I don't know why.  We
> have the pro license I thought I could run any number of instances on the
> same server, right?  I guess I'm going to have to transition this app off
> JSW so I can do these quick debug installs...the way it is now I spend way
> more time getting JSW to run than on the issue at hand.  Do you have a fix
> for this with JSW?  Is there some flag I can set to temp use non-pro
> (non-licensed) version...I don't need any of those features especially for
> debugging session.
>
> -Dave
>
> On Thu, Nov 13, 2014 at 8:13 PM, David Hoffer <dho...@gm...> wrote:
>>
>> Also I should mention I do have this new dev app in a different folder
>> and it has a different name, e.g. dev_ prefixed to name.  Everything else I
>> think is the same.
>>
>> -Dave
>>
>> On Thu, Nov 13, 2014 at 7:50 PM, David Hoffer <dho...@gm...> wrote:
>>
>>> I'm not sure I understand.  We do have the production app running...no
>>> idea what ports the wrapper is using there.  I need this to run at the same
>>> time...for debugging.
>>>
>>> -Dave
>>>
>>> On Thu, Nov 13, 2014 at 7:02 PM, Leif Mortenson <
>>> lei...@ta...> wrote:
>>>
>>>> David,
>>>> Looking at the command line, the Wrapper has bound and is listening on
>>>> port 32002, and the JVM will then use a port in the 31000-31999 range will
>>>> be used.  There are no other ports being used by the Wrapper.
>>>>
>>>> As 32002 was used rather than 32000, that means that 32000 and 32001
>>>> are in use or were recently in use.   Could you check to make sure that
>>>> another instance of your application is not currently running?  That would
>>>> explain the application level port failing to bind.
>>>>
>>>> Cheers,
>>>> Leif
>>>>
>>>>
>>>>
>>>> On Fri, Nov 14, 2014 at 10:34 AM, David Hoffer <dho...@gm...>
>>>> wrote:
>>>>
>>>>> I'm using 3.5.15-pro version.  I have the production service on this
>>>>> VM working but I need to standup a second instance so I can debug a
>>>>> problem.  I'm getting the following error with the second instance.  What
>>>>> is causing 'ERROR: transport error 202: bind failed: Address already in
>>>>> use'?  I have my app configured to use different ports so as to not
>>>>> conflict with the production app...is JSW using the same port?
>>>>>
>>>>> How can I fix this?
>>>>>
>>>>> wrapper  |
>>>>> ---------------------------------------------------------------------
>>>>> wrapper  | The JVM is being launched with a debugger enabled and could
>>>>> possibly
>>>>> wrapper  | be suspended.  To avoid unwanted shutdowns, timeouts will be
>>>>> wrapper  | disabled, removing the ability to detect and restart frozen
>>>>> JVMs.
>>>>> wrapper  |
>>>>> ---------------------------------------------------------------------
>>>>> wrapper  | Command[0] : /usr/lib/jvm/java-7-openjdk-amd64/jre/bin/java
>>>>> wrapper  | Command[1] : -Djava.util.Arrays.useLegacyMergeSort=true
>>>>> wrapper  | Command[2] : -Xdebug
>>>>> wrapper  | Command[3] :
>>>>> -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=5005
>>>>> wrapper  | Command[4] : -Xms64m
>>>>> wrapper  | Command[5] : -Xmx1024m
>>>>> wrapper  | Command[6] : -Djava.library.path=./
>>>>> wrapper  | Command[7] : -classpath
>>>>> wrapper  | Command[8] :
>>>>> ../lib/activation-1.1.jar:../lib/aopalliance-1.0.jar:../lib/asm-3.3.1.jar:../lib/bcmail-jdk15-1.44.jar:../lib/bcprov-jdk15-1.44.jar:../lib/commons-codec-1.5.jar:../lib/commons-email-1.2.jar:../lib/commons-lang-2.6.jar:../lib/commons-logging-1.1.1.jar:../lib/commons-math-1.2.jar:../lib/commons-net-3.1.jar:../lib/cxf-api-2.7.10.jar:../lib/cxf-rt-bindings-soap-2.7.10.jar:../lib/cxf-rt-bindings-xml-2.7.10.jar:../lib/cxf-rt-core-2.7.10.jar:../lib/cxf-rt-databinding-aegis-2.7.10.jar:../lib/cxf-rt-databinding-jaxb-2.7.10.jar:../lib/cxf-rt-frontend-jaxws-2.7.10.jar:../lib/cxf-rt-frontend-simple-2.7.10.jar:../lib/cxf-rt-management-2.7.10.jar:../lib/cxf-rt-transports-http-2.7.10.jar:../lib/cxf-rt-transports-http-jetty-2.7.10.jar:../lib/cxf-rt-transports-local-2.7.10.jar:../lib/cxf-rt-ws-addr-2.7.10.jar:../lib/cxf-rt-ws-policy-2.7.10.jar:../lib/dhs-commons-1.6-SNAPSHOT.jar:../lib/dom4j-1.6.1.jar:../lib/fluent-hc-4.3.1.jar:../lib/fontbox-1.8.2.jar:../lib/geronimo-servlet_3.0_spec-1.0.jar:../lib/guice-3.0.jar:../lib/guice-assistedinject-3.0.jar:../lib/httpclient-4.3.1.jar:../lib/httpcore-4.3.jar:../lib/ifreelance-pn27284-webservice-1.5-SNAPSHOT.jar:../lib/itext-2.1.7.jar:../lib/javax.inject-1.jar:../lib/jaxb2-basics-runtime-0.6.1.jar:../lib/jaxb-api-2.2.jar:../lib/jaxb-impl-2.2.4.jar:../lib/jdom-1.0.jar:../lib/jempbox-1.8.2.jar:../lib/jetty-continuation-8.1.14.v20131031.jar:../lib/jetty-http-8.1.14.v20131031.jar:../lib/jetty-io-8.1.14.v20131031.jar:../lib/jetty-security-8.1.14.v20131031.jar:../lib/jetty-server-8.1.14.v20131031.jar:../lib/jetty-util-8.1.14.v20131031.jar:../lib/jetty-xml-8.1.14.v20131031.jar:../lib/joda-time-1.6.2.jar:../lib/jsr173_api-1.0.jar:../lib/jsr250-api-1.0.jar:../lib/jsr305-1.3.9.jar:../lib/log4j-1.2.16.jar:../lib/mail-1.4.1.jar:../lib/mortgage-tools-1.4-SNAPSHOT.jar:../lib/neethi-3.0.3.jar:../lib/pdfbox-1.8.2.jar:../lib/poi-3.9.jar:../lib/poi-ooxml-3.9.jar:../lib/poi-ooxml-schemas-3.9.jar:../lib/runtime-0.4.1.jar:../lib/slf4j-api-1.7.5.jar:../lib/stax2-api-3.1.1.jar:../lib/stax-api-1.0-2.jar:../lib/stax-api-1.0.1.jar:../lib/woodstox-core-asl-4.2.0.jar:../lib/wrapper-3.5.15.jar:../lib/wsdl4j-1.6.3.jar:../lib/xml-apis-1.0.b2.jar:../lib/xml-resolver-1.2.jar:../lib/xmlbeans-2.3.0.jar:../lib/xmlschema-core-2.1.0.jar
>>>>> wrapper  | Command[9] : -Dwrapper.key=RI8htg3WrkuGSLCkOHiJ-0Dkk4OtOvbe
>>>>> wrapper  | Command[10] : -Dwrapper.port=32002
>>>>> wrapper  | Command[11] : -Dwrapper.jvm.port.min=31000
>>>>> wrapper  | Command[12] : -Dwrapper.jvm.port.max=31999
>>>>> wrapper  | Command[13] : -Dwrapper.pid=20122
>>>>> wrapper  | Command[14] : -Dwrapper.version=3.5.15-pro
>>>>> wrapper  | Command[15] : -Dwrapper.native_library=wrapper
>>>>> wrapper  | Command[16] : -Dwrapper.cpu.timeout=10
>>>>> wrapper  | Command[17] : -Dwrapper.jvmid=5
>>>>> wrapper  | Command[18] : -Dwrapper.lang.domain=wrapper
>>>>> wrapper  | Command[19] :
>>>>> com.qsd.service.jsw.JavaServiceWrapperWebServiceLauncher
>>>>> wrapper  | Launching a JVM...
>>>>> jvm 5    | ERROR: transport error 202: bind failed: Address already in
>>>>> use
>>>>> jvm 5    | ERROR: JDWP Transport dt_socket failed to initialize,
>>>>> TRANSPORT_INIT(510)
>>>>> jvm 5    | JDWP exit error AGENT_ERROR_TRANSPORT_INIT(197): No
>>>>> transports initialized [../../../src/share/back/debugInit.c:750]
>>>>> jvm 5    | FATAL ERROR in native method: JDWP No transports
>>>>> initialized, jvmtiError=AGENT_ERROR_TRANSPORT_INIT(197)
>>>>> wrapper  | JVM received a signal UNKNOWN (6).
>>>>> wrapper  | JVM process is gone.
>>>>> wrapper  | JVM exited while loading the application.
>>>>> wrapper  | There were 5 failed launches in a row, each lasting less
>>>>> than 300 seconds.  Giving up.
>>>>> wrapper  |   There may be a configuration problem: please check the
>>>>> logs.
>>>>> wrapper  | <-- Wrapper Stopped
>>>>>
>>>>

Re: [Wrapper-user] Getting transport error 202: bind failed: Address already in use error starting service

[David H. <dho...@gm...>]

The need to debug my app has occurred again and I always have trouble
getting this to work.  Last time I just used a separate trial license to
get me by.  My situation is that we have two production apps running and
occasionally I need to setup a separate instance of one of them to debug an
issue.  I typically only need the debug instance for about 30 minutes but I
can never get JSW to run the separate instance and I don't know why.  We
have the pro license I thought I could run any number of instances on the
same server, right?  I guess I'm going to have to transition this app off
JSW so I can do these quick debug installs...the way it is now I spend way
more time getting JSW to run than on the issue at hand.  Do you have a fix
for this with JSW?  Is there some flag I can set to temp use non-pro
(non-licensed) version...I don't need any of those features especially for
debugging session.

-Dave

On Thu, Nov 13, 2014 at 8:13 PM, David Hoffer <dho...@gm...> wrote:
>
> Also I should mention I do have this new dev app in a different folder and
> it has a different name, e.g. dev_ prefixed to name.  Everything else I
> think is the same.
>
> -Dave
>
> On Thu, Nov 13, 2014 at 7:50 PM, David Hoffer <dho...@gm...> wrote:
>
>> I'm not sure I understand.  We do have the production app running...no
>> idea what ports the wrapper is using there.  I need this to run at the same
>> time...for debugging.
>>
>> -Dave
>>
>> On Thu, Nov 13, 2014 at 7:02 PM, Leif Mortenson <
>> lei...@ta...> wrote:
>>
>>> David,
>>> Looking at the command line, the Wrapper has bound and is listening on
>>> port 32002, and the JVM will then use a port in the 31000-31999 range will
>>> be used.  There are no other ports being used by the Wrapper.
>>>
>>> As 32002 was used rather than 32000, that means that 32000 and 32001 are
>>> in use or were recently in use.   Could you check to make sure that another
>>> instance of your application is not currently running?  That would explain
>>> the application level port failing to bind.
>>>
>>> Cheers,
>>> Leif
>>>
>>>
>>>
>>> On Fri, Nov 14, 2014 at 10:34 AM, David Hoffer <dho...@gm...>
>>> wrote:
>>>
>>>> I'm using 3.5.15-pro version.  I have the production service on this VM
>>>> working but I need to standup a second instance so I can debug a problem.
>>>> I'm getting the following error with the second instance.  What is causing
>>>> 'ERROR: transport error 202: bind failed: Address already in use'?  I have
>>>> my app configured to use different ports so as to not conflict with the
>>>> production app...is JSW using the same port?
>>>>
>>>> How can I fix this?
>>>>
>>>> wrapper  |
>>>> ---------------------------------------------------------------------
>>>> wrapper  | The JVM is being launched with a debugger enabled and could
>>>> possibly
>>>> wrapper  | be suspended.  To avoid unwanted shutdowns, timeouts will be
>>>> wrapper  | disabled, removing the ability to detect and restart frozen
>>>> JVMs.
>>>> wrapper  |
>>>> ---------------------------------------------------------------------
>>>> wrapper  | Command[0] : /usr/lib/jvm/java-7-openjdk-amd64/jre/bin/java
>>>> wrapper  | Command[1] : -Djava.util.Arrays.useLegacyMergeSort=true
>>>> wrapper  | Command[2] : -Xdebug
>>>> wrapper  | Command[3] :
>>>> -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=5005
>>>> wrapper  | Command[4] : -Xms64m
>>>> wrapper  | Command[5] : -Xmx1024m
>>>> wrapper  | Command[6] : -Djava.library.path=./
>>>> wrapper  | Command[7] : -classpath
>>>> wrapper  | Command[8] :
>>>> ../lib/activation-1.1.jar:../lib/aopalliance-1.0.jar:../lib/asm-3.3.1.jar:../lib/bcmail-jdk15-1.44.jar:../lib/bcprov-jdk15-1.44.jar:../lib/commons-codec-1.5.jar:../lib/commons-email-1.2.jar:../lib/commons-lang-2.6.jar:../lib/commons-logging-1.1.1.jar:../lib/commons-math-1.2.jar:../lib/commons-net-3.1.jar:../lib/cxf-api-2.7.10.jar:../lib/cxf-rt-bindings-soap-2.7.10.jar:../lib/cxf-rt-bindings-xml-2.7.10.jar:../lib/cxf-rt-core-2.7.10.jar:../lib/cxf-rt-databinding-aegis-2.7.10.jar:../lib/cxf-rt-databinding-jaxb-2.7.10.jar:../lib/cxf-rt-frontend-jaxws-2.7.10.jar:../lib/cxf-rt-frontend-simple-2.7.10.jar:../lib/cxf-rt-management-2.7.10.jar:../lib/cxf-rt-transports-http-2.7.10.jar:../lib/cxf-rt-transports-http-jetty-2.7.10.jar:../lib/cxf-rt-transports-local-2.7.10.jar:../lib/cxf-rt-ws-addr-2.7.10.jar:../lib/cxf-rt-ws-policy-2.7.10.jar:../lib/dhs-commons-1.6-SNAPSHOT.jar:../lib/dom4j-1.6.1.jar:../lib/fluent-hc-4.3.1.jar:../lib/fontbox-1.8.2.jar:../lib/geronimo-servlet_3.0_spec-1.0.jar:../lib/guice-3.0.jar:../lib/guice-assistedinject-3.0.jar:../lib/httpclient-4.3.1.jar:../lib/httpcore-4.3.jar:../lib/ifreelance-pn27284-webservice-1.5-SNAPSHOT.jar:../lib/itext-2.1.7.jar:../lib/javax.inject-1.jar:../lib/jaxb2-basics-runtime-0.6.1.jar:../lib/jaxb-api-2.2.jar:../lib/jaxb-impl-2.2.4.jar:../lib/jdom-1.0.jar:../lib/jempbox-1.8.2.jar:../lib/jetty-continuation-8.1.14.v20131031.jar:../lib/jetty-http-8.1.14.v20131031.jar:../lib/jetty-io-8.1.14.v20131031.jar:../lib/jetty-security-8.1.14.v20131031.jar:../lib/jetty-server-8.1.14.v20131031.jar:../lib/jetty-util-8.1.14.v20131031.jar:../lib/jetty-xml-8.1.14.v20131031.jar:../lib/joda-time-1.6.2.jar:../lib/jsr173_api-1.0.jar:../lib/jsr250-api-1.0.jar:../lib/jsr305-1.3.9.jar:../lib/log4j-1.2.16.jar:../lib/mail-1.4.1.jar:../lib/mortgage-tools-1.4-SNAPSHOT.jar:../lib/neethi-3.0.3.jar:../lib/pdfbox-1.8.2.jar:../lib/poi-3.9.jar:../lib/poi-ooxml-3.9.jar:../lib/poi-ooxml-schemas-3.9.jar:../lib/runtime-0.4.1.jar:../lib/slf4j-api-1.7.5.jar:../lib/stax2-api-3.1.1.jar:../lib/stax-api-1.0-2.jar:../lib/stax-api-1.0.1.jar:../lib/woodstox-core-asl-4.2.0.jar:../lib/wrapper-3.5.15.jar:../lib/wsdl4j-1.6.3.jar:../lib/xml-apis-1.0.b2.jar:../lib/xml-resolver-1.2.jar:../lib/xmlbeans-2.3.0.jar:../lib/xmlschema-core-2.1.0.jar
>>>> wrapper  | Command[9] : -Dwrapper.key=RI8htg3WrkuGSLCkOHiJ-0Dkk4OtOvbe
>>>> wrapper  | Command[10] : -Dwrapper.port=32002
>>>> wrapper  | Command[11] : -Dwrapper.jvm.port.min=31000
>>>> wrapper  | Command[12] : -Dwrapper.jvm.port.max=31999
>>>> wrapper  | Command[13] : -Dwrapper.pid=20122
>>>> wrapper  | Command[14] : -Dwrapper.version=3.5.15-pro
>>>> wrapper  | Command[15] : -Dwrapper.native_library=wrapper
>>>> wrapper  | Command[16] : -Dwrapper.cpu.timeout=10
>>>> wrapper  | Command[17] : -Dwrapper.jvmid=5
>>>> wrapper  | Command[18] : -Dwrapper.lang.domain=wrapper
>>>> wrapper  | Command[19] :
>>>> com.qsd.service.jsw.JavaServiceWrapperWebServiceLauncher
>>>> wrapper  | Launching a JVM...
>>>> jvm 5    | ERROR: transport error 202: bind failed: Address already in
>>>> use
>>>> jvm 5    | ERROR: JDWP Transport dt_socket failed to initialize,
>>>> TRANSPORT_INIT(510)
>>>> jvm 5    | JDWP exit error AGENT_ERROR_TRANSPORT_INIT(197): No
>>>> transports initialized [../../../src/share/back/debugInit.c:750]
>>>> jvm 5    | FATAL ERROR in native method: JDWP No transports
>>>> initialized, jvmtiError=AGENT_ERROR_TRANSPORT_INIT(197)
>>>> wrapper  | JVM received a signal UNKNOWN (6).
>>>> wrapper  | JVM process is gone.
>>>> wrapper  | JVM exited while loading the application.
>>>> wrapper  | There were 5 failed launches in a row, each lasting less
>>>> than 300 seconds.  Giving up.
>>>> wrapper  |   There may be a configuration problem: please check the
>>>> logs.
>>>> wrapper  | <-- Wrapper Stopped
>>>>
>>>>
>>>
>>> ------------------------------------------------------------------------------
>>> Comprehensive Server Monitoring with Site24x7.
>>> Monitor 10 servers for $9/Month.
>>> Get alerted through email, SMS, voice calls or mobile push notifications.
>>> Take corrective actions from your mobile device.
>>>
>>> http://pubads.g.doubleclick.net/gampad/clk?id=154624111&iu=/4140/ostg.clktrk
>>> _______________________________________________
>>> Wrapper-user mailing list
>>> Wra...@li...
>>> https://lists.sourceforge.net/lists/listinfo/wrapper-user
>>>
>>>
>>
>

Re: [Wrapper-user] Advice on stopping wrapper from triggering JVM restarts

[Tim L. <tim...@gm...>]

Yes you can monitor this with top, but the speed depends on how much your
application logs.

On Wed, Dec 10, 2014 at 5:46 PM, Casey Jordan <cas...@jo...>
wrote:

> Cool thanks, will I be able to see this memory usage increase by looking
> at the top command?
>
> I would like to keep track of this a little more and see if I notice the
> wrapper memory growing.
>
> Thanks
>
> On Wed, Dec 10, 2014 at 11:06 AM, Tim Lammens <tim...@gm...>
> wrote:
>
>> That is indeed the right thread.
>>
>> Regards,
>> Tim
>>
>> On Wed, Dec 10, 2014 at 5:03 PM, Casey Jordan <cas...@jo...>
>> wrote:
>>
>>> Hi Tim,
>>>
>>> Thanks for the info, is this the thread you speak of:
>>> http://sourceforge.net/p/wrapper/mailman/message/33048371/
>>>
>>> If not could you point me to the right one?
>>>
>>> Thanks
>>>
>>> On Wed, Dec 10, 2014 at 10:58 AM, Tim Lammens <tim...@gm...>
>>> wrote:
>>>
>>>> You are most probably experiencing a glibc bug as I mentioned before in
>>>> another thread. The bug was fixed in recent glibc versions.
>>>> Try updating your glibc version for CentOS. If that is not possible you
>>>> can apply the patches I provided in a different thread on this mailing list.
>>>>
>>>> Regards,
>>>> Tim
>>>>
>>>> On Wed, Dec 10, 2014 at 4:49 PM, Casey Jordan <cas...@jo...>
>>>> wrote:
>>>>
>>>>> Ok so this happened again and I was able to capture some additional
>>>>> log information (not from wrapper.debug though as I can't update this
>>>>> server just yet)
>>>>>
>>>>> I got this in our logs:
>>>>>
>>>>> INFO   | jvm 1    | 2014/12/09 23:15:50 | <br></br>Max Mem: 2223 MB
>>>>> INFO   | jvm 1    | 2014/12/09 23:15:50 | <br></br>Total Mem: 2204 MB
>>>>> INFO   | jvm 1    | 2014/12/09 23:15:50 | <br></br>Used Mem: 1141 MB
>>>>> INFO   | jvm 1    | 2014/12/09 23:15:50 | <br></br>Raw free-mem: 1082 MB
>>>>> INFO   | jvm 1    | 2014/12/09 23:15:50 | <br></br>Total descriptors: 4096
>>>>> INFO   | jvm 1    | 2014/12/09 23:15:50 | <br></br>Open descriptors: 556
>>>>> INFO   | jvm 1    | 2014/12/09 23:15:50 | <br></br>Available descriptors: 3540
>>>>> INFO   | jvm 1    | 2014/12/09 23:15:50 | <br></br>Free disk space: 18 GB
>>>>> INFO   | jvm 1    | 2014/12/09 23:15:50 | <br></br>
>>>>> INFO   | wrapper  | 2014/12/09 23:17:09 | Wrapper Process has not received any CPU time for 1 seconds.  Extending timeouts.
>>>>> STATUS | wrapper  | 2014/12/09 23:17:09 | Pinging the JVM took 7 seconds to respond.
>>>>> INFO   | wrapper  | 2014/12/09 23:17:09 | Wrapper Process has not received any CPU time for 5 seconds.  Extending timeouts.
>>>>> STATUS | wrapper  | 2014/12/09 23:17:09 | JVM received a signal SIGKILL (9).
>>>>> STATUS | wrapper  | 2014/12/09 23:17:09 | JVM process is gone.
>>>>> ERROR  | wrapper  | 2014/12/09 23:17:09 | JVM exited unexpectedly.
>>>>> STATUS | wrapper  | 2014/12/09 23:17:09 | Automatic JVM Restarts disabled.  Shutting down.
>>>>> STATUS | wrapper  | 2014/12/09 23:17:09 | <-- Wrapper Stopped
>>>>>
>>>>>
>>>>> and I found this in the /var/log/messages
>>>>>
>>>>> Dec  9 23:17:09 jorsek-home2 kernel: Out of memory: Kill process 10262
>>>>> (java) score 591 or sacrifice child
>>>>> Dec  9 23:17:09 jorsek-home2 kernel: Killed process 10262 (java)
>>>>> total-vm:6478728kB, anon-rss:2600344kB, file-rss:0kB
>>>>>
>>>>> So this does appear to be the kernel killing the process, but
>>>>> according to java I have plenty of memory available (>1GB).
>>>>>
>>>>> I know this is not really a wrapper issue any longer, but I would be
>>>>> very appreciative if anyone can give me some advice or point me to some
>>>>> resources that might help figure this out.
>>>>>
>>>>> For reference this is a Centos7 box with 4GB of RAM and a 1GB swap
>>>>> space, and 40GB SSD drive.
>>>>>
>>>>> Tonight I am going to add NewRelic monitoring to this server to see if
>>>>> I can get a better picture of what is going on.
>>>>>
>>>>> Thanks!
>>>>>
>>>>> On Tue, Dec 2, 2014 at 7:30 PM, Roberto Espinoza <
>>>>> rob...@ta...> wrote:
>>>>>
>>>>>> Casey,
>>>>>>
>>>>>> Have you checked the kernel logs to see if it wasn't killed because
>>>>>> your system ran out of memory?
>>>>>>
>>>>>> Certainly there is no way to know this inside the JVM because the
>>>>>> kernel out of memory killer will just score the current processes and
>>>>>> issue a kill.
>>>>>>
>>>>>> On Wed, Dec 3, 2014 at 12:39 AM, Casey Jordan <
>>>>>> cas...@jo...> wrote:
>>>>>> > Hi Leif,
>>>>>> >
>>>>>> > Thanks for the feedback. This is quite a mystery then because I am
>>>>>> 100% sure
>>>>>> > that the process wasn't killed manually or by any other systems we
>>>>>> have
>>>>>> > running. (This is just a standard CentOS 7 setup)
>>>>>> >
>>>>>> > At about the same time that this log appeared I was running a very
>>>>>> heavy
>>>>>> > build process in a separate JVM instance.
>>>>>> >
>>>>>> > It seems more than just coincidental that this happened at the same
>>>>>> time. I
>>>>>> > assume that if the JVM crashed it wouldn't have received a SIG 9?
>>>>>> Can you
>>>>>> > think of any scenario where this makes sense? Perhaps I am missing
>>>>>> some
>>>>>> > knowledge of java architecture here that is making this mysterious.
>>>>>> >
>>>>>> > Thanks
>>>>>> >
>>>>>> >
>>>>>> > On Tue, Dec 2, 2014 at 4:27 AM, Leif Mortenson
>>>>>> > <lei...@ta...> wrote:
>>>>>> >>
>>>>>> >> Casey,
>>>>>> >> The configuration you specified will disable all pings and cpu
>>>>>> warnings
>>>>>> >> and should do what you want.
>>>>>> >> The Wrapper should never timeout  in this case.  There are other
>>>>>> things
>>>>>> >> such as a JVM crash would would still result in a restart after
>>>>>> the fact.
>>>>>> >>
>>>>>> >> The log that you send shows that the JVM will killed with a kill
>>>>>> -9.  Not
>>>>>> >> sure if that was a test on your part, but if the source of the
>>>>>> signal has
>>>>>> >> permission to kill the JVM's process then there is nothing we can
>>>>>> do to
>>>>>> >> prevent that.
>>>>>> >>
>>>>>> >> Cheers,
>>>>>> >> Leif
>>>>>> >>
>>>>>> >> On Tue, Dec 2, 2014 at 12:16 PM, Casey Jordan <
>>>>>> cas...@jo...>
>>>>>> >> wrote:
>>>>>> >>>
>>>>>> >>> Hi all,
>>>>>> >>>
>>>>>> >>> I am sure this is a common question, but I am having trouble
>>>>>> extracting
>>>>>> >>> details I need from the documentation. From time to time we get a
>>>>>> situation
>>>>>> >>> where our JVM gets killed and we see something like this in the
>>>>>> logs:
>>>>>> >>>
>>>>>> >>> INFO   | wrapper  | 2014/12/02 02:23:18 | Wrapper Process has not
>>>>>> >>> received any CPU time for 1 seconds.  Extending timeouts.
>>>>>> >>> STATUS | wrapper  | 2014/12/02 02:23:37 | JVM received a signal
>>>>>> SIGKILL
>>>>>> >>> (9).
>>>>>> >>> STATUS | wrapper  | 2014/12/02 02:23:37 | JVM process is gone.
>>>>>> >>> ERROR  | wrapper  | 2014/12/02 02:23:37 | JVM exited unexpectedly.
>>>>>> >>> STATUS | wrapper  | 2014/12/02 02:23:38 | Automatic JVM Restarts
>>>>>> >>> disabled.  Shutting down.
>>>>>> >>> STATUS | wrapper  | 2014/12/02 02:23:38 | <-- Wrapper Stopped
>>>>>> >>>
>>>>>> >>>
>>>>>> >>> This always happens in the case where some other process is
>>>>>> eating up all
>>>>>> >>> the cpu for more than a few seconds.
>>>>>> >>>
>>>>>> >>> It's crucial for us that the wrapper never, ever restarts the JVM
>>>>>> as this
>>>>>> >>> is something we always want to look into manually.
>>>>>> >>>
>>>>>> >>> So my question is, what is the best way to make sure this doesn't
>>>>>> happen?
>>>>>> >>>
>>>>>> >>> I am using community version 3.5.17, and have the following
>>>>>> values in our
>>>>>> >>> wrapper.conf:
>>>>>> >>>
>>>>>> >>>
>>>>>> #********************************************************************
>>>>>> >>>    # Timeouts
>>>>>> >>>
>>>>>> #********************************************************************
>>>>>> >>>    wrapper.ping.timeout=0
>>>>>> >>>    wrapper.ping.timeout.action=DEBUG
>>>>>> >>>    wrapper.cpu.timeout=0
>>>>>> >>>    wrapper.startup.timeout=300
>>>>>> >>>
>>>>>> >>> Any help is much appreciated.
>>>>>> >>>
>>>>>> >>> Thanks!
>>>>>> >>>
>>>>>> >>
>>>>>> >>
>>>>>> >>
>>>>>> ------------------------------------------------------------------------------
>>>>>> >> Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
>>>>>> >> from Actuate! Instantly Supercharge Your Business Reports and
>>>>>> Dashboards
>>>>>> >> with Interactivity, Sharing, Native Excel Exports, App Integration
>>>>>> & more
>>>>>> >> Get technology previously reserved for billion-dollar
>>>>>> corporations, FREE
>>>>>> >>
>>>>>> >>
>>>>>> http://pubads.g.doubleclick.net/gampad/clk?id=157005751&iu=/4140/ostg.clktrk
>>>>>> >> _______________________________________________
>>>>>> >> Wrapper-user mailing list
>>>>>> >> Wra...@li...
>>>>>> >> https://lists.sourceforge.net/lists/listinfo/wrapper-user
>>>>>> >>
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> > --
>>>>>> > --
>>>>>> > Casey Jordan
>>>>>> > easyDITA a product of Jorsek LLC
>>>>>> > "CaseyDJordan" on LinkedIn, Twitter & Facebook
>>>>>> > (585) 348 7399
>>>>>> > easydita.com
>>>>>> >
>>>>>> >
>>>>>> > This message is intended only for the use of the Addressee(s) and
>>>>>> may
>>>>>> > contain information that is privileged, confidential, and/or exempt
>>>>>> from
>>>>>> > disclosure under applicable law.  If you are not the intended
>>>>>> recipient,
>>>>>> > please be advised that any disclosure  copying, distribution, or
>>>>>> use of
>>>>>> > the information contained herein is prohibited.  If you have
>>>>>> received
>>>>>> > this communication in error, please destroy all copies of the
>>>>>> message,
>>>>>> > whether in electronic or hard copy format, as well as attachments,
>>>>>> and
>>>>>> > immediately contact the sender by replying to this e-mail or by
>>>>>> phone.
>>>>>> > Thank you.
>>>>>> >
>>>>>> >
>>>>>> ------------------------------------------------------------------------------
>>>>>> > Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
>>>>>> > from Actuate! Instantly Supercharge Your Business Reports and
>>>>>> Dashboards
>>>>>> > with Interactivity, Sharing, Native Excel Exports, App Integration
>>>>>> & more
>>>>>> > Get technology previously reserved for billion-dollar corporations,
>>>>>> FREE
>>>>>> >
>>>>>> http://pubads.g.doubleclick.net/gampad/clk?id=157005751&iu=/4140/ostg.clktrk
>>>>>> > _______________________________________________
>>>>>> > Wrapper-user mailing list
>>>>>> > Wra...@li...
>>>>>> > https://lists.sourceforge.net/lists/listinfo/wrapper-user
>>>>>> >
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Roberto Espinoza
>>>>>> Tanuki Software, Ltd.
>>>>>> 6-16-7-1001 Nishi-Kasai, Edogawa-ku
>>>>>> Tokyo 134-0088 Japan
>>>>>> Tel/Fax: +81-3-3878-3211
>>>>>> http://www.tanukisoftware.com
>>>>>> rob...@ta...
>>>>>>
>>>>>>
>>>>>> ------------------------------------------------------------------------------
>>>>>> Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
>>>>>> from Actuate! Instantly Supercharge Your Business Reports and
>>>>>> Dashboards
>>>>>> with Interactivity, Sharing, Native Excel Exports, App Integration &
>>>>>> more
>>>>>> Get technology previously reserved for billion-dollar corporations,
>>>>>> FREE
>>>>>>
>>>>>> http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk
>>>>>> _______________________________________________
>>>>>> Wrapper-user mailing list
>>>>>> Wra...@li...
>>>>>> https://lists.sourceforge.net/lists/listinfo/wrapper-user
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> --
>>>>> Casey Jordan
>>>>> easyDITA a product of Jorsek LLC
>>>>> "CaseyDJordan" on LinkedIn, Twitter & Facebook
>>>>> (585) 348 7399
>>>>> easydita.com
>>>>>
>>>>>
>>>>> This message is intended only for the use of the Addressee(s) and may
>>>>> contain information that is privileged, confidential, and/or exempt
>>>>> from
>>>>> disclosure under applicable law.  If you are not the intended
>>>>> recipient,
>>>>> please be advised that any disclosure  copying, distribution, or use of
>>>>> the information contained herein is prohibited.  If you have received
>>>>> this communication in error, please destroy all copies of the message,
>>>>> whether in electronic or hard copy format, as well as attachments, and
>>>>> immediately contact the sender by replying to this e-mail or by phone.
>>>>> Thank you.
>>>>>
>>>>>
>>>>> ------------------------------------------------------------------------------
>>>>> Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
>>>>> from Actuate! Instantly Supercharge Your Business Reports and
>>>>> Dashboards
>>>>> with Interactivity, Sharing, Native Excel Exports, App Integration &
>>>>> more
>>>>> Get technology previously reserved for billion-dollar corporations,
>>>>> FREE
>>>>>
>>>>> http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk
>>>>> _______________________________________________
>>>>> Wrapper-user mailing list
>>>>> Wra...@li...
>>>>> https://lists.sourceforge.net/lists/listinfo/wrapper-user
>>>>>
>>>>>
>>>>
>>>>
>>>> ------------------------------------------------------------------------------
>>>> Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
>>>> from Actuate! Instantly Supercharge Your Business Reports and Dashboards
>>>> with Interactivity, Sharing, Native Excel Exports, App Integration &
>>>> more
>>>> Get technology previously reserved for billion-dollar corporations, FREE
>>>>
>>>> http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk
>>>> _______________________________________________
>>>> Wrapper-user mailing list
>>>> Wra...@li...
>>>> https://lists.sourceforge.net/lists/listinfo/wrapper-user
>>>>
>>>>
>>>
>>>
>>> --
>>> --
>>> Casey Jordan
>>> easyDITA a product of Jorsek LLC
>>> "CaseyDJordan" on LinkedIn, Twitter & Facebook
>>> (585) 348 7399
>>> easydita.com
>>>
>>>
>>> This message is intended only for the use of the Addressee(s) and may
>>> contain information that is privileged, confidential, and/or exempt from
>>> disclosure under applicable law.  If you are not the intended recipient,
>>> please be advised that any disclosure  copying, distribution, or use of
>>> the information contained herein is prohibited.  If you have received
>>> this communication in error, please destroy all copies of the message,
>>> whether in electronic or hard copy format, as well as attachments, and
>>> immediately contact the sender by replying to this e-mail or by phone.
>>> Thank you.
>>>
>>>
>>> ------------------------------------------------------------------------------
>>> Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
>>> from Actuate! Instantly Supercharge Your Business Reports and Dashboards
>>> with Interactivity, Sharing, Native Excel Exports, App Integration & more
>>> Get technology previously reserved for billion-dollar corporations, FREE
>>>
>>> http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk
>>> _______________________________________________
>>> Wrapper-user mailing list
>>> Wra...@li...
>>> https://lists.sourceforge.net/lists/listinfo/wrapper-user
>>>
>>>
>>
>>
>> ------------------------------------------------------------------------------
>> Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
>> from Actuate! Instantly Supercharge Your Business Reports and Dashboards
>> with Interactivity, Sharing, Native Excel Exports, App Integration & more
>> Get technology previously reserved for billion-dollar corporations, FREE
>>
>> http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk
>> _______________________________________________
>> Wrapper-user mailing list
>> Wra...@li...
>> https://lists.sourceforge.net/lists/listinfo/wrapper-user
>>
>>
>
>
> --
> --
> Casey Jordan
> easyDITA a product of Jorsek LLC
> "CaseyDJordan" on LinkedIn, Twitter & Facebook
> (585) 348 7399
> easydita.com
>
>
> This message is intended only for the use of the Addressee(s) and may
> contain information that is privileged, confidential, and/or exempt from
> disclosure under applicable law.  If you are not the intended recipient,
> please be advised that any disclosure  copying, distribution, or use of
> the information contained herein is prohibited.  If you have received
> this communication in error, please destroy all copies of the message,
> whether in electronic or hard copy format, as well as attachments, and
> immediately contact the sender by replying to this e-mail or by phone.
> Thank you.
>
>
> ------------------------------------------------------------------------------
> Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
> from Actuate! Instantly Supercharge Your Business Reports and Dashboards
> with Interactivity, Sharing, Native Excel Exports, App Integration & more
> Get technology previously reserved for billion-dollar corporations, FREE
>
> http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk
> _______________________________________________
> Wrapper-user mailing list
> Wra...@li...
> https://lists.sourceforge.net/lists/listinfo/wrapper-user
>
>

Re: [Wrapper-user] Advice on stopping wrapper from triggering JVM restarts

[Casey J. <cas...@jo...>]

Cool thanks, will I be able to see this memory usage increase by looking at
the top command?

I would like to keep track of this a little more and see if I notice the
wrapper memory growing.

Thanks

On Wed, Dec 10, 2014 at 11:06 AM, Tim Lammens <tim...@gm...> wrote:

> That is indeed the right thread.
>
> Regards,
> Tim
>
> On Wed, Dec 10, 2014 at 5:03 PM, Casey Jordan <cas...@jo...>
> wrote:
>
>> Hi Tim,
>>
>> Thanks for the info, is this the thread you speak of:
>> http://sourceforge.net/p/wrapper/mailman/message/33048371/
>>
>> If not could you point me to the right one?
>>
>> Thanks
>>
>> On Wed, Dec 10, 2014 at 10:58 AM, Tim Lammens <tim...@gm...>
>> wrote:
>>
>>> You are most probably experiencing a glibc bug as I mentioned before in
>>> another thread. The bug was fixed in recent glibc versions.
>>> Try updating your glibc version for CentOS. If that is not possible you
>>> can apply the patches I provided in a different thread on this mailing list.
>>>
>>> Regards,
>>> Tim
>>>
>>> On Wed, Dec 10, 2014 at 4:49 PM, Casey Jordan <cas...@jo...>
>>> wrote:
>>>
>>>> Ok so this happened again and I was able to capture some additional log
>>>> information (not from wrapper.debug though as I can't update this server
>>>> just yet)
>>>>
>>>> I got this in our logs:
>>>>
>>>> INFO   | jvm 1    | 2014/12/09 23:15:50 | <br></br>Max Mem: 2223 MB
>>>> INFO   | jvm 1    | 2014/12/09 23:15:50 | <br></br>Total Mem: 2204 MB
>>>> INFO   | jvm 1    | 2014/12/09 23:15:50 | <br></br>Used Mem: 1141 MB
>>>> INFO   | jvm 1    | 2014/12/09 23:15:50 | <br></br>Raw free-mem: 1082 MB
>>>> INFO   | jvm 1    | 2014/12/09 23:15:50 | <br></br>Total descriptors: 4096
>>>> INFO   | jvm 1    | 2014/12/09 23:15:50 | <br></br>Open descriptors: 556
>>>> INFO   | jvm 1    | 2014/12/09 23:15:50 | <br></br>Available descriptors: 3540
>>>> INFO   | jvm 1    | 2014/12/09 23:15:50 | <br></br>Free disk space: 18 GB
>>>> INFO   | jvm 1    | 2014/12/09 23:15:50 | <br></br>
>>>> INFO   | wrapper  | 2014/12/09 23:17:09 | Wrapper Process has not received any CPU time for 1 seconds.  Extending timeouts.
>>>> STATUS | wrapper  | 2014/12/09 23:17:09 | Pinging the JVM took 7 seconds to respond.
>>>> INFO   | wrapper  | 2014/12/09 23:17:09 | Wrapper Process has not received any CPU time for 5 seconds.  Extending timeouts.
>>>> STATUS | wrapper  | 2014/12/09 23:17:09 | JVM received a signal SIGKILL (9).
>>>> STATUS | wrapper  | 2014/12/09 23:17:09 | JVM process is gone.
>>>> ERROR  | wrapper  | 2014/12/09 23:17:09 | JVM exited unexpectedly.
>>>> STATUS | wrapper  | 2014/12/09 23:17:09 | Automatic JVM Restarts disabled.  Shutting down.
>>>> STATUS | wrapper  | 2014/12/09 23:17:09 | <-- Wrapper Stopped
>>>>
>>>>
>>>> and I found this in the /var/log/messages
>>>>
>>>> Dec  9 23:17:09 jorsek-home2 kernel: Out of memory: Kill process 10262
>>>> (java) score 591 or sacrifice child
>>>> Dec  9 23:17:09 jorsek-home2 kernel: Killed process 10262 (java)
>>>> total-vm:6478728kB, anon-rss:2600344kB, file-rss:0kB
>>>>
>>>> So this does appear to be the kernel killing the process, but according
>>>> to java I have plenty of memory available (>1GB).
>>>>
>>>> I know this is not really a wrapper issue any longer, but I would be
>>>> very appreciative if anyone can give me some advice or point me to some
>>>> resources that might help figure this out.
>>>>
>>>> For reference this is a Centos7 box with 4GB of RAM and a 1GB swap
>>>> space, and 40GB SSD drive.
>>>>
>>>> Tonight I am going to add NewRelic monitoring to this server to see if
>>>> I can get a better picture of what is going on.
>>>>
>>>> Thanks!
>>>>
>>>> On Tue, Dec 2, 2014 at 7:30 PM, Roberto Espinoza <
>>>> rob...@ta...> wrote:
>>>>
>>>>> Casey,
>>>>>
>>>>> Have you checked the kernel logs to see if it wasn't killed because
>>>>> your system ran out of memory?
>>>>>
>>>>> Certainly there is no way to know this inside the JVM because the
>>>>> kernel out of memory killer will just score the current processes and
>>>>> issue a kill.
>>>>>
>>>>> On Wed, Dec 3, 2014 at 12:39 AM, Casey Jordan <cas...@jo...>
>>>>> wrote:
>>>>> > Hi Leif,
>>>>> >
>>>>> > Thanks for the feedback. This is quite a mystery then because I am
>>>>> 100% sure
>>>>> > that the process wasn't killed manually or by any other systems we
>>>>> have
>>>>> > running. (This is just a standard CentOS 7 setup)
>>>>> >
>>>>> > At about the same time that this log appeared I was running a very
>>>>> heavy
>>>>> > build process in a separate JVM instance.
>>>>> >
>>>>> > It seems more than just coincidental that this happened at the same
>>>>> time. I
>>>>> > assume that if the JVM crashed it wouldn't have received a SIG 9?
>>>>> Can you
>>>>> > think of any scenario where this makes sense? Perhaps I am missing
>>>>> some
>>>>> > knowledge of java architecture here that is making this mysterious.
>>>>> >
>>>>> > Thanks
>>>>> >
>>>>> >
>>>>> > On Tue, Dec 2, 2014 at 4:27 AM, Leif Mortenson
>>>>> > <lei...@ta...> wrote:
>>>>> >>
>>>>> >> Casey,
>>>>> >> The configuration you specified will disable all pings and cpu
>>>>> warnings
>>>>> >> and should do what you want.
>>>>> >> The Wrapper should never timeout  in this case.  There are other
>>>>> things
>>>>> >> such as a JVM crash would would still result in a restart after the
>>>>> fact.
>>>>> >>
>>>>> >> The log that you send shows that the JVM will killed with a kill
>>>>> -9.  Not
>>>>> >> sure if that was a test on your part, but if the source of the
>>>>> signal has
>>>>> >> permission to kill the JVM's process then there is nothing we can
>>>>> do to
>>>>> >> prevent that.
>>>>> >>
>>>>> >> Cheers,
>>>>> >> Leif
>>>>> >>
>>>>> >> On Tue, Dec 2, 2014 at 12:16 PM, Casey Jordan <
>>>>> cas...@jo...>
>>>>> >> wrote:
>>>>> >>>
>>>>> >>> Hi all,
>>>>> >>>
>>>>> >>> I am sure this is a common question, but I am having trouble
>>>>> extracting
>>>>> >>> details I need from the documentation. From time to time we get a
>>>>> situation
>>>>> >>> where our JVM gets killed and we see something like this in the
>>>>> logs:
>>>>> >>>
>>>>> >>> INFO   | wrapper  | 2014/12/02 02:23:18 | Wrapper Process has not
>>>>> >>> received any CPU time for 1 seconds.  Extending timeouts.
>>>>> >>> STATUS | wrapper  | 2014/12/02 02:23:37 | JVM received a signal
>>>>> SIGKILL
>>>>> >>> (9).
>>>>> >>> STATUS | wrapper  | 2014/12/02 02:23:37 | JVM process is gone.
>>>>> >>> ERROR  | wrapper  | 2014/12/02 02:23:37 | JVM exited unexpectedly.
>>>>> >>> STATUS | wrapper  | 2014/12/02 02:23:38 | Automatic JVM Restarts
>>>>> >>> disabled.  Shutting down.
>>>>> >>> STATUS | wrapper  | 2014/12/02 02:23:38 | <-- Wrapper Stopped
>>>>> >>>
>>>>> >>>
>>>>> >>> This always happens in the case where some other process is eating
>>>>> up all
>>>>> >>> the cpu for more than a few seconds.
>>>>> >>>
>>>>> >>> It's crucial for us that the wrapper never, ever restarts the JVM
>>>>> as this
>>>>> >>> is something we always want to look into manually.
>>>>> >>>
>>>>> >>> So my question is, what is the best way to make sure this doesn't
>>>>> happen?
>>>>> >>>
>>>>> >>> I am using community version 3.5.17, and have the following values
>>>>> in our
>>>>> >>> wrapper.conf:
>>>>> >>>
>>>>> >>>
>>>>> #********************************************************************
>>>>> >>>    # Timeouts
>>>>> >>>
>>>>> #********************************************************************
>>>>> >>>    wrapper.ping.timeout=0
>>>>> >>>    wrapper.ping.timeout.action=DEBUG
>>>>> >>>    wrapper.cpu.timeout=0
>>>>> >>>    wrapper.startup.timeout=300
>>>>> >>>
>>>>> >>> Any help is much appreciated.
>>>>> >>>
>>>>> >>> Thanks!
>>>>> >>>
>>>>> >>
>>>>> >>
>>>>> >>
>>>>> ------------------------------------------------------------------------------
>>>>> >> Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
>>>>> >> from Actuate! Instantly Supercharge Your Business Reports and
>>>>> Dashboards
>>>>> >> with Interactivity, Sharing, Native Excel Exports, App Integration
>>>>> & more
>>>>> >> Get technology previously reserved for billion-dollar corporations,
>>>>> FREE
>>>>> >>
>>>>> >>
>>>>> http://pubads.g.doubleclick.net/gampad/clk?id=157005751&iu=/4140/ostg.clktrk
>>>>> >> _______________________________________________
>>>>> >> Wrapper-user mailing list
>>>>> >> Wra...@li...
>>>>> >> https://lists.sourceforge.net/lists/listinfo/wrapper-user
>>>>> >>
>>>>> >
>>>>> >
>>>>> >
>>>>> > --
>>>>> > --
>>>>> > Casey Jordan
>>>>> > easyDITA a product of Jorsek LLC
>>>>> > "CaseyDJordan" on LinkedIn, Twitter & Facebook
>>>>> > (585) 348 7399
>>>>> > easydita.com
>>>>> >
>>>>> >
>>>>> > This message is intended only for the use of the Addressee(s) and may
>>>>> > contain information that is privileged, confidential, and/or exempt
>>>>> from
>>>>> > disclosure under applicable law.  If you are not the intended
>>>>> recipient,
>>>>> > please be advised that any disclosure  copying, distribution, or use
>>>>> of
>>>>> > the information contained herein is prohibited.  If you have received
>>>>> > this communication in error, please destroy all copies of the
>>>>> message,
>>>>> > whether in electronic or hard copy format, as well as attachments,
>>>>> and
>>>>> > immediately contact the sender by replying to this e-mail or by
>>>>> phone.
>>>>> > Thank you.
>>>>> >
>>>>> >
>>>>> ------------------------------------------------------------------------------
>>>>> > Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
>>>>> > from Actuate! Instantly Supercharge Your Business Reports and
>>>>> Dashboards
>>>>> > with Interactivity, Sharing, Native Excel Exports, App Integration &
>>>>> more
>>>>> > Get technology previously reserved for billion-dollar corporations,
>>>>> FREE
>>>>> >
>>>>> http://pubads.g.doubleclick.net/gampad/clk?id=157005751&iu=/4140/ostg.clktrk
>>>>> > _______________________________________________
>>>>> > Wrapper-user mailing list
>>>>> > Wra...@li...
>>>>> > https://lists.sourceforge.net/lists/listinfo/wrapper-user
>>>>> >
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Roberto Espinoza
>>>>> Tanuki Software, Ltd.
>>>>> 6-16-7-1001 Nishi-Kasai, Edogawa-ku
>>>>> Tokyo 134-0088 Japan
>>>>> Tel/Fax: +81-3-3878-3211
>>>>> http://www.tanukisoftware.com
>>>>> rob...@ta...
>>>>>
>>>>>
>>>>> ------------------------------------------------------------------------------
>>>>> Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
>>>>> from Actuate! Instantly Supercharge Your Business Reports and
>>>>> Dashboards
>>>>> with Interactivity, Sharing, Native Excel Exports, App Integration &
>>>>> more
>>>>> Get technology previously reserved for billion-dollar corporations,
>>>>> FREE
>>>>>
>>>>> http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk
>>>>> _______________________________________________
>>>>> Wrapper-user mailing list
>>>>> Wra...@li...
>>>>> https://lists.sourceforge.net/lists/listinfo/wrapper-user
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> --
>>>> Casey Jordan
>>>> easyDITA a product of Jorsek LLC
>>>> "CaseyDJordan" on LinkedIn, Twitter & Facebook
>>>> (585) 348 7399
>>>> easydita.com
>>>>
>>>>
>>>> This message is intended only for the use of the Addressee(s) and may
>>>> contain information that is privileged, confidential, and/or exempt from
>>>> disclosure under applicable law.  If you are not the intended recipient,
>>>> please be advised that any disclosure  copying, distribution, or use of
>>>> the information contained herein is prohibited.  If you have received
>>>> this communication in error, please destroy all copies of the message,
>>>> whether in electronic or hard copy format, as well as attachments, and
>>>> immediately contact the sender by replying to this e-mail or by phone.
>>>> Thank you.
>>>>
>>>>
>>>> ------------------------------------------------------------------------------
>>>> Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
>>>> from Actuate! Instantly Supercharge Your Business Reports and Dashboards
>>>> with Interactivity, Sharing, Native Excel Exports, App Integration &
>>>> more
>>>> Get technology previously reserved for billion-dollar corporations, FREE
>>>>
>>>> http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk
>>>> _______________________________________________
>>>> Wrapper-user mailing list
>>>> Wra...@li...
>>>> https://lists.sourceforge.net/lists/listinfo/wrapper-user
>>>>
>>>>
>>>
>>>
>>> ------------------------------------------------------------------------------
>>> Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
>>> from Actuate! Instantly Supercharge Your Business Reports and Dashboards
>>> with Interactivity, Sharing, Native Excel Exports, App Integration & more
>>> Get technology previously reserved for billion-dollar corporations, FREE
>>>
>>> http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk
>>> _______________________________________________
>>> Wrapper-user mailing list
>>> Wra...@li...
>>> https://lists.sourceforge.net/lists/listinfo/wrapper-user
>>>
>>>
>>
>>
>> --
>> --
>> Casey Jordan
>> easyDITA a product of Jorsek LLC
>> "CaseyDJordan" on LinkedIn, Twitter & Facebook
>> (585) 348 7399
>> easydita.com
>>
>>
>> This message is intended only for the use of the Addressee(s) and may
>> contain information that is privileged, confidential, and/or exempt from
>> disclosure under applicable law.  If you are not the intended recipient,
>> please be advised that any disclosure  copying, distribution, or use of
>> the information contained herein is prohibited.  If you have received
>> this communication in error, please destroy all copies of the message,
>> whether in electronic or hard copy format, as well as attachments, and
>> immediately contact the sender by replying to this e-mail or by phone.
>> Thank you.
>>
>>
>> ------------------------------------------------------------------------------
>> Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
>> from Actuate! Instantly Supercharge Your Business Reports and Dashboards
>> with Interactivity, Sharing, Native Excel Exports, App Integration & more
>> Get technology previously reserved for billion-dollar corporations, FREE
>>
>> http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk
>> _______________________________________________
>> Wrapper-user mailing list
>> Wra...@li...
>> https://lists.sourceforge.net/lists/listinfo/wrapper-user
>>
>>
>
>
> ------------------------------------------------------------------------------
> Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
> from Actuate! Instantly Supercharge Your Business Reports and Dashboards
> with Interactivity, Sharing, Native Excel Exports, App Integration & more
> Get technology previously reserved for billion-dollar corporations, FREE
>
> http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk
> _______________________________________________
> Wrapper-user mailing list
> Wra...@li...
> https://lists.sourceforge.net/lists/listinfo/wrapper-user
>
>


-- 
--
Casey Jordan
easyDITA a product of Jorsek LLC
"CaseyDJordan" on LinkedIn, Twitter & Facebook
(585) 348 7399
easydita.com


This message is intended only for the use of the Addressee(s) and may
contain information that is privileged, confidential, and/or exempt from
disclosure under applicable law.  If you are not the intended recipient,
please be advised that any disclosure  copying, distribution, or use of
the information contained herein is prohibited.  If you have received
this communication in error, please destroy all copies of the message,
whether in electronic or hard copy format, as well as attachments, and
immediately contact the sender by replying to this e-mail or by phone.
Thank you.

Re: [Wrapper-user] Advice on stopping wrapper from triggering JVM restarts

[Tim L. <tim...@gm...>]

That is indeed the right thread.

Regards,
Tim

On Wed, Dec 10, 2014 at 5:03 PM, Casey Jordan <cas...@jo...>
wrote:

> Hi Tim,
>
> Thanks for the info, is this the thread you speak of:
> http://sourceforge.net/p/wrapper/mailman/message/33048371/
>
> If not could you point me to the right one?
>
> Thanks
>
> On Wed, Dec 10, 2014 at 10:58 AM, Tim Lammens <tim...@gm...>
> wrote:
>
>> You are most probably experiencing a glibc bug as I mentioned before in
>> another thread. The bug was fixed in recent glibc versions.
>> Try updating your glibc version for CentOS. If that is not possible you
>> can apply the patches I provided in a different thread on this mailing list.
>>
>> Regards,
>> Tim
>>
>> On Wed, Dec 10, 2014 at 4:49 PM, Casey Jordan <cas...@jo...>
>> wrote:
>>
>>> Ok so this happened again and I was able to capture some additional log
>>> information (not from wrapper.debug though as I can't update this server
>>> just yet)
>>>
>>> I got this in our logs:
>>>
>>> INFO   | jvm 1    | 2014/12/09 23:15:50 | <br></br>Max Mem: 2223 MB
>>> INFO   | jvm 1    | 2014/12/09 23:15:50 | <br></br>Total Mem: 2204 MB
>>> INFO   | jvm 1    | 2014/12/09 23:15:50 | <br></br>Used Mem: 1141 MB
>>> INFO   | jvm 1    | 2014/12/09 23:15:50 | <br></br>Raw free-mem: 1082 MB
>>> INFO   | jvm 1    | 2014/12/09 23:15:50 | <br></br>Total descriptors: 4096
>>> INFO   | jvm 1    | 2014/12/09 23:15:50 | <br></br>Open descriptors: 556
>>> INFO   | jvm 1    | 2014/12/09 23:15:50 | <br></br>Available descriptors: 3540
>>> INFO   | jvm 1    | 2014/12/09 23:15:50 | <br></br>Free disk space: 18 GB
>>> INFO   | jvm 1    | 2014/12/09 23:15:50 | <br></br>
>>> INFO   | wrapper  | 2014/12/09 23:17:09 | Wrapper Process has not received any CPU time for 1 seconds.  Extending timeouts.
>>> STATUS | wrapper  | 2014/12/09 23:17:09 | Pinging the JVM took 7 seconds to respond.
>>> INFO   | wrapper  | 2014/12/09 23:17:09 | Wrapper Process has not received any CPU time for 5 seconds.  Extending timeouts.
>>> STATUS | wrapper  | 2014/12/09 23:17:09 | JVM received a signal SIGKILL (9).
>>> STATUS | wrapper  | 2014/12/09 23:17:09 | JVM process is gone.
>>> ERROR  | wrapper  | 2014/12/09 23:17:09 | JVM exited unexpectedly.
>>> STATUS | wrapper  | 2014/12/09 23:17:09 | Automatic JVM Restarts disabled.  Shutting down.
>>> STATUS | wrapper  | 2014/12/09 23:17:09 | <-- Wrapper Stopped
>>>
>>>
>>> and I found this in the /var/log/messages
>>>
>>> Dec  9 23:17:09 jorsek-home2 kernel: Out of memory: Kill process 10262
>>> (java) score 591 or sacrifice child
>>> Dec  9 23:17:09 jorsek-home2 kernel: Killed process 10262 (java)
>>> total-vm:6478728kB, anon-rss:2600344kB, file-rss:0kB
>>>
>>> So this does appear to be the kernel killing the process, but according
>>> to java I have plenty of memory available (>1GB).
>>>
>>> I know this is not really a wrapper issue any longer, but I would be
>>> very appreciative if anyone can give me some advice or point me to some
>>> resources that might help figure this out.
>>>
>>> For reference this is a Centos7 box with 4GB of RAM and a 1GB swap
>>> space, and 40GB SSD drive.
>>>
>>> Tonight I am going to add NewRelic monitoring to this server to see if I
>>> can get a better picture of what is going on.
>>>
>>> Thanks!
>>>
>>> On Tue, Dec 2, 2014 at 7:30 PM, Roberto Espinoza <
>>> rob...@ta...> wrote:
>>>
>>>> Casey,
>>>>
>>>> Have you checked the kernel logs to see if it wasn't killed because
>>>> your system ran out of memory?
>>>>
>>>> Certainly there is no way to know this inside the JVM because the
>>>> kernel out of memory killer will just score the current processes and
>>>> issue a kill.
>>>>
>>>> On Wed, Dec 3, 2014 at 12:39 AM, Casey Jordan <cas...@jo...>
>>>> wrote:
>>>> > Hi Leif,
>>>> >
>>>> > Thanks for the feedback. This is quite a mystery then because I am
>>>> 100% sure
>>>> > that the process wasn't killed manually or by any other systems we
>>>> have
>>>> > running. (This is just a standard CentOS 7 setup)
>>>> >
>>>> > At about the same time that this log appeared I was running a very
>>>> heavy
>>>> > build process in a separate JVM instance.
>>>> >
>>>> > It seems more than just coincidental that this happened at the same
>>>> time. I
>>>> > assume that if the JVM crashed it wouldn't have received a SIG 9? Can
>>>> you
>>>> > think of any scenario where this makes sense? Perhaps I am missing
>>>> some
>>>> > knowledge of java architecture here that is making this mysterious.
>>>> >
>>>> > Thanks
>>>> >
>>>> >
>>>> > On Tue, Dec 2, 2014 at 4:27 AM, Leif Mortenson
>>>> > <lei...@ta...> wrote:
>>>> >>
>>>> >> Casey,
>>>> >> The configuration you specified will disable all pings and cpu
>>>> warnings
>>>> >> and should do what you want.
>>>> >> The Wrapper should never timeout  in this case.  There are other
>>>> things
>>>> >> such as a JVM crash would would still result in a restart after the
>>>> fact.
>>>> >>
>>>> >> The log that you send shows that the JVM will killed with a kill
>>>> -9.  Not
>>>> >> sure if that was a test on your part, but if the source of the
>>>> signal has
>>>> >> permission to kill the JVM's process then there is nothing we can do
>>>> to
>>>> >> prevent that.
>>>> >>
>>>> >> Cheers,
>>>> >> Leif
>>>> >>
>>>> >> On Tue, Dec 2, 2014 at 12:16 PM, Casey Jordan <
>>>> cas...@jo...>
>>>> >> wrote:
>>>> >>>
>>>> >>> Hi all,
>>>> >>>
>>>> >>> I am sure this is a common question, but I am having trouble
>>>> extracting
>>>> >>> details I need from the documentation. From time to time we get a
>>>> situation
>>>> >>> where our JVM gets killed and we see something like this in the
>>>> logs:
>>>> >>>
>>>> >>> INFO   | wrapper  | 2014/12/02 02:23:18 | Wrapper Process has not
>>>> >>> received any CPU time for 1 seconds.  Extending timeouts.
>>>> >>> STATUS | wrapper  | 2014/12/02 02:23:37 | JVM received a signal
>>>> SIGKILL
>>>> >>> (9).
>>>> >>> STATUS | wrapper  | 2014/12/02 02:23:37 | JVM process is gone.
>>>> >>> ERROR  | wrapper  | 2014/12/02 02:23:37 | JVM exited unexpectedly.
>>>> >>> STATUS | wrapper  | 2014/12/02 02:23:38 | Automatic JVM Restarts
>>>> >>> disabled.  Shutting down.
>>>> >>> STATUS | wrapper  | 2014/12/02 02:23:38 | <-- Wrapper Stopped
>>>> >>>
>>>> >>>
>>>> >>> This always happens in the case where some other process is eating
>>>> up all
>>>> >>> the cpu for more than a few seconds.
>>>> >>>
>>>> >>> It's crucial for us that the wrapper never, ever restarts the JVM
>>>> as this
>>>> >>> is something we always want to look into manually.
>>>> >>>
>>>> >>> So my question is, what is the best way to make sure this doesn't
>>>> happen?
>>>> >>>
>>>> >>> I am using community version 3.5.17, and have the following values
>>>> in our
>>>> >>> wrapper.conf:
>>>> >>>
>>>> >>>
>>>> #********************************************************************
>>>> >>>    # Timeouts
>>>> >>>
>>>> #********************************************************************
>>>> >>>    wrapper.ping.timeout=0
>>>> >>>    wrapper.ping.timeout.action=DEBUG
>>>> >>>    wrapper.cpu.timeout=0
>>>> >>>    wrapper.startup.timeout=300
>>>> >>>
>>>> >>> Any help is much appreciated.
>>>> >>>
>>>> >>> Thanks!
>>>> >>>
>>>> >>
>>>> >>
>>>> >>
>>>> ------------------------------------------------------------------------------
>>>> >> Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
>>>> >> from Actuate! Instantly Supercharge Your Business Reports and
>>>> Dashboards
>>>> >> with Interactivity, Sharing, Native Excel Exports, App Integration &
>>>> more
>>>> >> Get technology previously reserved for billion-dollar corporations,
>>>> FREE
>>>> >>
>>>> >>
>>>> http://pubads.g.doubleclick.net/gampad/clk?id=157005751&iu=/4140/ostg.clktrk
>>>> >> _______________________________________________
>>>> >> Wrapper-user mailing list
>>>> >> Wra...@li...
>>>> >> https://lists.sourceforge.net/lists/listinfo/wrapper-user
>>>> >>
>>>> >
>>>> >
>>>> >
>>>> > --
>>>> > --
>>>> > Casey Jordan
>>>> > easyDITA a product of Jorsek LLC
>>>> > "CaseyDJordan" on LinkedIn, Twitter & Facebook
>>>> > (585) 348 7399
>>>> > easydita.com
>>>> >
>>>> >
>>>> > This message is intended only for the use of the Addressee(s) and may
>>>> > contain information that is privileged, confidential, and/or exempt
>>>> from
>>>> > disclosure under applicable law.  If you are not the intended
>>>> recipient,
>>>> > please be advised that any disclosure  copying, distribution, or use
>>>> of
>>>> > the information contained herein is prohibited.  If you have received
>>>> > this communication in error, please destroy all copies of the message,
>>>> > whether in electronic or hard copy format, as well as attachments, and
>>>> > immediately contact the sender by replying to this e-mail or by phone.
>>>> > Thank you.
>>>> >
>>>> >
>>>> ------------------------------------------------------------------------------
>>>> > Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
>>>> > from Actuate! Instantly Supercharge Your Business Reports and
>>>> Dashboards
>>>> > with Interactivity, Sharing, Native Excel Exports, App Integration &
>>>> more
>>>> > Get technology previously reserved for billion-dollar corporations,
>>>> FREE
>>>> >
>>>> http://pubads.g.doubleclick.net/gampad/clk?id=157005751&iu=/4140/ostg.clktrk
>>>> > _______________________________________________
>>>> > Wrapper-user mailing list
>>>> > Wra...@li...
>>>> > https://lists.sourceforge.net/lists/listinfo/wrapper-user
>>>> >
>>>>
>>>>
>>>>
>>>> --
>>>> Roberto Espinoza
>>>> Tanuki Software, Ltd.
>>>> 6-16-7-1001 Nishi-Kasai, Edogawa-ku
>>>> Tokyo 134-0088 Japan
>>>> Tel/Fax: +81-3-3878-3211
>>>> http://www.tanukisoftware.com
>>>> rob...@ta...
>>>>
>>>>
>>>> ------------------------------------------------------------------------------
>>>> Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
>>>> from Actuate! Instantly Supercharge Your Business Reports and Dashboards
>>>> with Interactivity, Sharing, Native Excel Exports, App Integration &
>>>> more
>>>> Get technology previously reserved for billion-dollar corporations, FREE
>>>>
>>>> http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk
>>>> _______________________________________________
>>>> Wrapper-user mailing list
>>>> Wra...@li...
>>>> https://lists.sourceforge.net/lists/listinfo/wrapper-user
>>>>
>>>
>>>
>>>
>>> --
>>> --
>>> Casey Jordan
>>> easyDITA a product of Jorsek LLC
>>> "CaseyDJordan" on LinkedIn, Twitter & Facebook
>>> (585) 348 7399
>>> easydita.com
>>>
>>>
>>> This message is intended only for the use of the Addressee(s) and may
>>> contain information that is privileged, confidential, and/or exempt from
>>> disclosure under applicable law.  If you are not the intended recipient,
>>> please be advised that any disclosure  copying, distribution, or use of
>>> the information contained herein is prohibited.  If you have received
>>> this communication in error, please destroy all copies of the message,
>>> whether in electronic or hard copy format, as well as attachments, and
>>> immediately contact the sender by replying to this e-mail or by phone.
>>> Thank you.
>>>
>>>
>>> ------------------------------------------------------------------------------
>>> Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
>>> from Actuate! Instantly Supercharge Your Business Reports and Dashboards
>>> with Interactivity, Sharing, Native Excel Exports, App Integration & more
>>> Get technology previously reserved for billion-dollar corporations, FREE
>>>
>>> http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk
>>> _______________________________________________
>>> Wrapper-user mailing list
>>> Wra...@li...
>>> https://lists.sourceforge.net/lists/listinfo/wrapper-user
>>>
>>>
>>
>>
>> ------------------------------------------------------------------------------
>> Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
>> from Actuate! Instantly Supercharge Your Business Reports and Dashboards
>> with Interactivity, Sharing, Native Excel Exports, App Integration & more
>> Get technology previously reserved for billion-dollar corporations, FREE
>>
>> http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk
>> _______________________________________________
>> Wrapper-user mailing list
>> Wra...@li...
>> https://lists.sourceforge.net/lists/listinfo/wrapper-user
>>
>>
>
>
> --
> --
> Casey Jordan
> easyDITA a product of Jorsek LLC
> "CaseyDJordan" on LinkedIn, Twitter & Facebook
> (585) 348 7399
> easydita.com
>
>
> This message is intended only for the use of the Addressee(s) and may
> contain information that is privileged, confidential, and/or exempt from
> disclosure under applicable law.  If you are not the intended recipient,
> please be advised that any disclosure  copying, distribution, or use of
> the information contained herein is prohibited.  If you have received
> this communication in error, please destroy all copies of the message,
> whether in electronic or hard copy format, as well as attachments, and
> immediately contact the sender by replying to this e-mail or by phone.
> Thank you.
>
>
> ------------------------------------------------------------------------------
> Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
> from Actuate! Instantly Supercharge Your Business Reports and Dashboards
> with Interactivity, Sharing, Native Excel Exports, App Integration & more
> Get technology previously reserved for billion-dollar corporations, FREE
>
> http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk
> _______________________________________________
> Wrapper-user mailing list
> Wra...@li...
> https://lists.sourceforge.net/lists/listinfo/wrapper-user
>
>

Re: [Wrapper-user] Advice on stopping wrapper from triggering JVM restarts

[Casey J. <cas...@jo...>]

By the way, this is my glibc version: glibc-2.17-55.el7_0.1.x86_64

On Wed, Dec 10, 2014 at 11:03 AM, Casey Jordan <cas...@jo...>
wrote:

> Hi Tim,
>
> Thanks for the info, is this the thread you speak of:
> http://sourceforge.net/p/wrapper/mailman/message/33048371/
>
> If not could you point me to the right one?
>
> Thanks
>
> On Wed, Dec 10, 2014 at 10:58 AM, Tim Lammens <tim...@gm...>
> wrote:
>
>> You are most probably experiencing a glibc bug as I mentioned before in
>> another thread. The bug was fixed in recent glibc versions.
>> Try updating your glibc version for CentOS. If that is not possible you
>> can apply the patches I provided in a different thread on this mailing list.
>>
>> Regards,
>> Tim
>>
>> On Wed, Dec 10, 2014 at 4:49 PM, Casey Jordan <cas...@jo...>
>> wrote:
>>
>>> Ok so this happened again and I was able to capture some additional log
>>> information (not from wrapper.debug though as I can't update this server
>>> just yet)
>>>
>>> I got this in our logs:
>>>
>>> INFO   | jvm 1    | 2014/12/09 23:15:50 | <br></br>Max Mem: 2223 MB
>>> INFO   | jvm 1    | 2014/12/09 23:15:50 | <br></br>Total Mem: 2204 MB
>>> INFO   | jvm 1    | 2014/12/09 23:15:50 | <br></br>Used Mem: 1141 MB
>>> INFO   | jvm 1    | 2014/12/09 23:15:50 | <br></br>Raw free-mem: 1082 MB
>>> INFO   | jvm 1    | 2014/12/09 23:15:50 | <br></br>Total descriptors: 4096
>>> INFO   | jvm 1    | 2014/12/09 23:15:50 | <br></br>Open descriptors: 556
>>> INFO   | jvm 1    | 2014/12/09 23:15:50 | <br></br>Available descriptors: 3540
>>> INFO   | jvm 1    | 2014/12/09 23:15:50 | <br></br>Free disk space: 18 GB
>>> INFO   | jvm 1    | 2014/12/09 23:15:50 | <br></br>
>>> INFO   | wrapper  | 2014/12/09 23:17:09 | Wrapper Process has not received any CPU time for 1 seconds.  Extending timeouts.
>>> STATUS | wrapper  | 2014/12/09 23:17:09 | Pinging the JVM took 7 seconds to respond.
>>> INFO   | wrapper  | 2014/12/09 23:17:09 | Wrapper Process has not received any CPU time for 5 seconds.  Extending timeouts.
>>> STATUS | wrapper  | 2014/12/09 23:17:09 | JVM received a signal SIGKILL (9).
>>> STATUS | wrapper  | 2014/12/09 23:17:09 | JVM process is gone.
>>> ERROR  | wrapper  | 2014/12/09 23:17:09 | JVM exited unexpectedly.
>>> STATUS | wrapper  | 2014/12/09 23:17:09 | Automatic JVM Restarts disabled.  Shutting down.
>>> STATUS | wrapper  | 2014/12/09 23:17:09 | <-- Wrapper Stopped
>>>
>>>
>>> and I found this in the /var/log/messages
>>>
>>> Dec  9 23:17:09 jorsek-home2 kernel: Out of memory: Kill process 10262
>>> (java) score 591 or sacrifice child
>>> Dec  9 23:17:09 jorsek-home2 kernel: Killed process 10262 (java)
>>> total-vm:6478728kB, anon-rss:2600344kB, file-rss:0kB
>>>
>>> So this does appear to be the kernel killing the process, but according
>>> to java I have plenty of memory available (>1GB).
>>>
>>> I know this is not really a wrapper issue any longer, but I would be
>>> very appreciative if anyone can give me some advice or point me to some
>>> resources that might help figure this out.
>>>
>>> For reference this is a Centos7 box with 4GB of RAM and a 1GB swap
>>> space, and 40GB SSD drive.
>>>
>>> Tonight I am going to add NewRelic monitoring to this server to see if I
>>> can get a better picture of what is going on.
>>>
>>> Thanks!
>>>
>>> On Tue, Dec 2, 2014 at 7:30 PM, Roberto Espinoza <
>>> rob...@ta...> wrote:
>>>
>>>> Casey,
>>>>
>>>> Have you checked the kernel logs to see if it wasn't killed because
>>>> your system ran out of memory?
>>>>
>>>> Certainly there is no way to know this inside the JVM because the
>>>> kernel out of memory killer will just score the current processes and
>>>> issue a kill.
>>>>
>>>> On Wed, Dec 3, 2014 at 12:39 AM, Casey Jordan <cas...@jo...>
>>>> wrote:
>>>> > Hi Leif,
>>>> >
>>>> > Thanks for the feedback. This is quite a mystery then because I am
>>>> 100% sure
>>>> > that the process wasn't killed manually or by any other systems we
>>>> have
>>>> > running. (This is just a standard CentOS 7 setup)
>>>> >
>>>> > At about the same time that this log appeared I was running a very
>>>> heavy
>>>> > build process in a separate JVM instance.
>>>> >
>>>> > It seems more than just coincidental that this happened at the same
>>>> time. I
>>>> > assume that if the JVM crashed it wouldn't have received a SIG 9? Can
>>>> you
>>>> > think of any scenario where this makes sense? Perhaps I am missing
>>>> some
>>>> > knowledge of java architecture here that is making this mysterious.
>>>> >
>>>> > Thanks
>>>> >
>>>> >
>>>> > On Tue, Dec 2, 2014 at 4:27 AM, Leif Mortenson
>>>> > <lei...@ta...> wrote:
>>>> >>
>>>> >> Casey,
>>>> >> The configuration you specified will disable all pings and cpu
>>>> warnings
>>>> >> and should do what you want.
>>>> >> The Wrapper should never timeout  in this case.  There are other
>>>> things
>>>> >> such as a JVM crash would would still result in a restart after the
>>>> fact.
>>>> >>
>>>> >> The log that you send shows that the JVM will killed with a kill
>>>> -9.  Not
>>>> >> sure if that was a test on your part, but if the source of the
>>>> signal has
>>>> >> permission to kill the JVM's process then there is nothing we can do
>>>> to
>>>> >> prevent that.
>>>> >>
>>>> >> Cheers,
>>>> >> Leif
>>>> >>
>>>> >> On Tue, Dec 2, 2014 at 12:16 PM, Casey Jordan <
>>>> cas...@jo...>
>>>> >> wrote:
>>>> >>>
>>>> >>> Hi all,
>>>> >>>
>>>> >>> I am sure this is a common question, but I am having trouble
>>>> extracting
>>>> >>> details I need from the documentation. From time to time we get a
>>>> situation
>>>> >>> where our JVM gets killed and we see something like this in the
>>>> logs:
>>>> >>>
>>>> >>> INFO   | wrapper  | 2014/12/02 02:23:18 | Wrapper Process has not
>>>> >>> received any CPU time for 1 seconds.  Extending timeouts.
>>>> >>> STATUS | wrapper  | 2014/12/02 02:23:37 | JVM received a signal
>>>> SIGKILL
>>>> >>> (9).
>>>> >>> STATUS | wrapper  | 2014/12/02 02:23:37 | JVM process is gone.
>>>> >>> ERROR  | wrapper  | 2014/12/02 02:23:37 | JVM exited unexpectedly.
>>>> >>> STATUS | wrapper  | 2014/12/02 02:23:38 | Automatic JVM Restarts
>>>> >>> disabled.  Shutting down.
>>>> >>> STATUS | wrapper  | 2014/12/02 02:23:38 | <-- Wrapper Stopped
>>>> >>>
>>>> >>>
>>>> >>> This always happens in the case where some other process is eating
>>>> up all
>>>> >>> the cpu for more than a few seconds.
>>>> >>>
>>>> >>> It's crucial for us that the wrapper never, ever restarts the JVM
>>>> as this
>>>> >>> is something we always want to look into manually.
>>>> >>>
>>>> >>> So my question is, what is the best way to make sure this doesn't
>>>> happen?
>>>> >>>
>>>> >>> I am using community version 3.5.17, and have the following values
>>>> in our
>>>> >>> wrapper.conf:
>>>> >>>
>>>> >>>
>>>> #********************************************************************
>>>> >>>    # Timeouts
>>>> >>>
>>>> #********************************************************************
>>>> >>>    wrapper.ping.timeout=0
>>>> >>>    wrapper.ping.timeout.action=DEBUG
>>>> >>>    wrapper.cpu.timeout=0
>>>> >>>    wrapper.startup.timeout=300
>>>> >>>
>>>> >>> Any help is much appreciated.
>>>> >>>
>>>> >>> Thanks!
>>>> >>>
>>>> >>
>>>> >>
>>>> >>
>>>> ------------------------------------------------------------------------------
>>>> >> Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
>>>> >> from Actuate! Instantly Supercharge Your Business Reports and
>>>> Dashboards
>>>> >> with Interactivity, Sharing, Native Excel Exports, App Integration &
>>>> more
>>>> >> Get technology previously reserved for billion-dollar corporations,
>>>> FREE
>>>> >>
>>>> >>
>>>> http://pubads.g.doubleclick.net/gampad/clk?id=157005751&iu=/4140/ostg.clktrk
>>>> >> _______________________________________________
>>>> >> Wrapper-user mailing list
>>>> >> Wra...@li...
>>>> >> https://lists.sourceforge.net/lists/listinfo/wrapper-user
>>>> >>
>>>> >
>>>> >
>>>> >
>>>> > --
>>>> > --
>>>> > Casey Jordan
>>>> > easyDITA a product of Jorsek LLC
>>>> > "CaseyDJordan" on LinkedIn, Twitter & Facebook
>>>> > (585) 348 7399
>>>> > easydita.com
>>>> >
>>>> >
>>>> > This message is intended only for the use of the Addressee(s) and may
>>>> > contain information that is privileged, confidential, and/or exempt
>>>> from
>>>> > disclosure under applicable law.  If you are not the intended
>>>> recipient,
>>>> > please be advised that any disclosure  copying, distribution, or use
>>>> of
>>>> > the information contained herein is prohibited.  If you have received
>>>> > this communication in error, please destroy all copies of the message,
>>>> > whether in electronic or hard copy format, as well as attachments, and
>>>> > immediately contact the sender by replying to this e-mail or by phone.
>>>> > Thank you.
>>>> >
>>>> >
>>>> ------------------------------------------------------------------------------
>>>> > Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
>>>> > from Actuate! Instantly Supercharge Your Business Reports and
>>>> Dashboards
>>>> > with Interactivity, Sharing, Native Excel Exports, App Integration &
>>>> more
>>>> > Get technology previously reserved for billion-dollar corporations,
>>>> FREE
>>>> >
>>>> http://pubads.g.doubleclick.net/gampad/clk?id=157005751&iu=/4140/ostg.clktrk
>>>> > _______________________________________________
>>>> > Wrapper-user mailing list
>>>> > Wra...@li...
>>>> > https://lists.sourceforge.net/lists/listinfo/wrapper-user
>>>> >
>>>>
>>>>
>>>>
>>>> --
>>>> Roberto Espinoza
>>>> Tanuki Software, Ltd.
>>>> 6-16-7-1001 Nishi-Kasai, Edogawa-ku
>>>> Tokyo 134-0088 Japan
>>>> Tel/Fax: +81-3-3878-3211
>>>> http://www.tanukisoftware.com
>>>> rob...@ta...
>>>>
>>>>
>>>> ------------------------------------------------------------------------------
>>>> Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
>>>> from Actuate! Instantly Supercharge Your Business Reports and Dashboards
>>>> with Interactivity, Sharing, Native Excel Exports, App Integration &
>>>> more
>>>> Get technology previously reserved for billion-dollar corporations, FREE
>>>>
>>>> http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk
>>>> _______________________________________________
>>>> Wrapper-user mailing list
>>>> Wra...@li...
>>>> https://lists.sourceforge.net/lists/listinfo/wrapper-user
>>>>
>>>
>>>
>>>
>>> --
>>> --
>>> Casey Jordan
>>> easyDITA a product of Jorsek LLC
>>> "CaseyDJordan" on LinkedIn, Twitter & Facebook
>>> (585) 348 7399
>>> easydita.com
>>>
>>>
>>> This message is intended only for the use of the Addressee(s) and may
>>> contain information that is privileged, confidential, and/or exempt from
>>> disclosure under applicable law.  If you are not the intended recipient,
>>> please be advised that any disclosure  copying, distribution, or use of
>>> the information contained herein is prohibited.  If you have received
>>> this communication in error, please destroy all copies of the message,
>>> whether in electronic or hard copy format, as well as attachments, and
>>> immediately contact the sender by replying to this e-mail or by phone.
>>> Thank you.
>>>
>>>
>>> ------------------------------------------------------------------------------
>>> Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
>>> from Actuate! Instantly Supercharge Your Business Reports and Dashboards
>>> with Interactivity, Sharing, Native Excel Exports, App Integration & more
>>> Get technology previously reserved for billion-dollar corporations, FREE
>>>
>>> http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk
>>> _______________________________________________
>>> Wrapper-user mailing list
>>> Wra...@li...
>>> https://lists.sourceforge.net/lists/listinfo/wrapper-user
>>>
>>>
>>
>>
>> ------------------------------------------------------------------------------
>> Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
>> from Actuate! Instantly Supercharge Your Business Reports and Dashboards
>> with Interactivity, Sharing, Native Excel Exports, App Integration & more
>> Get technology previously reserved for billion-dollar corporations, FREE
>>
>> http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk
>> _______________________________________________
>> Wrapper-user mailing list
>> Wra...@li...
>> https://lists.sourceforge.net/lists/listinfo/wrapper-user
>>
>>
>
>
> --
> --
> Casey Jordan
> easyDITA a product of Jorsek LLC
> "CaseyDJordan" on LinkedIn, Twitter & Facebook
> (585) 348 7399
> easydita.com
>
>
> This message is intended only for the use of the Addressee(s) and may
> contain information that is privileged, confidential, and/or exempt from
> disclosure under applicable law.  If you are not the intended recipient,
> please be advised that any disclosure  copying, distribution, or use of
> the information contained herein is prohibited.  If you have received
> this communication in error, please destroy all copies of the message,
> whether in electronic or hard copy format, as well as attachments, and
> immediately contact the sender by replying to this e-mail or by phone.
> Thank you.
>



-- 
--
Casey Jordan
easyDITA a product of Jorsek LLC
"CaseyDJordan" on LinkedIn, Twitter & Facebook
(585) 348 7399
easydita.com


This message is intended only for the use of the Addressee(s) and may
contain information that is privileged, confidential, and/or exempt from
disclosure under applicable law.  If you are not the intended recipient,
please be advised that any disclosure  copying, distribution, or use of
the information contained herein is prohibited.  If you have received
this communication in error, please destroy all copies of the message,
whether in electronic or hard copy format, as well as attachments, and
immediately contact the sender by replying to this e-mail or by phone.
Thank you.

Re: [Wrapper-user] Advice on stopping wrapper from triggering JVM restarts

[Casey J. <cas...@jo...>]

Hi Tim,

Thanks for the info, is this the thread you speak of:
http://sourceforge.net/p/wrapper/mailman/message/33048371/

If not could you point me to the right one?

Thanks

On Wed, Dec 10, 2014 at 10:58 AM, Tim Lammens <tim...@gm...> wrote:

> You are most probably experiencing a glibc bug as I mentioned before in
> another thread. The bug was fixed in recent glibc versions.
> Try updating your glibc version for CentOS. If that is not possible you
> can apply the patches I provided in a different thread on this mailing list.
>
> Regards,
> Tim
>
> On Wed, Dec 10, 2014 at 4:49 PM, Casey Jordan <cas...@jo...>
> wrote:
>
>> Ok so this happened again and I was able to capture some additional log
>> information (not from wrapper.debug though as I can't update this server
>> just yet)
>>
>> I got this in our logs:
>>
>> INFO   | jvm 1    | 2014/12/09 23:15:50 | <br></br>Max Mem: 2223 MB
>> INFO   | jvm 1    | 2014/12/09 23:15:50 | <br></br>Total Mem: 2204 MB
>> INFO   | jvm 1    | 2014/12/09 23:15:50 | <br></br>Used Mem: 1141 MB
>> INFO   | jvm 1    | 2014/12/09 23:15:50 | <br></br>Raw free-mem: 1082 MB
>> INFO   | jvm 1    | 2014/12/09 23:15:50 | <br></br>Total descriptors: 4096
>> INFO   | jvm 1    | 2014/12/09 23:15:50 | <br></br>Open descriptors: 556
>> INFO   | jvm 1    | 2014/12/09 23:15:50 | <br></br>Available descriptors: 3540
>> INFO   | jvm 1    | 2014/12/09 23:15:50 | <br></br>Free disk space: 18 GB
>> INFO   | jvm 1    | 2014/12/09 23:15:50 | <br></br>
>> INFO   | wrapper  | 2014/12/09 23:17:09 | Wrapper Process has not received any CPU time for 1 seconds.  Extending timeouts.
>> STATUS | wrapper  | 2014/12/09 23:17:09 | Pinging the JVM took 7 seconds to respond.
>> INFO   | wrapper  | 2014/12/09 23:17:09 | Wrapper Process has not received any CPU time for 5 seconds.  Extending timeouts.
>> STATUS | wrapper  | 2014/12/09 23:17:09 | JVM received a signal SIGKILL (9).
>> STATUS | wrapper  | 2014/12/09 23:17:09 | JVM process is gone.
>> ERROR  | wrapper  | 2014/12/09 23:17:09 | JVM exited unexpectedly.
>> STATUS | wrapper  | 2014/12/09 23:17:09 | Automatic JVM Restarts disabled.  Shutting down.
>> STATUS | wrapper  | 2014/12/09 23:17:09 | <-- Wrapper Stopped
>>
>>
>> and I found this in the /var/log/messages
>>
>> Dec  9 23:17:09 jorsek-home2 kernel: Out of memory: Kill process 10262
>> (java) score 591 or sacrifice child
>> Dec  9 23:17:09 jorsek-home2 kernel: Killed process 10262 (java)
>> total-vm:6478728kB, anon-rss:2600344kB, file-rss:0kB
>>
>> So this does appear to be the kernel killing the process, but according
>> to java I have plenty of memory available (>1GB).
>>
>> I know this is not really a wrapper issue any longer, but I would be very
>> appreciative if anyone can give me some advice or point me to some
>> resources that might help figure this out.
>>
>> For reference this is a Centos7 box with 4GB of RAM and a 1GB swap space,
>> and 40GB SSD drive.
>>
>> Tonight I am going to add NewRelic monitoring to this server to see if I
>> can get a better picture of what is going on.
>>
>> Thanks!
>>
>> On Tue, Dec 2, 2014 at 7:30 PM, Roberto Espinoza <
>> rob...@ta...> wrote:
>>
>>> Casey,
>>>
>>> Have you checked the kernel logs to see if it wasn't killed because
>>> your system ran out of memory?
>>>
>>> Certainly there is no way to know this inside the JVM because the
>>> kernel out of memory killer will just score the current processes and
>>> issue a kill.
>>>
>>> On Wed, Dec 3, 2014 at 12:39 AM, Casey Jordan <cas...@jo...>
>>> wrote:
>>> > Hi Leif,
>>> >
>>> > Thanks for the feedback. This is quite a mystery then because I am
>>> 100% sure
>>> > that the process wasn't killed manually or by any other systems we have
>>> > running. (This is just a standard CentOS 7 setup)
>>> >
>>> > At about the same time that this log appeared I was running a very
>>> heavy
>>> > build process in a separate JVM instance.
>>> >
>>> > It seems more than just coincidental that this happened at the same
>>> time. I
>>> > assume that if the JVM crashed it wouldn't have received a SIG 9? Can
>>> you
>>> > think of any scenario where this makes sense? Perhaps I am missing some
>>> > knowledge of java architecture here that is making this mysterious.
>>> >
>>> > Thanks
>>> >
>>> >
>>> > On Tue, Dec 2, 2014 at 4:27 AM, Leif Mortenson
>>> > <lei...@ta...> wrote:
>>> >>
>>> >> Casey,
>>> >> The configuration you specified will disable all pings and cpu
>>> warnings
>>> >> and should do what you want.
>>> >> The Wrapper should never timeout  in this case.  There are other
>>> things
>>> >> such as a JVM crash would would still result in a restart after the
>>> fact.
>>> >>
>>> >> The log that you send shows that the JVM will killed with a kill -9.
>>> Not
>>> >> sure if that was a test on your part, but if the source of the signal
>>> has
>>> >> permission to kill the JVM's process then there is nothing we can do
>>> to
>>> >> prevent that.
>>> >>
>>> >> Cheers,
>>> >> Leif
>>> >>
>>> >> On Tue, Dec 2, 2014 at 12:16 PM, Casey Jordan <
>>> cas...@jo...>
>>> >> wrote:
>>> >>>
>>> >>> Hi all,
>>> >>>
>>> >>> I am sure this is a common question, but I am having trouble
>>> extracting
>>> >>> details I need from the documentation. From time to time we get a
>>> situation
>>> >>> where our JVM gets killed and we see something like this in the logs:
>>> >>>
>>> >>> INFO   | wrapper  | 2014/12/02 02:23:18 | Wrapper Process has not
>>> >>> received any CPU time for 1 seconds.  Extending timeouts.
>>> >>> STATUS | wrapper  | 2014/12/02 02:23:37 | JVM received a signal
>>> SIGKILL
>>> >>> (9).
>>> >>> STATUS | wrapper  | 2014/12/02 02:23:37 | JVM process is gone.
>>> >>> ERROR  | wrapper  | 2014/12/02 02:23:37 | JVM exited unexpectedly.
>>> >>> STATUS | wrapper  | 2014/12/02 02:23:38 | Automatic JVM Restarts
>>> >>> disabled.  Shutting down.
>>> >>> STATUS | wrapper  | 2014/12/02 02:23:38 | <-- Wrapper Stopped
>>> >>>
>>> >>>
>>> >>> This always happens in the case where some other process is eating
>>> up all
>>> >>> the cpu for more than a few seconds.
>>> >>>
>>> >>> It's crucial for us that the wrapper never, ever restarts the JVM as
>>> this
>>> >>> is something we always want to look into manually.
>>> >>>
>>> >>> So my question is, what is the best way to make sure this doesn't
>>> happen?
>>> >>>
>>> >>> I am using community version 3.5.17, and have the following values
>>> in our
>>> >>> wrapper.conf:
>>> >>>
>>> >>>
>>> #********************************************************************
>>> >>>    # Timeouts
>>> >>>
>>> #********************************************************************
>>> >>>    wrapper.ping.timeout=0
>>> >>>    wrapper.ping.timeout.action=DEBUG
>>> >>>    wrapper.cpu.timeout=0
>>> >>>    wrapper.startup.timeout=300
>>> >>>
>>> >>> Any help is much appreciated.
>>> >>>
>>> >>> Thanks!
>>> >>>
>>> >>
>>> >>
>>> >>
>>> ------------------------------------------------------------------------------
>>> >> Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
>>> >> from Actuate! Instantly Supercharge Your Business Reports and
>>> Dashboards
>>> >> with Interactivity, Sharing, Native Excel Exports, App Integration &
>>> more
>>> >> Get technology previously reserved for billion-dollar corporations,
>>> FREE
>>> >>
>>> >>
>>> http://pubads.g.doubleclick.net/gampad/clk?id=157005751&iu=/4140/ostg.clktrk
>>> >> _______________________________________________
>>> >> Wrapper-user mailing list
>>> >> Wra...@li...
>>> >> https://lists.sourceforge.net/lists/listinfo/wrapper-user
>>> >>
>>> >
>>> >
>>> >
>>> > --
>>> > --
>>> > Casey Jordan
>>> > easyDITA a product of Jorsek LLC
>>> > "CaseyDJordan" on LinkedIn, Twitter & Facebook
>>> > (585) 348 7399
>>> > easydita.com
>>> >
>>> >
>>> > This message is intended only for the use of the Addressee(s) and may
>>> > contain information that is privileged, confidential, and/or exempt
>>> from
>>> > disclosure under applicable law.  If you are not the intended
>>> recipient,
>>> > please be advised that any disclosure  copying, distribution, or use of
>>> > the information contained herein is prohibited.  If you have received
>>> > this communication in error, please destroy all copies of the message,
>>> > whether in electronic or hard copy format, as well as attachments, and
>>> > immediately contact the sender by replying to this e-mail or by phone.
>>> > Thank you.
>>> >
>>> >
>>> ------------------------------------------------------------------------------
>>> > Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
>>> > from Actuate! Instantly Supercharge Your Business Reports and
>>> Dashboards
>>> > with Interactivity, Sharing, Native Excel Exports, App Integration &
>>> more
>>> > Get technology previously reserved for billion-dollar corporations,
>>> FREE
>>> >
>>> http://pubads.g.doubleclick.net/gampad/clk?id=157005751&iu=/4140/ostg.clktrk
>>> > _______________________________________________
>>> > Wrapper-user mailing list
>>> > Wra...@li...
>>> > https://lists.sourceforge.net/lists/listinfo/wrapper-user
>>> >
>>>
>>>
>>>
>>> --
>>> Roberto Espinoza
>>> Tanuki Software, Ltd.
>>> 6-16-7-1001 Nishi-Kasai, Edogawa-ku
>>> Tokyo 134-0088 Japan
>>> Tel/Fax: +81-3-3878-3211
>>> http://www.tanukisoftware.com
>>> rob...@ta...
>>>
>>>
>>> ------------------------------------------------------------------------------
>>> Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
>>> from Actuate! Instantly Supercharge Your Business Reports and Dashboards
>>> with Interactivity, Sharing, Native Excel Exports, App Integration & more
>>> Get technology previously reserved for billion-dollar corporations, FREE
>>>
>>> http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk
>>> _______________________________________________
>>> Wrapper-user mailing list
>>> Wra...@li...
>>> https://lists.sourceforge.net/lists/listinfo/wrapper-user
>>>
>>
>>
>>
>> --
>> --
>> Casey Jordan
>> easyDITA a product of Jorsek LLC
>> "CaseyDJordan" on LinkedIn, Twitter & Facebook
>> (585) 348 7399
>> easydita.com
>>
>>
>> This message is intended only for the use of the Addressee(s) and may
>> contain information that is privileged, confidential, and/or exempt from
>> disclosure under applicable law.  If you are not the intended recipient,
>> please be advised that any disclosure  copying, distribution, or use of
>> the information contained herein is prohibited.  If you have received
>> this communication in error, please destroy all copies of the message,
>> whether in electronic or hard copy format, as well as attachments, and
>> immediately contact the sender by replying to this e-mail or by phone.
>> Thank you.
>>
>>
>> ------------------------------------------------------------------------------
>> Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
>> from Actuate! Instantly Supercharge Your Business Reports and Dashboards
>> with Interactivity, Sharing, Native Excel Exports, App Integration & more
>> Get technology previously reserved for billion-dollar corporations, FREE
>>
>> http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk
>> _______________________________________________
>> Wrapper-user mailing list
>> Wra...@li...
>> https://lists.sourceforge.net/lists/listinfo/wrapper-user
>>
>>
>
>
> ------------------------------------------------------------------------------
> Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
> from Actuate! Instantly Supercharge Your Business Reports and Dashboards
> with Interactivity, Sharing, Native Excel Exports, App Integration & more
> Get technology previously reserved for billion-dollar corporations, FREE
>
> http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk
> _______________________________________________
> Wrapper-user mailing list
> Wra...@li...
> https://lists.sourceforge.net/lists/listinfo/wrapper-user
>
>


-- 
--
Casey Jordan
easyDITA a product of Jorsek LLC
"CaseyDJordan" on LinkedIn, Twitter & Facebook
(585) 348 7399
easydita.com


This message is intended only for the use of the Addressee(s) and may
contain information that is privileged, confidential, and/or exempt from
disclosure under applicable law.  If you are not the intended recipient,
please be advised that any disclosure  copying, distribution, or use of
the information contained herein is prohibited.  If you have received
this communication in error, please destroy all copies of the message,
whether in electronic or hard copy format, as well as attachments, and
immediately contact the sender by replying to this e-mail or by phone.
Thank you.

Re: [Wrapper-user] Advice on stopping wrapper from triggering JVM restarts

[Tim L. <tim...@gm...>]

You are most probably experiencing a glibc bug as I mentioned before in
another thread. The bug was fixed in recent glibc versions.
Try updating your glibc version for CentOS. If that is not possible you can
apply the patches I provided in a different thread on this mailing list.

Regards,
Tim

On Wed, Dec 10, 2014 at 4:49 PM, Casey Jordan <cas...@jo...>
wrote:

> Ok so this happened again and I was able to capture some additional log
> information (not from wrapper.debug though as I can't update this server
> just yet)
>
> I got this in our logs:
>
> INFO   | jvm 1    | 2014/12/09 23:15:50 | <br></br>Max Mem: 2223 MB
> INFO   | jvm 1    | 2014/12/09 23:15:50 | <br></br>Total Mem: 2204 MB
> INFO   | jvm 1    | 2014/12/09 23:15:50 | <br></br>Used Mem: 1141 MB
> INFO   | jvm 1    | 2014/12/09 23:15:50 | <br></br>Raw free-mem: 1082 MB
> INFO   | jvm 1    | 2014/12/09 23:15:50 | <br></br>Total descriptors: 4096
> INFO   | jvm 1    | 2014/12/09 23:15:50 | <br></br>Open descriptors: 556
> INFO   | jvm 1    | 2014/12/09 23:15:50 | <br></br>Available descriptors: 3540
> INFO   | jvm 1    | 2014/12/09 23:15:50 | <br></br>Free disk space: 18 GB
> INFO   | jvm 1    | 2014/12/09 23:15:50 | <br></br>
> INFO   | wrapper  | 2014/12/09 23:17:09 | Wrapper Process has not received any CPU time for 1 seconds.  Extending timeouts.
> STATUS | wrapper  | 2014/12/09 23:17:09 | Pinging the JVM took 7 seconds to respond.
> INFO   | wrapper  | 2014/12/09 23:17:09 | Wrapper Process has not received any CPU time for 5 seconds.  Extending timeouts.
> STATUS | wrapper  | 2014/12/09 23:17:09 | JVM received a signal SIGKILL (9).
> STATUS | wrapper  | 2014/12/09 23:17:09 | JVM process is gone.
> ERROR  | wrapper  | 2014/12/09 23:17:09 | JVM exited unexpectedly.
> STATUS | wrapper  | 2014/12/09 23:17:09 | Automatic JVM Restarts disabled.  Shutting down.
> STATUS | wrapper  | 2014/12/09 23:17:09 | <-- Wrapper Stopped
>
>
> and I found this in the /var/log/messages
>
> Dec  9 23:17:09 jorsek-home2 kernel: Out of memory: Kill process 10262
> (java) score 591 or sacrifice child
> Dec  9 23:17:09 jorsek-home2 kernel: Killed process 10262 (java)
> total-vm:6478728kB, anon-rss:2600344kB, file-rss:0kB
>
> So this does appear to be the kernel killing the process, but according to
> java I have plenty of memory available (>1GB).
>
> I know this is not really a wrapper issue any longer, but I would be very
> appreciative if anyone can give me some advice or point me to some
> resources that might help figure this out.
>
> For reference this is a Centos7 box with 4GB of RAM and a 1GB swap space,
> and 40GB SSD drive.
>
> Tonight I am going to add NewRelic monitoring to this server to see if I
> can get a better picture of what is going on.
>
> Thanks!
>
> On Tue, Dec 2, 2014 at 7:30 PM, Roberto Espinoza <
> rob...@ta...> wrote:
>
>> Casey,
>>
>> Have you checked the kernel logs to see if it wasn't killed because
>> your system ran out of memory?
>>
>> Certainly there is no way to know this inside the JVM because the
>> kernel out of memory killer will just score the current processes and
>> issue a kill.
>>
>> On Wed, Dec 3, 2014 at 12:39 AM, Casey Jordan <cas...@jo...>
>> wrote:
>> > Hi Leif,
>> >
>> > Thanks for the feedback. This is quite a mystery then because I am 100%
>> sure
>> > that the process wasn't killed manually or by any other systems we have
>> > running. (This is just a standard CentOS 7 setup)
>> >
>> > At about the same time that this log appeared I was running a very heavy
>> > build process in a separate JVM instance.
>> >
>> > It seems more than just coincidental that this happened at the same
>> time. I
>> > assume that if the JVM crashed it wouldn't have received a SIG 9? Can
>> you
>> > think of any scenario where this makes sense? Perhaps I am missing some
>> > knowledge of java architecture here that is making this mysterious.
>> >
>> > Thanks
>> >
>> >
>> > On Tue, Dec 2, 2014 at 4:27 AM, Leif Mortenson
>> > <lei...@ta...> wrote:
>> >>
>> >> Casey,
>> >> The configuration you specified will disable all pings and cpu warnings
>> >> and should do what you want.
>> >> The Wrapper should never timeout  in this case.  There are other things
>> >> such as a JVM crash would would still result in a restart after the
>> fact.
>> >>
>> >> The log that you send shows that the JVM will killed with a kill -9.
>> Not
>> >> sure if that was a test on your part, but if the source of the signal
>> has
>> >> permission to kill the JVM's process then there is nothing we can do to
>> >> prevent that.
>> >>
>> >> Cheers,
>> >> Leif
>> >>
>> >> On Tue, Dec 2, 2014 at 12:16 PM, Casey Jordan <cas...@jo...
>> >
>> >> wrote:
>> >>>
>> >>> Hi all,
>> >>>
>> >>> I am sure this is a common question, but I am having trouble
>> extracting
>> >>> details I need from the documentation. From time to time we get a
>> situation
>> >>> where our JVM gets killed and we see something like this in the logs:
>> >>>
>> >>> INFO   | wrapper  | 2014/12/02 02:23:18 | Wrapper Process has not
>> >>> received any CPU time for 1 seconds.  Extending timeouts.
>> >>> STATUS | wrapper  | 2014/12/02 02:23:37 | JVM received a signal
>> SIGKILL
>> >>> (9).
>> >>> STATUS | wrapper  | 2014/12/02 02:23:37 | JVM process is gone.
>> >>> ERROR  | wrapper  | 2014/12/02 02:23:37 | JVM exited unexpectedly.
>> >>> STATUS | wrapper  | 2014/12/02 02:23:38 | Automatic JVM Restarts
>> >>> disabled.  Shutting down.
>> >>> STATUS | wrapper  | 2014/12/02 02:23:38 | <-- Wrapper Stopped
>> >>>
>> >>>
>> >>> This always happens in the case where some other process is eating up
>> all
>> >>> the cpu for more than a few seconds.
>> >>>
>> >>> It's crucial for us that the wrapper never, ever restarts the JVM as
>> this
>> >>> is something we always want to look into manually.
>> >>>
>> >>> So my question is, what is the best way to make sure this doesn't
>> happen?
>> >>>
>> >>> I am using community version 3.5.17, and have the following values in
>> our
>> >>> wrapper.conf:
>> >>>
>> >>>
>> #********************************************************************
>> >>>    # Timeouts
>> >>>
>> #********************************************************************
>> >>>    wrapper.ping.timeout=0
>> >>>    wrapper.ping.timeout.action=DEBUG
>> >>>    wrapper.cpu.timeout=0
>> >>>    wrapper.startup.timeout=300
>> >>>
>> >>> Any help is much appreciated.
>> >>>
>> >>> Thanks!
>> >>>
>> >>
>> >>
>> >>
>> ------------------------------------------------------------------------------
>> >> Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
>> >> from Actuate! Instantly Supercharge Your Business Reports and
>> Dashboards
>> >> with Interactivity, Sharing, Native Excel Exports, App Integration &
>> more
>> >> Get technology previously reserved for billion-dollar corporations,
>> FREE
>> >>
>> >>
>> http://pubads.g.doubleclick.net/gampad/clk?id=157005751&iu=/4140/ostg.clktrk
>> >> _______________________________________________
>> >> Wrapper-user mailing list
>> >> Wra...@li...
>> >> https://lists.sourceforge.net/lists/listinfo/wrapper-user
>> >>
>> >
>> >
>> >
>> > --
>> > --
>> > Casey Jordan
>> > easyDITA a product of Jorsek LLC
>> > "CaseyDJordan" on LinkedIn, Twitter & Facebook
>> > (585) 348 7399
>> > easydita.com
>> >
>> >
>> > This message is intended only for the use of the Addressee(s) and may
>> > contain information that is privileged, confidential, and/or exempt from
>> > disclosure under applicable law.  If you are not the intended recipient,
>> > please be advised that any disclosure  copying, distribution, or use of
>> > the information contained herein is prohibited.  If you have received
>> > this communication in error, please destroy all copies of the message,
>> > whether in electronic or hard copy format, as well as attachments, and
>> > immediately contact the sender by replying to this e-mail or by phone.
>> > Thank you.
>> >
>> >
>> ------------------------------------------------------------------------------
>> > Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
>> > from Actuate! Instantly Supercharge Your Business Reports and Dashboards
>> > with Interactivity, Sharing, Native Excel Exports, App Integration &
>> more
>> > Get technology previously reserved for billion-dollar corporations, FREE
>> >
>> http://pubads.g.doubleclick.net/gampad/clk?id=157005751&iu=/4140/ostg.clktrk
>> > _______________________________________________
>> > Wrapper-user mailing list
>> > Wra...@li...
>> > https://lists.sourceforge.net/lists/listinfo/wrapper-user
>> >
>>
>>
>>
>> --
>> Roberto Espinoza
>> Tanuki Software, Ltd.
>> 6-16-7-1001 Nishi-Kasai, Edogawa-ku
>> Tokyo 134-0088 Japan
>> Tel/Fax: +81-3-3878-3211
>> http://www.tanukisoftware.com
>> rob...@ta...
>>
>>
>> ------------------------------------------------------------------------------
>> Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
>> from Actuate! Instantly Supercharge Your Business Reports and Dashboards
>> with Interactivity, Sharing, Native Excel Exports, App Integration & more
>> Get technology previously reserved for billion-dollar corporations, FREE
>>
>> http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk
>> _______________________________________________
>> Wrapper-user mailing list
>> Wra...@li...
>> https://lists.sourceforge.net/lists/listinfo/wrapper-user
>>
>
>
>
> --
> --
> Casey Jordan
> easyDITA a product of Jorsek LLC
> "CaseyDJordan" on LinkedIn, Twitter & Facebook
> (585) 348 7399
> easydita.com
>
>
> This message is intended only for the use of the Addressee(s) and may
> contain information that is privileged, confidential, and/or exempt from
> disclosure under applicable law.  If you are not the intended recipient,
> please be advised that any disclosure  copying, distribution, or use of
> the information contained herein is prohibited.  If you have received
> this communication in error, please destroy all copies of the message,
> whether in electronic or hard copy format, as well as attachments, and
> immediately contact the sender by replying to this e-mail or by phone.
> Thank you.
>
>
> ------------------------------------------------------------------------------
> Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
> from Actuate! Instantly Supercharge Your Business Reports and Dashboards
> with Interactivity, Sharing, Native Excel Exports, App Integration & more
> Get technology previously reserved for billion-dollar corporations, FREE
>
> http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk
> _______________________________________________
> Wrapper-user mailing list
> Wra...@li...
> https://lists.sourceforge.net/lists/listinfo/wrapper-user
>
>

Re: [Wrapper-user] Advice on stopping wrapper from triggering JVM restarts

[Casey J. <cas...@jo...>]

Ok so this happened again and I was able to capture some additional log
information (not from wrapper.debug though as I can't update this server
just yet)

I got this in our logs:

INFO   | jvm 1    | 2014/12/09 23:15:50 | <br></br>Max Mem: 2223 MB
INFO   | jvm 1    | 2014/12/09 23:15:50 | <br></br>Total Mem: 2204 MB
INFO   | jvm 1    | 2014/12/09 23:15:50 | <br></br>Used Mem: 1141 MB
INFO   | jvm 1    | 2014/12/09 23:15:50 | <br></br>Raw free-mem: 1082 MB
INFO   | jvm 1    | 2014/12/09 23:15:50 | <br></br>Total descriptors: 4096
INFO   | jvm 1    | 2014/12/09 23:15:50 | <br></br>Open descriptors: 556
INFO   | jvm 1    | 2014/12/09 23:15:50 | <br></br>Available descriptors: 3540
INFO   | jvm 1    | 2014/12/09 23:15:50 | <br></br>Free disk space: 18 GB
INFO   | jvm 1    | 2014/12/09 23:15:50 | <br></br>
INFO   | wrapper  | 2014/12/09 23:17:09 | Wrapper Process has not
received any CPU time for 1 seconds.  Extending timeouts.
STATUS | wrapper  | 2014/12/09 23:17:09 | Pinging the JVM took 7
seconds to respond.
INFO   | wrapper  | 2014/12/09 23:17:09 | Wrapper Process has not
received any CPU time for 5 seconds.  Extending timeouts.
STATUS | wrapper  | 2014/12/09 23:17:09 | JVM received a signal SIGKILL (9).
STATUS | wrapper  | 2014/12/09 23:17:09 | JVM process is gone.
ERROR  | wrapper  | 2014/12/09 23:17:09 | JVM exited unexpectedly.
STATUS | wrapper  | 2014/12/09 23:17:09 | Automatic JVM Restarts
disabled.  Shutting down.
STATUS | wrapper  | 2014/12/09 23:17:09 | <-- Wrapper Stopped


and I found this in the /var/log/messages

Dec  9 23:17:09 jorsek-home2 kernel: Out of memory: Kill process 10262
(java) score 591 or sacrifice child
Dec  9 23:17:09 jorsek-home2 kernel: Killed process 10262 (java)
total-vm:6478728kB, anon-rss:2600344kB, file-rss:0kB

So this does appear to be the kernel killing the process, but according to
java I have plenty of memory available (>1GB).

I know this is not really a wrapper issue any longer, but I would be very
appreciative if anyone can give me some advice or point me to some
resources that might help figure this out.

For reference this is a Centos7 box with 4GB of RAM and a 1GB swap space,
and 40GB SSD drive.

Tonight I am going to add NewRelic monitoring to this server to see if I
can get a better picture of what is going on.

Thanks!

On Tue, Dec 2, 2014 at 7:30 PM, Roberto Espinoza <
rob...@ta...> wrote:

> Casey,
>
> Have you checked the kernel logs to see if it wasn't killed because
> your system ran out of memory?
>
> Certainly there is no way to know this inside the JVM because the
> kernel out of memory killer will just score the current processes and
> issue a kill.
>
> On Wed, Dec 3, 2014 at 12:39 AM, Casey Jordan <cas...@jo...>
> wrote:
> > Hi Leif,
> >
> > Thanks for the feedback. This is quite a mystery then because I am 100%
> sure
> > that the process wasn't killed manually or by any other systems we have
> > running. (This is just a standard CentOS 7 setup)
> >
> > At about the same time that this log appeared I was running a very heavy
> > build process in a separate JVM instance.
> >
> > It seems more than just coincidental that this happened at the same
> time. I
> > assume that if the JVM crashed it wouldn't have received a SIG 9? Can you
> > think of any scenario where this makes sense? Perhaps I am missing some
> > knowledge of java architecture here that is making this mysterious.
> >
> > Thanks
> >
> >
> > On Tue, Dec 2, 2014 at 4:27 AM, Leif Mortenson
> > <lei...@ta...> wrote:
> >>
> >> Casey,
> >> The configuration you specified will disable all pings and cpu warnings
> >> and should do what you want.
> >> The Wrapper should never timeout  in this case.  There are other things
> >> such as a JVM crash would would still result in a restart after the
> fact.
> >>
> >> The log that you send shows that the JVM will killed with a kill -9.
> Not
> >> sure if that was a test on your part, but if the source of the signal
> has
> >> permission to kill the JVM's process then there is nothing we can do to
> >> prevent that.
> >>
> >> Cheers,
> >> Leif
> >>
> >> On Tue, Dec 2, 2014 at 12:16 PM, Casey Jordan <cas...@jo...>
> >> wrote:
> >>>
> >>> Hi all,
> >>>
> >>> I am sure this is a common question, but I am having trouble extracting
> >>> details I need from the documentation. From time to time we get a
> situation
> >>> where our JVM gets killed and we see something like this in the logs:
> >>>
> >>> INFO   | wrapper  | 2014/12/02 02:23:18 | Wrapper Process has not
> >>> received any CPU time for 1 seconds.  Extending timeouts.
> >>> STATUS | wrapper  | 2014/12/02 02:23:37 | JVM received a signal SIGKILL
> >>> (9).
> >>> STATUS | wrapper  | 2014/12/02 02:23:37 | JVM process is gone.
> >>> ERROR  | wrapper  | 2014/12/02 02:23:37 | JVM exited unexpectedly.
> >>> STATUS | wrapper  | 2014/12/02 02:23:38 | Automatic JVM Restarts
> >>> disabled.  Shutting down.
> >>> STATUS | wrapper  | 2014/12/02 02:23:38 | <-- Wrapper Stopped
> >>>
> >>>
> >>> This always happens in the case where some other process is eating up
> all
> >>> the cpu for more than a few seconds.
> >>>
> >>> It's crucial for us that the wrapper never, ever restarts the JVM as
> this
> >>> is something we always want to look into manually.
> >>>
> >>> So my question is, what is the best way to make sure this doesn't
> happen?
> >>>
> >>> I am using community version 3.5.17, and have the following values in
> our
> >>> wrapper.conf:
> >>>
> >>>
> #********************************************************************
> >>>    # Timeouts
> >>>
> #********************************************************************
> >>>    wrapper.ping.timeout=0
> >>>    wrapper.ping.timeout.action=DEBUG
> >>>    wrapper.cpu.timeout=0
> >>>    wrapper.startup.timeout=300
> >>>
> >>> Any help is much appreciated.
> >>>
> >>> Thanks!
> >>>
> >>
> >>
> >>
> ------------------------------------------------------------------------------
> >> Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
> >> from Actuate! Instantly Supercharge Your Business Reports and Dashboards
> >> with Interactivity, Sharing, Native Excel Exports, App Integration &
> more
> >> Get technology previously reserved for billion-dollar corporations, FREE
> >>
> >>
> http://pubads.g.doubleclick.net/gampad/clk?id=157005751&iu=/4140/ostg.clktrk
> >> _______________________________________________
> >> Wrapper-user mailing list
> >> Wra...@li...
> >> https://lists.sourceforge.net/lists/listinfo/wrapper-user
> >>
> >
> >
> >
> > --
> > --
> > Casey Jordan
> > easyDITA a product of Jorsek LLC
> > "CaseyDJordan" on LinkedIn, Twitter & Facebook
> > (585) 348 7399
> > easydita.com
> >
> >
> > This message is intended only for the use of the Addressee(s) and may
> > contain information that is privileged, confidential, and/or exempt from
> > disclosure under applicable law.  If you are not the intended recipient,
> > please be advised that any disclosure  copying, distribution, or use of
> > the information contained herein is prohibited.  If you have received
> > this communication in error, please destroy all copies of the message,
> > whether in electronic or hard copy format, as well as attachments, and
> > immediately contact the sender by replying to this e-mail or by phone.
> > Thank you.
> >
> >
> ------------------------------------------------------------------------------
> > Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
> > from Actuate! Instantly Supercharge Your Business Reports and Dashboards
> > with Interactivity, Sharing, Native Excel Exports, App Integration & more
> > Get technology previously reserved for billion-dollar corporations, FREE
> >
> http://pubads.g.doubleclick.net/gampad/clk?id=157005751&iu=/4140/ostg.clktrk
> > _______________________________________________
> > Wrapper-user mailing list
> > Wra...@li...
> > https://lists.sourceforge.net/lists/listinfo/wrapper-user
> >
>
>
>
> --
> Roberto Espinoza
> Tanuki Software, Ltd.
> 6-16-7-1001 Nishi-Kasai, Edogawa-ku
> Tokyo 134-0088 Japan
> Tel/Fax: +81-3-3878-3211
> http://www.tanukisoftware.com
> rob...@ta...
>
>
> ------------------------------------------------------------------------------
> Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
> from Actuate! Instantly Supercharge Your Business Reports and Dashboards
> with Interactivity, Sharing, Native Excel Exports, App Integration & more
> Get technology previously reserved for billion-dollar corporations, FREE
>
> http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk
> _______________________________________________
> Wrapper-user mailing list
> Wra...@li...
> https://lists.sourceforge.net/lists/listinfo/wrapper-user
>



-- 
--
Casey Jordan
easyDITA a product of Jorsek LLC
"CaseyDJordan" on LinkedIn, Twitter & Facebook
(585) 348 7399
easydita.com


This message is intended only for the use of the Addressee(s) and may
contain information that is privileged, confidential, and/or exempt from
disclosure under applicable law.  If you are not the intended recipient,
please be advised that any disclosure  copying, distribution, or use of
the information contained herein is prohibited.  If you have received
this communication in error, please destroy all copies of the message,
whether in electronic or hard copy format, as well as attachments, and
immediately contact the sender by replying to this e-mail or by phone.
Thank you.

Re: [Wrapper-user] Pinger Thread

[Leif M. <lei...@ta...>]

Steve,
The Wrapper does several things to monitor the health of the Java process
that it is managing.

One of the simpler ones is to ping the JVM at regular intervals.  The JVM
then responds as soon as it sees the ping request.
Newer versions of the Wrapper include a timestamp with the pings to help
measure the return time.
When the JVM fails to respond to a ping for longer than the
wrapper.ping.timeout threshold, the Wrapper decides that the JVM is frozen
and restarts it.
See the following page for details:
http://wrapper.tanukisoftware.com/doc/english/prop-ping-timeout.html

If you set wrapper.debug=true you will be able to see all of these pings
and their payloads in the log file.

Cheers,
Leif


On Tue, Dec 9, 2014 at 12:36 AM, <Ste...@su...> wrote:

>  Hello,
>
>
>
> Could anyone please explain how the pinger threads work? Is it as simple
> as the wrapper and the wrapped JVMs communicating using port 15003? What do
> these threads send to each other?
>
>
>
> Thanks,
>
> Steve
>

--
Leif Mortenson
Tanuki Software, Ltd.
6-16-7-1001 Nishi-Kasai, Edogawa-ku
Tokyo 134-0088 Japan
Tel: +81-3-3878-3211
Fax: +81-3-3878-0313
http://www.tanukisoftware.com
lei...@ta...

[Wrapper-user] Pinger Thread

[<Ste...@su...>]

Hello,

Could anyone please explain how the pinger threads work? Is it as simple as the wrapper and the wrapped JVMs communicating using port 15003? What do these threads send to each other?

Thanks,
Steve

Re: [Wrapper-user] Advice on stopping wrapper from triggering JVM restarts

[Roberto E. <rob...@ta...>]

Casey,

Have you checked the kernel logs to see if it wasn't killed because
your system ran out of memory?

Certainly there is no way to know this inside the JVM because the
kernel out of memory killer will just score the current processes and
issue a kill.

On Wed, Dec 3, 2014 at 12:39 AM, Casey Jordan <cas...@jo...> wrote:
> Hi Leif,
>
> Thanks for the feedback. This is quite a mystery then because I am 100% sure
> that the process wasn't killed manually or by any other systems we have
> running. (This is just a standard CentOS 7 setup)
>
> At about the same time that this log appeared I was running a very heavy
> build process in a separate JVM instance.
>
> It seems more than just coincidental that this happened at the same time. I
> assume that if the JVM crashed it wouldn't have received a SIG 9? Can you
> think of any scenario where this makes sense? Perhaps I am missing some
> knowledge of java architecture here that is making this mysterious.
>
> Thanks
>
>
> On Tue, Dec 2, 2014 at 4:27 AM, Leif Mortenson
> <lei...@ta...> wrote:
>>
>> Casey,
>> The configuration you specified will disable all pings and cpu warnings
>> and should do what you want.
>> The Wrapper should never timeout  in this case.  There are other things
>> such as a JVM crash would would still result in a restart after the fact.
>>
>> The log that you send shows that the JVM will killed with a kill -9.  Not
>> sure if that was a test on your part, but if the source of the signal has
>> permission to kill the JVM's process then there is nothing we can do to
>> prevent that.
>>
>> Cheers,
>> Leif
>>
>> On Tue, Dec 2, 2014 at 12:16 PM, Casey Jordan <cas...@jo...>
>> wrote:
>>>
>>> Hi all,
>>>
>>> I am sure this is a common question, but I am having trouble extracting
>>> details I need from the documentation. From time to time we get a situation
>>> where our JVM gets killed and we see something like this in the logs:
>>>
>>> INFO   | wrapper  | 2014/12/02 02:23:18 | Wrapper Process has not
>>> received any CPU time for 1 seconds.  Extending timeouts.
>>> STATUS | wrapper  | 2014/12/02 02:23:37 | JVM received a signal SIGKILL
>>> (9).
>>> STATUS | wrapper  | 2014/12/02 02:23:37 | JVM process is gone.
>>> ERROR  | wrapper  | 2014/12/02 02:23:37 | JVM exited unexpectedly.
>>> STATUS | wrapper  | 2014/12/02 02:23:38 | Automatic JVM Restarts
>>> disabled.  Shutting down.
>>> STATUS | wrapper  | 2014/12/02 02:23:38 | <-- Wrapper Stopped
>>>
>>>
>>> This always happens in the case where some other process is eating up all
>>> the cpu for more than a few seconds.
>>>
>>> It's crucial for us that the wrapper never, ever restarts the JVM as this
>>> is something we always want to look into manually.
>>>
>>> So my question is, what is the best way to make sure this doesn't happen?
>>>
>>> I am using community version 3.5.17, and have the following values in our
>>> wrapper.conf:
>>>
>>>    #********************************************************************
>>>    # Timeouts
>>>    #********************************************************************
>>>    wrapper.ping.timeout=0
>>>    wrapper.ping.timeout.action=DEBUG
>>>    wrapper.cpu.timeout=0
>>>    wrapper.startup.timeout=300
>>>
>>> Any help is much appreciated.
>>>
>>> Thanks!
>>>
>>
>>
>> ------------------------------------------------------------------------------
>> Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
>> from Actuate! Instantly Supercharge Your Business Reports and Dashboards
>> with Interactivity, Sharing, Native Excel Exports, App Integration & more
>> Get technology previously reserved for billion-dollar corporations, FREE
>>
>> http://pubads.g.doubleclick.net/gampad/clk?id=157005751&iu=/4140/ostg.clktrk
>> _______________________________________________
>> Wrapper-user mailing list
>> Wra...@li...
>> https://lists.sourceforge.net/lists/listinfo/wrapper-user
>>
>
>
>
> --
> --
> Casey Jordan
> easyDITA a product of Jorsek LLC
> "CaseyDJordan" on LinkedIn, Twitter & Facebook
> (585) 348 7399
> easydita.com
>
>
> This message is intended only for the use of the Addressee(s) and may
> contain information that is privileged, confidential, and/or exempt from
> disclosure under applicable law.  If you are not the intended recipient,
> please be advised that any disclosure  copying, distribution, or use of
> the information contained herein is prohibited.  If you have received
> this communication in error, please destroy all copies of the message,
> whether in electronic or hard copy format, as well as attachments, and
> immediately contact the sender by replying to this e-mail or by phone.
> Thank you.
>
> ------------------------------------------------------------------------------
> Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
> from Actuate! Instantly Supercharge Your Business Reports and Dashboards
> with Interactivity, Sharing, Native Excel Exports, App Integration & more
> Get technology previously reserved for billion-dollar corporations, FREE
> http://pubads.g.doubleclick.net/gampad/clk?id=157005751&iu=/4140/ostg.clktrk
> _______________________________________________
> Wrapper-user mailing list
> Wra...@li...
> https://lists.sourceforge.net/lists/listinfo/wrapper-user
>



-- 
Roberto Espinoza
Tanuki Software, Ltd.
6-16-7-1001 Nishi-Kasai, Edogawa-ku
Tokyo 134-0088 Japan
Tel/Fax: +81-3-3878-3211
http://www.tanukisoftware.com
rob...@ta...

Re: [Wrapper-user] Advice on stopping wrapper from triggering JVM restarts

[Mike P. <MP...@np...>]

I’ve been commenting on a similar issue where we occasionally have a JVM killed and restarted by the wrapper because it doesn’t get a ping response. We still haven’t tracked down the root cause but we don’t get the CPU warning but rather “Timed out waiting for signal from JVM”. It always seems to happen around midnight which is when we have some background jobs kick off in various processes so we think it is related to that, but we don’t have any hard evidence. We’re running RedHat Linux guests in VMWare. Here’s our log output if it is any help:

STATUS | wrapper  | 2014/12/01 00:05:51 | JVM appears hung: Timed out waiting for signal from JVM.  Restarting JVM.
ERROR  | wrapper  | 2014/12/01 00:06:38 | Shutdown failed: Timed out waiting for the JVM to terminate.
ERROR  | wrapper  | 2014/12/01 00:06:39 | JVM did not exit on request, termination requested.
STATUS | wrapper  | 2014/12/01 00:06:39 | JVM received a signal SIGKILL (9).
STATUS | wrapper  | 2014/12/01 00:06:39 | JVM process is gone.
STATUS | wrapper  | 2014/12/01 00:06:39 | JVM exited after being requested to terminate.

In most cases the wrapper attempts to restart the JVM 5 times, each of them timing out. It then finally just gives up and the process remains down.

-mike


NPR | Mike Pilone | Software Architect, Distribution | 1111 North Capital St., NE | Washington, DC 20002





<https://twitter.com/PRSS_NOC>
<https://twitter.com/PRSS_NOC>

On Dec 2, 2014, at 10:39 AM, Casey Jordan <cas...@jo...<mailto:cas...@jo...>> wrote:

Hi Leif,

Thanks for the feedback. This is quite a mystery then because I am 100% sure that the process wasn't killed manually or by any other systems we have running. (This is just a standard CentOS 7 setup)

At about the same time that this log appeared I was running a very heavy build process in a separate JVM instance.

It seems more than just coincidental that this happened at the same time. I assume that if the JVM crashed it wouldn't have received a SIG 9? Can you think of any scenario where this makes sense? Perhaps I am missing some knowledge of java architecture here that is making this mysterious.

Thanks


On Tue, Dec 2, 2014 at 4:27 AM, Leif Mortenson <lei...@ta...<mailto:lei...@ta...>> wrote:
Casey,
The configuration you specified will disable all pings and cpu warnings and should do what you want.
The Wrapper should never timeout  in this case.  There are other things such as a JVM crash would would still result in a restart after the fact.

The log that you send shows that the JVM will killed with a kill -9.  Not sure if that was a test on your part, but if the source of the signal has permission to kill the JVM's process then there is nothing we can do to prevent that.

Cheers,
Leif

On Tue, Dec 2, 2014 at 12:16 PM, Casey Jordan <cas...@jo...<mailto:cas...@jo...>> wrote:
Hi all,

I am sure this is a common question, but I am having trouble extracting details I need from the documentation. From time to time we get a situation where our JVM gets killed and we see something like this in the logs:

INFO   | wrapper  | 2014/12/02 02:23:18 | Wrapper Process has not received any CPU time for 1 seconds.  Extending timeouts.
STATUS | wrapper  | 2014/12/02 02:23:37 | JVM received a signal SIGKILL (9).
STATUS | wrapper  | 2014/12/02 02:23:37 | JVM process is gone.
ERROR  | wrapper  | 2014/12/02 02:23:37 | JVM exited unexpectedly.
STATUS | wrapper  | 2014/12/02 02:23:38 | Automatic JVM Restarts disabled.  Shutting down.
STATUS | wrapper  | 2014/12/02 02:23:38 | <-- Wrapper Stopped


This always happens in the case where some other process is eating up all the cpu for more than a few seconds.

It's crucial for us that the wrapper never, ever restarts the JVM as this is something we always want to look into manually.

So my question is, what is the best way to make sure this doesn't happen?

I am using community version 3.5.17, and have the following values in our wrapper.conf:

   #********************************************************************
   # Timeouts
   #********************************************************************
   wrapper.ping.timeout=0
   wrapper.ping.timeout.action=DEBUG
   wrapper.cpu.timeout=0
   wrapper.startup.timeout=300

Any help is much appreciated.

Thanks!


------------------------------------------------------------------------------
Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
from Actuate! Instantly Supercharge Your Business Reports and Dashboards
with Interactivity, Sharing, Native Excel Exports, App Integration & more
Get technology previously reserved for billion-dollar corporations, FREE
http://pubads.g.doubleclick.net/gampad/clk?id=157005751&iu=/4140/ostg.clktrk
_______________________________________________
Wrapper-user mailing list
Wra...@li...<mailto:Wra...@li...>
https://lists.sourceforge.net/lists/listinfo/wrapper-user




--
--
Casey Jordan
easyDITA a product of Jorsek LLC
"CaseyDJordan" on LinkedIn, Twitter & Facebook
(585) 348 7399
easydita.com<http://easydita.com/>


This message is intended only for the use of the Addressee(s) and may
contain information that is privileged, confidential, and/or exempt from
disclosure under applicable law.  If you are not the intended recipient,
please be advised that any disclosure  copying, distribution, or use of
the information contained herein is prohibited.  If you have received
this communication in error, please destroy all copies of the message,
whether in electronic or hard copy format, as well as attachments, and
immediately contact the sender by replying to this e-mail or by phone.
Thank you.
------------------------------------------------------------------------------
Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
from Actuate! Instantly Supercharge Your Business Reports and Dashboards
with Interactivity, Sharing, Native Excel Exports, App Integration & more
Get technology previously reserved for billion-dollar corporations, FREE
http://pubads.g.doubleclick.net/gampad/clk?id=157005751&iu=/4140/ostg.clktrk_______________________________________________
Wrapper-user mailing list
Wra...@li...<mailto:Wra...@li...>
https://lists.sourceforge.net/lists/listinfo/wrapper-user

Re: [Wrapper-user] Advice on stopping wrapper from triggering JVM restarts

[Casey J. <cas...@jo...>]

Yeah I can try that, thanks!

On Tue, Dec 2, 2014 at 11:43 AM, Leif Mortenson <
lei...@ta...> wrote:

> Casey,
> If you are able to reproduce this, can you set wrapper.debug=true and
> repeat?  This log output will shed more light on exactly what is
> happening.  Depending on the platform, we will also be able to see the pid
> of the source of the kill signal.
>
> Cheers,
> Leif
>
>
> On Wed, Dec 3, 2014 at 12:39 AM, Casey Jordan <cas...@jo...>
> wrote:
>
>> Hi Leif,
>>
>> Thanks for the feedback. This is quite a mystery then because I am 100%
>> sure that the process wasn't killed manually or by any other systems we
>> have running. (This is just a standard CentOS 7 setup)
>>
>> At about the same time that this log appeared I was running a very heavy
>> build process in a separate JVM instance.
>>
>> It seems more than just coincidental that this happened at the same time.
>> I assume that if the JVM crashed it wouldn't have received a SIG 9? Can you
>> think of any scenario where this makes sense? Perhaps I am missing some
>> knowledge of java architecture here that is making this mysterious.
>>
>> Thanks
>>
>>
>> On Tue, Dec 2, 2014 at 4:27 AM, Leif Mortenson <
>> lei...@ta...> wrote:
>>
>>> Casey,
>>> The configuration you specified will disable all pings and cpu warnings
>>> and should do what you want.
>>> The Wrapper should never timeout  in this case.  There are other things
>>> such as a JVM crash would would still result in a restart after the fact.
>>>
>>> The log that you send shows that the JVM will killed with a kill -9.
>>> Not sure if that was a test on your part, but if the source of the signal
>>> has permission to kill the JVM's process then there is nothing we can do to
>>> prevent that.
>>>
>>> Cheers,
>>> Leif
>>>
>>> On Tue, Dec 2, 2014 at 12:16 PM, Casey Jordan <cas...@jo...>
>>> wrote:
>>>
>>>> Hi all,
>>>>
>>>> I am sure this is a common question, but I am having trouble extracting
>>>> details I need from the documentation. From time to time we get a situation
>>>> where our JVM gets killed and we see something like this in the logs:
>>>>
>>>> INFO   | wrapper  | 2014/12/02 02:23:18 | Wrapper Process has not
>>>> received any CPU time for 1 seconds.  Extending timeouts.
>>>> STATUS | wrapper  | 2014/12/02 02:23:37 | JVM received a signal SIGKILL
>>>> (9).
>>>> STATUS | wrapper  | 2014/12/02 02:23:37 | JVM process is gone.
>>>> ERROR  | wrapper  | 2014/12/02 02:23:37 | JVM exited unexpectedly.
>>>> STATUS | wrapper  | 2014/12/02 02:23:38 | Automatic JVM Restarts
>>>> disabled.  Shutting down.
>>>> STATUS | wrapper  | 2014/12/02 02:23:38 | <-- Wrapper Stopped
>>>>
>>>>
>>>> This always happens in the case where some other process is eating up
>>>> all the cpu for more than a few seconds.
>>>>
>>>> It's crucial for us that the wrapper never, ever restarts the JVM as
>>>> this is something we always want to look into manually.
>>>>
>>>> So my question is, what is the best way to make sure this doesn't
>>>> happen?
>>>>
>>>> I am using community version 3.5.17, and have the following values in
>>>> our wrapper.conf:
>>>>
>>>>    #********************************************************************
>>>>    # Timeouts
>>>>    #********************************************************************
>>>>    wrapper.ping.timeout=0
>>>>    wrapper.ping.timeout.action=DEBUG
>>>>    wrapper.cpu.timeout=0
>>>>    wrapper.startup.timeout=300
>>>>
>>>> Any help is much appreciated.
>>>>
>>>> Thanks!
>>>>
>>>>
>
> ------------------------------------------------------------------------------
> Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
> from Actuate! Instantly Supercharge Your Business Reports and Dashboards
> with Interactivity, Sharing, Native Excel Exports, App Integration & more
> Get technology previously reserved for billion-dollar corporations, FREE
>
> http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk
> _______________________________________________
> Wrapper-user mailing list
> Wra...@li...
> https://lists.sourceforge.net/lists/listinfo/wrapper-user
>
>


-- 
--
Casey Jordan
easyDITA a product of Jorsek LLC
"CaseyDJordan" on LinkedIn, Twitter & Facebook
(585) 348 7399
easydita.com


This message is intended only for the use of the Addressee(s) and may
contain information that is privileged, confidential, and/or exempt from
disclosure under applicable law.  If you are not the intended recipient,
please be advised that any disclosure  copying, distribution, or use of
the information contained herein is prohibited.  If you have received
this communication in error, please destroy all copies of the message,
whether in electronic or hard copy format, as well as attachments, and
immediately contact the sender by replying to this e-mail or by phone.
Thank you.

Re: [Wrapper-user] Advice on stopping wrapper from triggering JVM restarts

[Leif M. <lei...@ta...>]

Casey,
If you are able to reproduce this, can you set wrapper.debug=true and
repeat?  This log output will shed more light on exactly what is
happening.  Depending on the platform, we will also be able to see the pid
of the source of the kill signal.

Cheers,
Leif

On Wed, Dec 3, 2014 at 12:39 AM, Casey Jordan <cas...@jo...>
wrote:

> Hi Leif,
>
> Thanks for the feedback. This is quite a mystery then because I am 100%
> sure that the process wasn't killed manually or by any other systems we
> have running. (This is just a standard CentOS 7 setup)
>
> At about the same time that this log appeared I was running a very heavy
> build process in a separate JVM instance.
>
> It seems more than just coincidental that this happened at the same time.
> I assume that if the JVM crashed it wouldn't have received a SIG 9? Can you
> think of any scenario where this makes sense? Perhaps I am missing some
> knowledge of java architecture here that is making this mysterious.
>
> Thanks
>
>
> On Tue, Dec 2, 2014 at 4:27 AM, Leif Mortenson <
> lei...@ta...> wrote:
>
>> Casey,
>> The configuration you specified will disable all pings and cpu warnings
>> and should do what you want.
>> The Wrapper should never timeout  in this case.  There are other things
>> such as a JVM crash would would still result in a restart after the fact.
>>
>> The log that you send shows that the JVM will killed with a kill -9.  Not
>> sure if that was a test on your part, but if the source of the signal has
>> permission to kill the JVM's process then there is nothing we can do to
>> prevent that.
>>
>> Cheers,
>> Leif
>>
>> On Tue, Dec 2, 2014 at 12:16 PM, Casey Jordan <cas...@jo...>
>> wrote:
>>
>>> Hi all,
>>>
>>> I am sure this is a common question, but I am having trouble extracting
>>> details I need from the documentation. From time to time we get a situation
>>> where our JVM gets killed and we see something like this in the logs:
>>>
>>> INFO   | wrapper  | 2014/12/02 02:23:18 | Wrapper Process has not
>>> received any CPU time for 1 seconds.  Extending timeouts.
>>> STATUS | wrapper  | 2014/12/02 02:23:37 | JVM received a signal SIGKILL
>>> (9).
>>> STATUS | wrapper  | 2014/12/02 02:23:37 | JVM process is gone.
>>> ERROR  | wrapper  | 2014/12/02 02:23:37 | JVM exited unexpectedly.
>>> STATUS | wrapper  | 2014/12/02 02:23:38 | Automatic JVM Restarts
>>> disabled.  Shutting down.
>>> STATUS | wrapper  | 2014/12/02 02:23:38 | <-- Wrapper Stopped
>>>
>>>
>>> This always happens in the case where some other process is eating up
>>> all the cpu for more than a few seconds.
>>>
>>> It's crucial for us that the wrapper never, ever restarts the JVM as
>>> this is something we always want to look into manually.
>>>
>>> So my question is, what is the best way to make sure this doesn't happen?
>>>
>>> I am using community version 3.5.17, and have the following values in
>>> our wrapper.conf:
>>>
>>>    #********************************************************************
>>>    # Timeouts
>>>    #********************************************************************
>>>    wrapper.ping.timeout=0
>>>    wrapper.ping.timeout.action=DEBUG
>>>    wrapper.cpu.timeout=0
>>>    wrapper.startup.timeout=300
>>>
>>> Any help is much appreciated.
>>>
>>> Thanks!
>>>
>>>

Re: [Wrapper-user] Advice on stopping wrapper from triggering JVM restarts

[Casey J. <cas...@jo...>]

Hi Leif,

Thanks for the feedback. This is quite a mystery then because I am 100%
sure that the process wasn't killed manually or by any other systems we
have running. (This is just a standard CentOS 7 setup)

At about the same time that this log appeared I was running a very heavy
build process in a separate JVM instance.

It seems more than just coincidental that this happened at the same time. I
assume that if the JVM crashed it wouldn't have received a SIG 9? Can you
think of any scenario where this makes sense? Perhaps I am missing some
knowledge of java architecture here that is making this mysterious.

Thanks


On Tue, Dec 2, 2014 at 4:27 AM, Leif Mortenson <
lei...@ta...> wrote:

> Casey,
> The configuration you specified will disable all pings and cpu warnings
> and should do what you want.
> The Wrapper should never timeout  in this case.  There are other things
> such as a JVM crash would would still result in a restart after the fact.
>
> The log that you send shows that the JVM will killed with a kill -9.  Not
> sure if that was a test on your part, but if the source of the signal has
> permission to kill the JVM's process then there is nothing we can do to
> prevent that.
>
> Cheers,
> Leif
>
> On Tue, Dec 2, 2014 at 12:16 PM, Casey Jordan <cas...@jo...>
> wrote:
>
>> Hi all,
>>
>> I am sure this is a common question, but I am having trouble extracting
>> details I need from the documentation. From time to time we get a situation
>> where our JVM gets killed and we see something like this in the logs:
>>
>> INFO   | wrapper  | 2014/12/02 02:23:18 | Wrapper Process has not
>> received any CPU time for 1 seconds.  Extending timeouts.
>> STATUS | wrapper  | 2014/12/02 02:23:37 | JVM received a signal SIGKILL
>> (9).
>> STATUS | wrapper  | 2014/12/02 02:23:37 | JVM process is gone.
>> ERROR  | wrapper  | 2014/12/02 02:23:37 | JVM exited unexpectedly.
>> STATUS | wrapper  | 2014/12/02 02:23:38 | Automatic JVM Restarts
>> disabled.  Shutting down.
>> STATUS | wrapper  | 2014/12/02 02:23:38 | <-- Wrapper Stopped
>>
>>
>> This always happens in the case where some other process is eating up all
>> the cpu for more than a few seconds.
>>
>> It's crucial for us that the wrapper never, ever restarts the JVM as this
>> is something we always want to look into manually.
>>
>> So my question is, what is the best way to make sure this doesn't happen?
>>
>> I am using community version 3.5.17, and have the following values in our
>> wrapper.conf:
>>
>>    #********************************************************************
>>    # Timeouts
>>    #********************************************************************
>>    wrapper.ping.timeout=0
>>    wrapper.ping.timeout.action=DEBUG
>>    wrapper.cpu.timeout=0
>>    wrapper.startup.timeout=300
>>
>> Any help is much appreciated.
>>
>> Thanks!
>>
>>
>
> ------------------------------------------------------------------------------
> Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
> from Actuate! Instantly Supercharge Your Business Reports and Dashboards
> with Interactivity, Sharing, Native Excel Exports, App Integration & more
> Get technology previously reserved for billion-dollar corporations, FREE
>
> http://pubads.g.doubleclick.net/gampad/clk?id=157005751&iu=/4140/ostg.clktrk
> _______________________________________________
> Wrapper-user mailing list
> Wra...@li...
> https://lists.sourceforge.net/lists/listinfo/wrapper-user
>
>


-- 
--
Casey Jordan
easyDITA a product of Jorsek LLC
"CaseyDJordan" on LinkedIn, Twitter & Facebook
(585) 348 7399
easydita.com


This message is intended only for the use of the Addressee(s) and may
contain information that is privileged, confidential, and/or exempt from
disclosure under applicable law.  If you are not the intended recipient,
please be advised that any disclosure  copying, distribution, or use of
the information contained herein is prohibited.  If you have received
this communication in error, please destroy all copies of the message,
whether in electronic or hard copy format, as well as attachments, and
immediately contact the sender by replying to this e-mail or by phone.
Thank you.

Re: [Wrapper-user] Advice on stopping wrapper from triggering JVM restarts

[Leif M. <lei...@ta...>]

Casey,
The configuration you specified will disable all pings and cpu warnings and
should do what you want.
The Wrapper should never timeout  in this case.  There are other things
such as a JVM crash would would still result in a restart after the fact.

The log that you send shows that the JVM will killed with a kill -9.  Not
sure if that was a test on your part, but if the source of the signal has
permission to kill the JVM's process then there is nothing we can do to
prevent that.

Cheers,
Leif

On Tue, Dec 2, 2014 at 12:16 PM, Casey Jordan <cas...@jo...>
wrote:

> Hi all,
>
> I am sure this is a common question, but I am having trouble extracting
> details I need from the documentation. From time to time we get a situation
> where our JVM gets killed and we see something like this in the logs:
>
> INFO   | wrapper  | 2014/12/02 02:23:18 | Wrapper Process has not received
> any CPU time for 1 seconds.  Extending timeouts.
> STATUS | wrapper  | 2014/12/02 02:23:37 | JVM received a signal SIGKILL
> (9).
> STATUS | wrapper  | 2014/12/02 02:23:37 | JVM process is gone.
> ERROR  | wrapper  | 2014/12/02 02:23:37 | JVM exited unexpectedly.
> STATUS | wrapper  | 2014/12/02 02:23:38 | Automatic JVM Restarts
> disabled.  Shutting down.
> STATUS | wrapper  | 2014/12/02 02:23:38 | <-- Wrapper Stopped
>
>
> This always happens in the case where some other process is eating up all
> the cpu for more than a few seconds.
>
> It's crucial for us that the wrapper never, ever restarts the JVM as this
> is something we always want to look into manually.
>
> So my question is, what is the best way to make sure this doesn't happen?
>
> I am using community version 3.5.17, and have the following values in our
> wrapper.conf:
>
>    #********************************************************************
>    # Timeouts
>    #********************************************************************
>    wrapper.ping.timeout=0
>    wrapper.ping.timeout.action=DEBUG
>    wrapper.cpu.timeout=0
>    wrapper.startup.timeout=300
>
> Any help is much appreciated.
>
> Thanks!
>
>

[Wrapper-user] Advice on stopping wrapper from triggering JVM restarts

[Casey J. <cas...@jo...>]

Hi all,

I am sure this is a common question, but I am having trouble extracting
details I need from the documentation. From time to time we get a situation
where our JVM gets killed and we see something like this in the logs:

INFO   | wrapper  | 2014/12/02 02:23:18 | Wrapper Process has not received
any CPU time for 1 seconds.  Extending timeouts.
STATUS | wrapper  | 2014/12/02 02:23:37 | JVM received a signal SIGKILL (9).
STATUS | wrapper  | 2014/12/02 02:23:37 | JVM process is gone.
ERROR  | wrapper  | 2014/12/02 02:23:37 | JVM exited unexpectedly.
STATUS | wrapper  | 2014/12/02 02:23:38 | Automatic JVM Restarts disabled.
Shutting down.
STATUS | wrapper  | 2014/12/02 02:23:38 | <-- Wrapper Stopped


This always happens in the case where some other process is eating up all
the cpu for more than a few seconds.

It's crucial for us that the wrapper never, ever restarts the JVM as this
is something we always want to look into manually.

So my question is, what is the best way to make sure this doesn't happen?

I am using community version 3.5.17, and have the following values in our
wrapper.conf:

   #********************************************************************
   # Timeouts
   #********************************************************************
   wrapper.ping.timeout=0
   wrapper.ping.timeout.action=DEBUG
   wrapper.cpu.timeout=0
   wrapper.startup.timeout=300

Any help is much appreciated.

Thanks!

-- 
--
Casey Jordan
easyDITA a product of Jorsek LLC
"CaseyDJordan" on LinkedIn, Twitter & Facebook
(585) 348 7399
easydita.com


This message is intended only for the use of the Addressee(s) and may
contain information that is privileged, confidential, and/or exempt from
disclosure under applicable law.  If you are not the intended recipient,
please be advised that any disclosure  copying, distribution, or use of
the information contained herein is prohibited.  If you have received
this communication in error, please destroy all copies of the message,
whether in electronic or hard copy format, as well as attachments, and
immediately contact the sender by replying to this e-mail or by phone.
Thank you.

Re: [Wrapper-user] RUN_LEVEL not working

[Alexandre K. <ale...@ta...>]

In case someone else is facing the same issue.

On RHEL (CentOS and Fedora), the script file is using "chkconfig". And to
set the desired priority you have to modify the following line:
# chkconfig: 2345 20 80

Do not remove '#'.
"2345" means the run level 2,3,4 and 5.
The last 2 numbers are the values for S and K.

The RUN_LEVEL parameter is used for other platforms (Solaris, HPUX....).

And make sure to uninstall the service first, before editing the script
file.

Here is a link about chkconfig:
http://linuxcommand.org/man_pages/chkconfig8.html

Alexandre Klein



Alexandre Klein
Tanuki Software, Ltd.
6-16-7-1001 Nishi-Kasai, Edogawa-ku
Tokyo 134-0088 Japan
Tel: +81-3-3878-3211
Fax: +81-3-3878-0313
http://www.tanukisoftware.com

On Fri, Nov 14, 2014 at 8:50 PM, Lars Schnoor <la...@if...> wrote:

> Hi
>
> I am using the wrapper in version 3.5.25 on Linux 32-bit (RHEL 6.3). I
> try to get my application to start after another application is started
> which is started at run level 50, so I set the RUN_LEVEL parameter to
> 60. If I now look in the /etc/rcX.d directories I can see that my
> application is started as S24 and stopped as K80. How can I get my
> application to be started after run level 50?
>
> Lars
>
>
> ------------------------------------------------------------------------------
> Comprehensive Server Monitoring with Site24x7.
> Monitor 10 servers for $9/Month.
> Get alerted through email, SMS, voice calls or mobile push notifications.
> Take corrective actions from your mobile device.
>
> http://pubads.g.doubleclick.net/gampad/clk?id=154624111&iu=/4140/ostg.clktrk
> _______________________________________________
> Wrapper-user mailing list
> Wra...@li...
> https://lists.sourceforge.net/lists/listinfo/wrapper-user
>

Re: [Wrapper-user] wrapper 3.5.25 memory leak on CentOS 7

[Aleksey C. <ale...@gm...>]

Thanks for the reply!

Unfortunately I can't update glibc unless it will be updated by upstream 
(CentOS maintainers).

So I patched wrapper sources using previously attached patches and quick 
test shows memory leak is fixed!

Do you have plans to release a new wrapper version includes the proposed 
fixes? Don't want to maintain custom wrapper version :)

Regards,
Aleksey


On 11/17/2014 11:39 AM, Tim Lammens wrote:
> There are two bugs at play here. One is in glibc, the other in the 
> wrapper itself.
>
> - glibc leak:
> https://sourceware.org/bugzilla/show_bug.cgi?id=17370
> It is fixed in newer releases of glibc, so if you can upgrade your 
> glibc libraries please do so.
>
> If you can not upgrade your glibc libraries, see patch 
> 02.workaround-memory-leak-in-ftell.
>
> - wrapper leak:
> I've also attached a fix for the memory leak in logging in the wrapper 
> itself in patch 01.fixes-memory-leak-in-error-logging.
> It uses the same solution as the windows version to prevent using up 
> all the memory.
>
> Regards,
> Tim
>

Re: [Wrapper-user] wrapper 3.5.25 memory leak on CentOS 7

[Tim L. <tim...@gm...>]

There are two bugs at play here. One is in glibc, the other in the wrapper
itself.

- glibc leak:
https://sourceware.org/bugzilla/show_bug.cgi?id=17370
It is fixed in newer releases of glibc, so if you can upgrade your glibc
libraries please do so.

If you can not upgrade your glibc libraries, see patch
02.workaround-memory-leak-in-ftell.

- wrapper leak:
I've also attached a fix for the memory leak in logging in the wrapper
itself in patch 01.fixes-memory-leak-in-error-logging.
It uses the same solution as the windows version to prevent using up all
the memory.

Regards,
Tim


On Sun, Nov 16, 2014 at 11:51 AM, Aleksey Chudov <ale...@gm...>
wrote:

> Finally I found that memory leak relates to wrapper.logfile options. As
> soon as I comment the following options in configuration file the
> problem goes away.
>
> wrapper.logfile=/var/log/myapp.log
> wrapper.logfile.format=LPTM
> wrapper.logfile.loglevel=INFO
> wrapper.logfile.maxsize=100m
> wrapper.logfile.maxfiles=10
>
> After enabling options one by one I found two problem options. As soon
> as I uncomment the following options wrapper starts to eat memory.
>
> wrapper.logfile.maxsize=100m
> wrapper.logfile.maxfiles=10
>
> I can reproduce the problem.
>
>
> Regards,
> Aleksey
>
>
>
> ------------------------------------------------------------------------------
> Comprehensive Server Monitoring with Site24x7.
> Monitor 10 servers for $9/Month.
> Get alerted through email, SMS, voice calls or mobile push notifications.
> Take corrective actions from your mobile device.
>
> http://pubads.g.doubleclick.net/gampad/clk?id=154624111&iu=/4140/ostg.clktrk
> _______________________________________________
> Wrapper-user mailing list
> Wra...@li...
> https://lists.sourceforge.net/lists/listinfo/wrapper-user
>

Re: [Wrapper-user] wrapper 3.5.25 memory leak on CentOS 7

[Aleksey C. <ale...@gm...>]

Finally I found that memory leak relates to wrapper.logfile options. As 
soon as I comment the following options in configuration file the 
problem goes away.

wrapper.logfile=/var/log/myapp.log
wrapper.logfile.format=LPTM
wrapper.logfile.loglevel=INFO
wrapper.logfile.maxsize=100m
wrapper.logfile.maxfiles=10

After enabling options one by one I found two problem options. As soon 
as I uncomment the following options wrapper starts to eat memory.

wrapper.logfile.maxsize=100m
wrapper.logfile.maxfiles=10

I can reproduce the problem.


Regards,
Aleksey

[Wrapper-user] wrapper 3.5.25 memory leak on CentOS 7

[Aleksey C. <ale...@gm...>]

Hi,

We are using wrapper 3.5.25 community edition on a bunch of high loaded 
servers without any issues.

Currently I'm testing the same wrapper and java application on a new 
CentOS 7 server and facing a huge memory leak by the wrapper process.

KiB Mem:  16423540 total, 16210652 used,   212888 free,      728 buffers
KiB Swap:  4194300 total,        0 used,  4194300 free.   262020 cached Mem

   PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
20381 root      20   0 7531520 6.979g    808 S   0.0 44.6   3:00.90 wrapper
20383 root      20   0 19.037g 6.959g   8860 S   1.3 44.4   1175:24 java


At the same time there where a lot of wrapper messages dropped by 
systemd-journal because of rate limiting

Nov 15 13:28:09 srv94 journal: Suppressed 27674 messages from 
/user.slice/user-1013.slice
Nov 15 13:28:39 srv94 journal: Suppressed 29040 messages from 
/user.slice/user-1013.slice
Nov 15 13:29:09 srv94 journal: Suppressed 28509 messages from 
/user.slice/user-1013.slice
Nov 15 13:29:39 srv94 journal: Suppressed 19124 messages from 
/user.slice/user-1013.slice
Nov 15 13:30:09 srv94 journal: Suppressed 18889 messages from 
/user.slice/user-1013.slice
Nov 15 13:30:40 srv94 journal: Suppressed 10076 messages from 
/user.slice/user-1013.slice

Do not ask why so many messages :) It's normal behaviour for our 
application.

Seem like wrapper leak memory on dropped syslog messages. May be some 
internal buffers are not cleaned?

We are using the following log related settings

wrapper.console.format=PM
wrapper.console.loglevel=INFO

wrapper.logfile=/var/log/myapp.log
wrapper.logfile.format=LPTM
wrapper.logfile.loglevel=INFO
wrapper.logfile.maxsize=100m
wrapper.logfile.maxfiles=10

wrapper.syslog.facility=LOCAL4
wrapper.syslog.loglevel=INFO
wrapper.syslog.ident=myapp


I can reproduce the problem.


Regards,
Aleksey

[Wrapper-user] RUN_LEVEL not working

[Lars S. <la...@if...>]

Hi

I am using the wrapper in version 3.5.25 on Linux 32-bit (RHEL 6.3). I 
try to get my application to start after another application is started 
which is started at run level 50, so I set the RUN_LEVEL parameter to 
60. If I now look in the /etc/rcX.d directories I can see that my 
application is started as S24 and stopped as K80. How can I get my 
application to be started after run level 50?

Lars