Menu
▾
▴
Re: [Wrapper-user] Advice on stopping wrapper from triggering JVM restarts
|
From: Casey J. <cas...@jo...> - 2014-12-10 15:49:35
|
Ok so this happened again and I was able to capture some additional log information (not from wrapper.debug though as I can't update this server just yet) I got this in our logs: INFO | jvm 1 | 2014/12/09 23:15:50 | <br></br>Max Mem: 2223 MB INFO | jvm 1 | 2014/12/09 23:15:50 | <br></br>Total Mem: 2204 MB INFO | jvm 1 | 2014/12/09 23:15:50 | <br></br>Used Mem: 1141 MB INFO | jvm 1 | 2014/12/09 23:15:50 | <br></br>Raw free-mem: 1082 MB INFO | jvm 1 | 2014/12/09 23:15:50 | <br></br>Total descriptors: 4096 INFO | jvm 1 | 2014/12/09 23:15:50 | <br></br>Open descriptors: 556 INFO | jvm 1 | 2014/12/09 23:15:50 | <br></br>Available descriptors: 3540 INFO | jvm 1 | 2014/12/09 23:15:50 | <br></br>Free disk space: 18 GB INFO | jvm 1 | 2014/12/09 23:15:50 | <br></br> INFO | wrapper | 2014/12/09 23:17:09 | Wrapper Process has not received any CPU time for 1 seconds. Extending timeouts. STATUS | wrapper | 2014/12/09 23:17:09 | Pinging the JVM took 7 seconds to respond. INFO | wrapper | 2014/12/09 23:17:09 | Wrapper Process has not received any CPU time for 5 seconds. Extending timeouts. STATUS | wrapper | 2014/12/09 23:17:09 | JVM received a signal SIGKILL (9). STATUS | wrapper | 2014/12/09 23:17:09 | JVM process is gone. ERROR | wrapper | 2014/12/09 23:17:09 | JVM exited unexpectedly. STATUS | wrapper | 2014/12/09 23:17:09 | Automatic JVM Restarts disabled. Shutting down. STATUS | wrapper | 2014/12/09 23:17:09 | <-- Wrapper Stopped and I found this in the /var/log/messages Dec 9 23:17:09 jorsek-home2 kernel: Out of memory: Kill process 10262 (java) score 591 or sacrifice child Dec 9 23:17:09 jorsek-home2 kernel: Killed process 10262 (java) total-vm:6478728kB, anon-rss:2600344kB, file-rss:0kB So this does appear to be the kernel killing the process, but according to java I have plenty of memory available (>1GB). I know this is not really a wrapper issue any longer, but I would be very appreciative if anyone can give me some advice or point me to some resources that might help figure this out. For reference this is a Centos7 box with 4GB of RAM and a 1GB swap space, and 40GB SSD drive. Tonight I am going to add NewRelic monitoring to this server to see if I can get a better picture of what is going on. Thanks! On Tue, Dec 2, 2014 at 7:30 PM, Roberto Espinoza < rob...@ta...> wrote: > Casey, > > Have you checked the kernel logs to see if it wasn't killed because > your system ran out of memory? > > Certainly there is no way to know this inside the JVM because the > kernel out of memory killer will just score the current processes and > issue a kill. > > On Wed, Dec 3, 2014 at 12:39 AM, Casey Jordan <cas...@jo...> > wrote: > > Hi Leif, > > > > Thanks for the feedback. This is quite a mystery then because I am 100% > sure > > that the process wasn't killed manually or by any other systems we have > > running. (This is just a standard CentOS 7 setup) > > > > At about the same time that this log appeared I was running a very heavy > > build process in a separate JVM instance. > > > > It seems more than just coincidental that this happened at the same > time. I > > assume that if the JVM crashed it wouldn't have received a SIG 9? Can you > > think of any scenario where this makes sense? Perhaps I am missing some > > knowledge of java architecture here that is making this mysterious. > > > > Thanks > > > > > > On Tue, Dec 2, 2014 at 4:27 AM, Leif Mortenson > > <lei...@ta...> wrote: > >> > >> Casey, > >> The configuration you specified will disable all pings and cpu warnings > >> and should do what you want. > >> The Wrapper should never timeout in this case. There are other things > >> such as a JVM crash would would still result in a restart after the > fact. > >> > >> The log that you send shows that the JVM will killed with a kill -9. > Not > >> sure if that was a test on your part, but if the source of the signal > has > >> permission to kill the JVM's process then there is nothing we can do to > >> prevent that. > >> > >> Cheers, > >> Leif > >> > >> On Tue, Dec 2, 2014 at 12:16 PM, Casey Jordan <cas...@jo...> > >> wrote: > >>> > >>> Hi all, > >>> > >>> I am sure this is a common question, but I am having trouble extracting > >>> details I need from the documentation. From time to time we get a > situation > >>> where our JVM gets killed and we see something like this in the logs: > >>> > >>> INFO | wrapper | 2014/12/02 02:23:18 | Wrapper Process has not > >>> received any CPU time for 1 seconds. Extending timeouts. > >>> STATUS | wrapper | 2014/12/02 02:23:37 | JVM received a signal SIGKILL > >>> (9). > >>> STATUS | wrapper | 2014/12/02 02:23:37 | JVM process is gone. > >>> ERROR | wrapper | 2014/12/02 02:23:37 | JVM exited unexpectedly. > >>> STATUS | wrapper | 2014/12/02 02:23:38 | Automatic JVM Restarts > >>> disabled. Shutting down. > >>> STATUS | wrapper | 2014/12/02 02:23:38 | <-- Wrapper Stopped > >>> > >>> > >>> This always happens in the case where some other process is eating up > all > >>> the cpu for more than a few seconds. > >>> > >>> It's crucial for us that the wrapper never, ever restarts the JVM as > this > >>> is something we always want to look into manually. > >>> > >>> So my question is, what is the best way to make sure this doesn't > happen? > >>> > >>> I am using community version 3.5.17, and have the following values in > our > >>> wrapper.conf: > >>> > >>> > #******************************************************************** > >>> # Timeouts > >>> > #******************************************************************** > >>> wrapper.ping.timeout=0 > >>> wrapper.ping.timeout.action=DEBUG > >>> wrapper.cpu.timeout=0 > >>> wrapper.startup.timeout=300 > >>> > >>> Any help is much appreciated. > >>> > >>> Thanks! > >>> > >> > >> > >> > ------------------------------------------------------------------------------ > >> Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server > >> from Actuate! Instantly Supercharge Your Business Reports and Dashboards > >> with Interactivity, Sharing, Native Excel Exports, App Integration & > more > >> Get technology previously reserved for billion-dollar corporations, FREE > >> > >> > http://pubads.g.doubleclick.net/gampad/clk?id=157005751&iu=/4140/ostg.clktrk > >> _______________________________________________ > >> Wrapper-user mailing list > >> Wra...@li... > >> https://lists.sourceforge.net/lists/listinfo/wrapper-user > >> > > > > > > > > -- > > -- > > Casey Jordan > > easyDITA a product of Jorsek LLC > > "CaseyDJordan" on LinkedIn, Twitter & Facebook > > (585) 348 7399 > > easydita.com > > > > > > This message is intended only for the use of the Addressee(s) and may > > contain information that is privileged, confidential, and/or exempt from > > disclosure under applicable law. If you are not the intended recipient, > > please be advised that any disclosure copying, distribution, or use of > > the information contained herein is prohibited. If you have received > > this communication in error, please destroy all copies of the message, > > whether in electronic or hard copy format, as well as attachments, and > > immediately contact the sender by replying to this e-mail or by phone. > > Thank you. > > > > > ------------------------------------------------------------------------------ > > Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server > > from Actuate! Instantly Supercharge Your Business Reports and Dashboards > > with Interactivity, Sharing, Native Excel Exports, App Integration & more > > Get technology previously reserved for billion-dollar corporations, FREE > > > http://pubads.g.doubleclick.net/gampad/clk?id=157005751&iu=/4140/ostg.clktrk > > _______________________________________________ > > Wrapper-user mailing list > > Wra...@li... > > https://lists.sourceforge.net/lists/listinfo/wrapper-user > > > > > > -- > Roberto Espinoza > Tanuki Software, Ltd. > 6-16-7-1001 Nishi-Kasai, Edogawa-ku > Tokyo 134-0088 Japan > Tel/Fax: +81-3-3878-3211 > http://www.tanukisoftware.com > rob...@ta... > > > ------------------------------------------------------------------------------ > Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server > from Actuate! Instantly Supercharge Your Business Reports and Dashboards > with Interactivity, Sharing, Native Excel Exports, App Integration & more > Get technology previously reserved for billion-dollar corporations, FREE > > http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk > _______________________________________________ > Wrapper-user mailing list > Wra...@li... > https://lists.sourceforge.net/lists/listinfo/wrapper-user > -- -- Casey Jordan easyDITA a product of Jorsek LLC "CaseyDJordan" on LinkedIn, Twitter & Facebook (585) 348 7399 easydita.com This message is intended only for the use of the Addressee(s) and may contain information that is privileged, confidential, and/or exempt from disclosure under applicable law. If you are not the intended recipient, please be advised that any disclosure copying, distribution, or use of the information contained herein is prohibited. If you have received this communication in error, please destroy all copies of the message, whether in electronic or hard copy format, as well as attachments, and immediately contact the sender by replying to this e-mail or by phone. Thank you. |
