Menu
▾
▴
Re: [Wrapper-user] Advice on stopping wrapper from triggering JVM restarts
|
From: Tim L. <tim...@gm...> - 2014-12-10 16:06:31
|
That is indeed the right thread. Regards, Tim On Wed, Dec 10, 2014 at 5:03 PM, Casey Jordan <cas...@jo...> wrote: > Hi Tim, > > Thanks for the info, is this the thread you speak of: > http://sourceforge.net/p/wrapper/mailman/message/33048371/ > > If not could you point me to the right one? > > Thanks > > On Wed, Dec 10, 2014 at 10:58 AM, Tim Lammens <tim...@gm...> > wrote: > >> You are most probably experiencing a glibc bug as I mentioned before in >> another thread. The bug was fixed in recent glibc versions. >> Try updating your glibc version for CentOS. If that is not possible you >> can apply the patches I provided in a different thread on this mailing list. >> >> Regards, >> Tim >> >> On Wed, Dec 10, 2014 at 4:49 PM, Casey Jordan <cas...@jo...> >> wrote: >> >>> Ok so this happened again and I was able to capture some additional log >>> information (not from wrapper.debug though as I can't update this server >>> just yet) >>> >>> I got this in our logs: >>> >>> INFO | jvm 1 | 2014/12/09 23:15:50 | <br></br>Max Mem: 2223 MB >>> INFO | jvm 1 | 2014/12/09 23:15:50 | <br></br>Total Mem: 2204 MB >>> INFO | jvm 1 | 2014/12/09 23:15:50 | <br></br>Used Mem: 1141 MB >>> INFO | jvm 1 | 2014/12/09 23:15:50 | <br></br>Raw free-mem: 1082 MB >>> INFO | jvm 1 | 2014/12/09 23:15:50 | <br></br>Total descriptors: 4096 >>> INFO | jvm 1 | 2014/12/09 23:15:50 | <br></br>Open descriptors: 556 >>> INFO | jvm 1 | 2014/12/09 23:15:50 | <br></br>Available descriptors: 3540 >>> INFO | jvm 1 | 2014/12/09 23:15:50 | <br></br>Free disk space: 18 GB >>> INFO | jvm 1 | 2014/12/09 23:15:50 | <br></br> >>> INFO | wrapper | 2014/12/09 23:17:09 | Wrapper Process has not received any CPU time for 1 seconds. Extending timeouts. >>> STATUS | wrapper | 2014/12/09 23:17:09 | Pinging the JVM took 7 seconds to respond. >>> INFO | wrapper | 2014/12/09 23:17:09 | Wrapper Process has not received any CPU time for 5 seconds. Extending timeouts. >>> STATUS | wrapper | 2014/12/09 23:17:09 | JVM received a signal SIGKILL (9). >>> STATUS | wrapper | 2014/12/09 23:17:09 | JVM process is gone. >>> ERROR | wrapper | 2014/12/09 23:17:09 | JVM exited unexpectedly. >>> STATUS | wrapper | 2014/12/09 23:17:09 | Automatic JVM Restarts disabled. Shutting down. >>> STATUS | wrapper | 2014/12/09 23:17:09 | <-- Wrapper Stopped >>> >>> >>> and I found this in the /var/log/messages >>> >>> Dec 9 23:17:09 jorsek-home2 kernel: Out of memory: Kill process 10262 >>> (java) score 591 or sacrifice child >>> Dec 9 23:17:09 jorsek-home2 kernel: Killed process 10262 (java) >>> total-vm:6478728kB, anon-rss:2600344kB, file-rss:0kB >>> >>> So this does appear to be the kernel killing the process, but according >>> to java I have plenty of memory available (>1GB). >>> >>> I know this is not really a wrapper issue any longer, but I would be >>> very appreciative if anyone can give me some advice or point me to some >>> resources that might help figure this out. >>> >>> For reference this is a Centos7 box with 4GB of RAM and a 1GB swap >>> space, and 40GB SSD drive. >>> >>> Tonight I am going to add NewRelic monitoring to this server to see if I >>> can get a better picture of what is going on. >>> >>> Thanks! >>> >>> On Tue, Dec 2, 2014 at 7:30 PM, Roberto Espinoza < >>> rob...@ta...> wrote: >>> >>>> Casey, >>>> >>>> Have you checked the kernel logs to see if it wasn't killed because >>>> your system ran out of memory? >>>> >>>> Certainly there is no way to know this inside the JVM because the >>>> kernel out of memory killer will just score the current processes and >>>> issue a kill. >>>> >>>> On Wed, Dec 3, 2014 at 12:39 AM, Casey Jordan <cas...@jo...> >>>> wrote: >>>> > Hi Leif, >>>> > >>>> > Thanks for the feedback. This is quite a mystery then because I am >>>> 100% sure >>>> > that the process wasn't killed manually or by any other systems we >>>> have >>>> > running. (This is just a standard CentOS 7 setup) >>>> > >>>> > At about the same time that this log appeared I was running a very >>>> heavy >>>> > build process in a separate JVM instance. >>>> > >>>> > It seems more than just coincidental that this happened at the same >>>> time. I >>>> > assume that if the JVM crashed it wouldn't have received a SIG 9? Can >>>> you >>>> > think of any scenario where this makes sense? Perhaps I am missing >>>> some >>>> > knowledge of java architecture here that is making this mysterious. >>>> > >>>> > Thanks >>>> > >>>> > >>>> > On Tue, Dec 2, 2014 at 4:27 AM, Leif Mortenson >>>> > <lei...@ta...> wrote: >>>> >> >>>> >> Casey, >>>> >> The configuration you specified will disable all pings and cpu >>>> warnings >>>> >> and should do what you want. >>>> >> The Wrapper should never timeout in this case. There are other >>>> things >>>> >> such as a JVM crash would would still result in a restart after the >>>> fact. >>>> >> >>>> >> The log that you send shows that the JVM will killed with a kill >>>> -9. Not >>>> >> sure if that was a test on your part, but if the source of the >>>> signal has >>>> >> permission to kill the JVM's process then there is nothing we can do >>>> to >>>> >> prevent that. >>>> >> >>>> >> Cheers, >>>> >> Leif >>>> >> >>>> >> On Tue, Dec 2, 2014 at 12:16 PM, Casey Jordan < >>>> cas...@jo...> >>>> >> wrote: >>>> >>> >>>> >>> Hi all, >>>> >>> >>>> >>> I am sure this is a common question, but I am having trouble >>>> extracting >>>> >>> details I need from the documentation. From time to time we get a >>>> situation >>>> >>> where our JVM gets killed and we see something like this in the >>>> logs: >>>> >>> >>>> >>> INFO | wrapper | 2014/12/02 02:23:18 | Wrapper Process has not >>>> >>> received any CPU time for 1 seconds. Extending timeouts. >>>> >>> STATUS | wrapper | 2014/12/02 02:23:37 | JVM received a signal >>>> SIGKILL >>>> >>> (9). >>>> >>> STATUS | wrapper | 2014/12/02 02:23:37 | JVM process is gone. >>>> >>> ERROR | wrapper | 2014/12/02 02:23:37 | JVM exited unexpectedly. >>>> >>> STATUS | wrapper | 2014/12/02 02:23:38 | Automatic JVM Restarts >>>> >>> disabled. Shutting down. >>>> >>> STATUS | wrapper | 2014/12/02 02:23:38 | <-- Wrapper Stopped >>>> >>> >>>> >>> >>>> >>> This always happens in the case where some other process is eating >>>> up all >>>> >>> the cpu for more than a few seconds. >>>> >>> >>>> >>> It's crucial for us that the wrapper never, ever restarts the JVM >>>> as this >>>> >>> is something we always want to look into manually. >>>> >>> >>>> >>> So my question is, what is the best way to make sure this doesn't >>>> happen? >>>> >>> >>>> >>> I am using community version 3.5.17, and have the following values >>>> in our >>>> >>> wrapper.conf: >>>> >>> >>>> >>> >>>> #******************************************************************** >>>> >>> # Timeouts >>>> >>> >>>> #******************************************************************** >>>> >>> wrapper.ping.timeout=0 >>>> >>> wrapper.ping.timeout.action=DEBUG >>>> >>> wrapper.cpu.timeout=0 >>>> >>> wrapper.startup.timeout=300 >>>> >>> >>>> >>> Any help is much appreciated. >>>> >>> >>>> >>> Thanks! >>>> >>> >>>> >> >>>> >> >>>> >> >>>> ------------------------------------------------------------------------------ >>>> >> Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server >>>> >> from Actuate! Instantly Supercharge Your Business Reports and >>>> Dashboards >>>> >> with Interactivity, Sharing, Native Excel Exports, App Integration & >>>> more >>>> >> Get technology previously reserved for billion-dollar corporations, >>>> FREE >>>> >> >>>> >> >>>> http://pubads.g.doubleclick.net/gampad/clk?id=157005751&iu=/4140/ostg.clktrk >>>> >> _______________________________________________ >>>> >> Wrapper-user mailing list >>>> >> Wra...@li... >>>> >> https://lists.sourceforge.net/lists/listinfo/wrapper-user >>>> >> >>>> > >>>> > >>>> > >>>> > -- >>>> > -- >>>> > Casey Jordan >>>> > easyDITA a product of Jorsek LLC >>>> > "CaseyDJordan" on LinkedIn, Twitter & Facebook >>>> > (585) 348 7399 >>>> > easydita.com >>>> > >>>> > >>>> > This message is intended only for the use of the Addressee(s) and may >>>> > contain information that is privileged, confidential, and/or exempt >>>> from >>>> > disclosure under applicable law. If you are not the intended >>>> recipient, >>>> > please be advised that any disclosure copying, distribution, or use >>>> of >>>> > the information contained herein is prohibited. If you have received >>>> > this communication in error, please destroy all copies of the message, >>>> > whether in electronic or hard copy format, as well as attachments, and >>>> > immediately contact the sender by replying to this e-mail or by phone. >>>> > Thank you. >>>> > >>>> > >>>> ------------------------------------------------------------------------------ >>>> > Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server >>>> > from Actuate! Instantly Supercharge Your Business Reports and >>>> Dashboards >>>> > with Interactivity, Sharing, Native Excel Exports, App Integration & >>>> more >>>> > Get technology previously reserved for billion-dollar corporations, >>>> FREE >>>> > >>>> http://pubads.g.doubleclick.net/gampad/clk?id=157005751&iu=/4140/ostg.clktrk >>>> > _______________________________________________ >>>> > Wrapper-user mailing list >>>> > Wra...@li... >>>> > https://lists.sourceforge.net/lists/listinfo/wrapper-user >>>> > >>>> >>>> >>>> >>>> -- >>>> Roberto Espinoza >>>> Tanuki Software, Ltd. >>>> 6-16-7-1001 Nishi-Kasai, Edogawa-ku >>>> Tokyo 134-0088 Japan >>>> Tel/Fax: +81-3-3878-3211 >>>> http://www.tanukisoftware.com >>>> rob...@ta... >>>> >>>> >>>> ------------------------------------------------------------------------------ >>>> Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server >>>> from Actuate! Instantly Supercharge Your Business Reports and Dashboards >>>> with Interactivity, Sharing, Native Excel Exports, App Integration & >>>> more >>>> Get technology previously reserved for billion-dollar corporations, FREE >>>> >>>> http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk >>>> _______________________________________________ >>>> Wrapper-user mailing list >>>> Wra...@li... >>>> https://lists.sourceforge.net/lists/listinfo/wrapper-user >>>> >>> >>> >>> >>> -- >>> -- >>> Casey Jordan >>> easyDITA a product of Jorsek LLC >>> "CaseyDJordan" on LinkedIn, Twitter & Facebook >>> (585) 348 7399 >>> easydita.com >>> >>> >>> This message is intended only for the use of the Addressee(s) and may >>> contain information that is privileged, confidential, and/or exempt from >>> disclosure under applicable law. If you are not the intended recipient, >>> please be advised that any disclosure copying, distribution, or use of >>> the information contained herein is prohibited. If you have received >>> this communication in error, please destroy all copies of the message, >>> whether in electronic or hard copy format, as well as attachments, and >>> immediately contact the sender by replying to this e-mail or by phone. >>> Thank you. >>> >>> >>> ------------------------------------------------------------------------------ >>> Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server >>> from Actuate! Instantly Supercharge Your Business Reports and Dashboards >>> with Interactivity, Sharing, Native Excel Exports, App Integration & more >>> Get technology previously reserved for billion-dollar corporations, FREE >>> >>> http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk >>> _______________________________________________ >>> Wrapper-user mailing list >>> Wra...@li... >>> https://lists.sourceforge.net/lists/listinfo/wrapper-user >>> >>> >> >> >> ------------------------------------------------------------------------------ >> Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server >> from Actuate! Instantly Supercharge Your Business Reports and Dashboards >> with Interactivity, Sharing, Native Excel Exports, App Integration & more >> Get technology previously reserved for billion-dollar corporations, FREE >> >> http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk >> _______________________________________________ >> Wrapper-user mailing list >> Wra...@li... >> https://lists.sourceforge.net/lists/listinfo/wrapper-user >> >> > > > -- > -- > Casey Jordan > easyDITA a product of Jorsek LLC > "CaseyDJordan" on LinkedIn, Twitter & Facebook > (585) 348 7399 > easydita.com > > > This message is intended only for the use of the Addressee(s) and may > contain information that is privileged, confidential, and/or exempt from > disclosure under applicable law. If you are not the intended recipient, > please be advised that any disclosure copying, distribution, or use of > the information contained herein is prohibited. If you have received > this communication in error, please destroy all copies of the message, > whether in electronic or hard copy format, as well as attachments, and > immediately contact the sender by replying to this e-mail or by phone. > Thank you. > > > ------------------------------------------------------------------------------ > Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server > from Actuate! Instantly Supercharge Your Business Reports and Dashboards > with Interactivity, Sharing, Native Excel Exports, App Integration & more > Get technology previously reserved for billion-dollar corporations, FREE > > http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk > _______________________________________________ > Wrapper-user mailing list > Wra...@li... > https://lists.sourceforge.net/lists/listinfo/wrapper-user > > |
