Re: [Wrapper-user] Advice on stopping wrapper from triggering JVM restarts

SourceForge Headquarters 1320 Columbia Street Suite 310 San Diego, CA 92101 +1 (858) 422-6466

By the way, this is my glibc version: glibc-2.17-55.el7_0.1.x86_64

On Wed, Dec 10, 2014 at 11:03 AM, Casey Jordan <cas...@jo...>
wrote:

> Hi Tim,
>
> Thanks for the info, is this the thread you speak of:
> http://sourceforge.net/p/wrapper/mailman/message/33048371/
>
> If not could you point me to the right one?
>
> Thanks
>
> On Wed, Dec 10, 2014 at 10:58 AM, Tim Lammens <tim...@gm...>
> wrote:
>
>> You are most probably experiencing a glibc bug as I mentioned before in
>> another thread. The bug was fixed in recent glibc versions.
>> Try updating your glibc version for CentOS. If that is not possible you
>> can apply the patches I provided in a different thread on this mailing list.
>>
>> Regards,
>> Tim
>>
>> On Wed, Dec 10, 2014 at 4:49 PM, Casey Jordan <cas...@jo...>
>> wrote:
>>
>>> Ok so this happened again and I was able to capture some additional log
>>> information (not from wrapper.debug though as I can't update this server
>>> just yet)
>>>
>>> I got this in our logs:
>>>
>>> INFO   | jvm 1    | 2014/12/09 23:15:50 | <br></br>Max Mem: 2223 MB
>>> INFO   | jvm 1    | 2014/12/09 23:15:50 | <br></br>Total Mem: 2204 MB
>>> INFO   | jvm 1    | 2014/12/09 23:15:50 | <br></br>Used Mem: 1141 MB
>>> INFO   | jvm 1    | 2014/12/09 23:15:50 | <br></br>Raw free-mem: 1082 MB
>>> INFO   | jvm 1    | 2014/12/09 23:15:50 | <br></br>Total descriptors: 4096
>>> INFO   | jvm 1    | 2014/12/09 23:15:50 | <br></br>Open descriptors: 556
>>> INFO   | jvm 1    | 2014/12/09 23:15:50 | <br></br>Available descriptors: 3540
>>> INFO   | jvm 1    | 2014/12/09 23:15:50 | <br></br>Free disk space: 18 GB
>>> INFO   | jvm 1    | 2014/12/09 23:15:50 | <br></br>
>>> INFO   | wrapper  | 2014/12/09 23:17:09 | Wrapper Process has not received any CPU time for 1 seconds.  Extending timeouts.
>>> STATUS | wrapper  | 2014/12/09 23:17:09 | Pinging the JVM took 7 seconds to respond.
>>> INFO   | wrapper  | 2014/12/09 23:17:09 | Wrapper Process has not received any CPU time for 5 seconds.  Extending timeouts.
>>> STATUS | wrapper  | 2014/12/09 23:17:09 | JVM received a signal SIGKILL (9).
>>> STATUS | wrapper  | 2014/12/09 23:17:09 | JVM process is gone.
>>> ERROR  | wrapper  | 2014/12/09 23:17:09 | JVM exited unexpectedly.
>>> STATUS | wrapper  | 2014/12/09 23:17:09 | Automatic JVM Restarts disabled.  Shutting down.
>>> STATUS | wrapper  | 2014/12/09 23:17:09 | <-- Wrapper Stopped
>>>
>>>
>>> and I found this in the /var/log/messages
>>>
>>> Dec  9 23:17:09 jorsek-home2 kernel: Out of memory: Kill process 10262
>>> (java) score 591 or sacrifice child
>>> Dec  9 23:17:09 jorsek-home2 kernel: Killed process 10262 (java)
>>> total-vm:6478728kB, anon-rss:2600344kB, file-rss:0kB
>>>
>>> So this does appear to be the kernel killing the process, but according
>>> to java I have plenty of memory available (>1GB).
>>>
>>> I know this is not really a wrapper issue any longer, but I would be
>>> very appreciative if anyone can give me some advice or point me to some
>>> resources that might help figure this out.
>>>
>>> For reference this is a Centos7 box with 4GB of RAM and a 1GB swap
>>> space, and 40GB SSD drive.
>>>
>>> Tonight I am going to add NewRelic monitoring to this server to see if I
>>> can get a better picture of what is going on.
>>>
>>> Thanks!
>>>
>>> On Tue, Dec 2, 2014 at 7:30 PM, Roberto Espinoza <
>>> rob...@ta...> wrote:
>>>
>>>> Casey,
>>>>
>>>> Have you checked the kernel logs to see if it wasn't killed because
>>>> your system ran out of memory?
>>>>
>>>> Certainly there is no way to know this inside the JVM because the
>>>> kernel out of memory killer will just score the current processes and
>>>> issue a kill.
>>>>
>>>> On Wed, Dec 3, 2014 at 12:39 AM, Casey Jordan <cas...@jo...>
>>>> wrote:
>>>> > Hi Leif,
>>>> >
>>>> > Thanks for the feedback. This is quite a mystery then because I am
>>>> 100% sure
>>>> > that the process wasn't killed manually or by any other systems we
>>>> have
>>>> > running. (This is just a standard CentOS 7 setup)
>>>> >
>>>> > At about the same time that this log appeared I was running a very
>>>> heavy
>>>> > build process in a separate JVM instance.
>>>> >
>>>> > It seems more than just coincidental that this happened at the same
>>>> time. I
>>>> > assume that if the JVM crashed it wouldn't have received a SIG 9? Can
>>>> you
>>>> > think of any scenario where this makes sense? Perhaps I am missing
>>>> some
>>>> > knowledge of java architecture here that is making this mysterious.
>>>> >
>>>> > Thanks
>>>> >
>>>> >
>>>> > On Tue, Dec 2, 2014 at 4:27 AM, Leif Mortenson
>>>> > <lei...@ta...> wrote:
>>>> >>
>>>> >> Casey,
>>>> >> The configuration you specified will disable all pings and cpu
>>>> warnings
>>>> >> and should do what you want.
>>>> >> The Wrapper should never timeout  in this case.  There are other
>>>> things
>>>> >> such as a JVM crash would would still result in a restart after the
>>>> fact.
>>>> >>
>>>> >> The log that you send shows that the JVM will killed with a kill
>>>> -9.  Not
>>>> >> sure if that was a test on your part, but if the source of the
>>>> signal has
>>>> >> permission to kill the JVM's process then there is nothing we can do
>>>> to
>>>> >> prevent that.
>>>> >>
>>>> >> Cheers,
>>>> >> Leif
>>>> >>
>>>> >> On Tue, Dec 2, 2014 at 12:16 PM, Casey Jordan <
>>>> cas...@jo...>
>>>> >> wrote:
>>>> >>>
>>>> >>> Hi all,
>>>> >>>
>>>> >>> I am sure this is a common question, but I am having trouble
>>>> extracting
>>>> >>> details I need from the documentation. From time to time we get a
>>>> situation
>>>> >>> where our JVM gets killed and we see something like this in the
>>>> logs:
>>>> >>>
>>>> >>> INFO   | wrapper  | 2014/12/02 02:23:18 | Wrapper Process has not
>>>> >>> received any CPU time for 1 seconds.  Extending timeouts.
>>>> >>> STATUS | wrapper  | 2014/12/02 02:23:37 | JVM received a signal
>>>> SIGKILL
>>>> >>> (9).
>>>> >>> STATUS | wrapper  | 2014/12/02 02:23:37 | JVM process is gone.
>>>> >>> ERROR  | wrapper  | 2014/12/02 02:23:37 | JVM exited unexpectedly.
>>>> >>> STATUS | wrapper  | 2014/12/02 02:23:38 | Automatic JVM Restarts
>>>> >>> disabled.  Shutting down.
>>>> >>> STATUS | wrapper  | 2014/12/02 02:23:38 | <-- Wrapper Stopped
>>>> >>>
>>>> >>>
>>>> >>> This always happens in the case where some other process is eating
>>>> up all
>>>> >>> the cpu for more than a few seconds.
>>>> >>>
>>>> >>> It's crucial for us that the wrapper never, ever restarts the JVM
>>>> as this
>>>> >>> is something we always want to look into manually.
>>>> >>>
>>>> >>> So my question is, what is the best way to make sure this doesn't
>>>> happen?
>>>> >>>
>>>> >>> I am using community version 3.5.17, and have the following values
>>>> in our
>>>> >>> wrapper.conf:
>>>> >>>
>>>> >>>
>>>> #********************************************************************
>>>> >>>    # Timeouts
>>>> >>>
>>>> #********************************************************************
>>>> >>>    wrapper.ping.timeout=0
>>>> >>>    wrapper.ping.timeout.action=DEBUG
>>>> >>>    wrapper.cpu.timeout=0
>>>> >>>    wrapper.startup.timeout=300
>>>> >>>
>>>> >>> Any help is much appreciated.
>>>> >>>
>>>> >>> Thanks!
>>>> >>>
>>>> >>
>>>> >>
>>>> >>
>>>> ------------------------------------------------------------------------------
>>>> >> Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
>>>> >> from Actuate! Instantly Supercharge Your Business Reports and
>>>> Dashboards
>>>> >> with Interactivity, Sharing, Native Excel Exports, App Integration &
>>>> more
>>>> >> Get technology previously reserved for billion-dollar corporations,
>>>> FREE
>>>> >>
>>>> >>
>>>> http://pubads.g.doubleclick.net/gampad/clk?id=157005751&iu=/4140/ostg.clktrk
>>>> >> _______________________________________________
>>>> >> Wrapper-user mailing list
>>>> >> Wra...@li...
>>>> >> https://lists.sourceforge.net/lists/listinfo/wrapper-user
>>>> >>
>>>> >
>>>> >
>>>> >
>>>> > --
>>>> > --
>>>> > Casey Jordan
>>>> > easyDITA a product of Jorsek LLC
>>>> > "CaseyDJordan" on LinkedIn, Twitter & Facebook
>>>> > (585) 348 7399
>>>> > easydita.com
>>>> >
>>>> >
>>>> > This message is intended only for the use of the Addressee(s) and may
>>>> > contain information that is privileged, confidential, and/or exempt
>>>> from
>>>> > disclosure under applicable law.  If you are not the intended
>>>> recipient,
>>>> > please be advised that any disclosure  copying, distribution, or use
>>>> of
>>>> > the information contained herein is prohibited.  If you have received
>>>> > this communication in error, please destroy all copies of the message,
>>>> > whether in electronic or hard copy format, as well as attachments, and
>>>> > immediately contact the sender by replying to this e-mail or by phone.
>>>> > Thank you.
>>>> >
>>>> >
>>>> ------------------------------------------------------------------------------
>>>> > Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
>>>> > from Actuate! Instantly Supercharge Your Business Reports and
>>>> Dashboards
>>>> > with Interactivity, Sharing, Native Excel Exports, App Integration &
>>>> more
>>>> > Get technology previously reserved for billion-dollar corporations,
>>>> FREE
>>>> >
>>>> http://pubads.g.doubleclick.net/gampad/clk?id=157005751&iu=/4140/ostg.clktrk
>>>> > _______________________________________________
>>>> > Wrapper-user mailing list
>>>> > Wra...@li...
>>>> > https://lists.sourceforge.net/lists/listinfo/wrapper-user
>>>> >
>>>>
>>>>
>>>>
>>>> --
>>>> Roberto Espinoza
>>>> Tanuki Software, Ltd.
>>>> 6-16-7-1001 Nishi-Kasai, Edogawa-ku
>>>> Tokyo 134-0088 Japan
>>>> Tel/Fax: +81-3-3878-3211
>>>> http://www.tanukisoftware.com
>>>> rob...@ta...
>>>>
>>>>
>>>> ------------------------------------------------------------------------------
>>>> Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
>>>> from Actuate! Instantly Supercharge Your Business Reports and Dashboards
>>>> with Interactivity, Sharing, Native Excel Exports, App Integration &
>>>> more
>>>> Get technology previously reserved for billion-dollar corporations, FREE
>>>>
>>>> http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk
>>>> _______________________________________________
>>>> Wrapper-user mailing list
>>>> Wra...@li...
>>>> https://lists.sourceforge.net/lists/listinfo/wrapper-user
>>>>
>>>
>>>
>>>
>>> --
>>> --
>>> Casey Jordan
>>> easyDITA a product of Jorsek LLC
>>> "CaseyDJordan" on LinkedIn, Twitter & Facebook
>>> (585) 348 7399
>>> easydita.com
>>>
>>>
>>> This message is intended only for the use of the Addressee(s) and may
>>> contain information that is privileged, confidential, and/or exempt from
>>> disclosure under applicable law.  If you are not the intended recipient,
>>> please be advised that any disclosure  copying, distribution, or use of
>>> the information contained herein is prohibited.  If you have received
>>> this communication in error, please destroy all copies of the message,
>>> whether in electronic or hard copy format, as well as attachments, and
>>> immediately contact the sender by replying to this e-mail or by phone.
>>> Thank you.
>>>
>>>
>>> ------------------------------------------------------------------------------
>>> Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
>>> from Actuate! Instantly Supercharge Your Business Reports and Dashboards
>>> with Interactivity, Sharing, Native Excel Exports, App Integration & more
>>> Get technology previously reserved for billion-dollar corporations, FREE
>>>
>>> http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk
>>> _______________________________________________
>>> Wrapper-user mailing list
>>> Wra...@li...
>>> https://lists.sourceforge.net/lists/listinfo/wrapper-user
>>>
>>>
>>
>>
>> ------------------------------------------------------------------------------
>> Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
>> from Actuate! Instantly Supercharge Your Business Reports and Dashboards
>> with Interactivity, Sharing, Native Excel Exports, App Integration & more
>> Get technology previously reserved for billion-dollar corporations, FREE
>>
>> http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk
>> _______________________________________________
>> Wrapper-user mailing list
>> Wra...@li...
>> https://lists.sourceforge.net/lists/listinfo/wrapper-user
>>
>>
>
>
> --
> --
> Casey Jordan
> easyDITA a product of Jorsek LLC
> "CaseyDJordan" on LinkedIn, Twitter & Facebook
> (585) 348 7399
> easydita.com
>
>
> This message is intended only for the use of the Addressee(s) and may
> contain information that is privileged, confidential, and/or exempt from
> disclosure under applicable law.  If you are not the intended recipient,
> please be advised that any disclosure  copying, distribution, or use of
> the information contained herein is prohibited.  If you have received
> this communication in error, please destroy all copies of the message,
> whether in electronic or hard copy format, as well as attachments, and
> immediately contact the sender by replying to this e-mail or by phone.
> Thank you.
>

-- 
--
Casey Jordan
easyDITA a product of Jorsek LLC
"CaseyDJordan" on LinkedIn, Twitter & Facebook
(585) 348 7399
easydita.com

This message is intended only for the use of the Addressee(s) and may
contain information that is privileged, confidential, and/or exempt from
disclosure under applicable law.  If you are not the intended recipient,
please be advised that any disclosure  copying, distribution, or use of
the information contained herein is prohibited.  If you have received
this communication in error, please destroy all copies of the message,
whether in electronic or hard copy format, as well as attachments, and
immediately contact the sender by replying to this e-mail or by phone.
Thank you.