Re: [Wrapper-user] Advice on stopping wrapper from triggering JVM restarts

[Tim L. <tim...@gm...>]

Yes you can monitor this with top, but the speed depends on how much your
application logs.

On Wed, Dec 10, 2014 at 5:46 PM, Casey Jordan <cas...@jo...>
wrote:

> Cool thanks, will I be able to see this memory usage increase by looking
> at the top command?
>
> I would like to keep track of this a little more and see if I notice the
> wrapper memory growing.
>
> Thanks
>
> On Wed, Dec 10, 2014 at 11:06 AM, Tim Lammens <tim...@gm...>
> wrote:
>
>> That is indeed the right thread.
>>
>> Regards,
>> Tim
>>
>> On Wed, Dec 10, 2014 at 5:03 PM, Casey Jordan <cas...@jo...>
>> wrote:
>>
>>> Hi Tim,
>>>
>>> Thanks for the info, is this the thread you speak of:
>>> http://sourceforge.net/p/wrapper/mailman/message/33048371/
>>>
>>> If not could you point me to the right one?
>>>
>>> Thanks
>>>
>>> On Wed, Dec 10, 2014 at 10:58 AM, Tim Lammens <tim...@gm...>
>>> wrote:
>>>
>>>> You are most probably experiencing a glibc bug as I mentioned before in
>>>> another thread. The bug was fixed in recent glibc versions.
>>>> Try updating your glibc version for CentOS. If that is not possible you
>>>> can apply the patches I provided in a different thread on this mailing list.
>>>>
>>>> Regards,
>>>> Tim
>>>>
>>>> On Wed, Dec 10, 2014 at 4:49 PM, Casey Jordan <cas...@jo...>
>>>> wrote:
>>>>
>>>>> Ok so this happened again and I was able to capture some additional
>>>>> log information (not from wrapper.debug though as I can't update this
>>>>> server just yet)
>>>>>
>>>>> I got this in our logs:
>>>>>
>>>>> INFO   | jvm 1    | 2014/12/09 23:15:50 | <br></br>Max Mem: 2223 MB
>>>>> INFO   | jvm 1    | 2014/12/09 23:15:50 | <br></br>Total Mem: 2204 MB
>>>>> INFO   | jvm 1    | 2014/12/09 23:15:50 | <br></br>Used Mem: 1141 MB
>>>>> INFO   | jvm 1    | 2014/12/09 23:15:50 | <br></br>Raw free-mem: 1082 MB
>>>>> INFO   | jvm 1    | 2014/12/09 23:15:50 | <br></br>Total descriptors: 4096
>>>>> INFO   | jvm 1    | 2014/12/09 23:15:50 | <br></br>Open descriptors: 556
>>>>> INFO   | jvm 1    | 2014/12/09 23:15:50 | <br></br>Available descriptors: 3540
>>>>> INFO   | jvm 1    | 2014/12/09 23:15:50 | <br></br>Free disk space: 18 GB
>>>>> INFO   | jvm 1    | 2014/12/09 23:15:50 | <br></br>
>>>>> INFO   | wrapper  | 2014/12/09 23:17:09 | Wrapper Process has not received any CPU time for 1 seconds.  Extending timeouts.
>>>>> STATUS | wrapper  | 2014/12/09 23:17:09 | Pinging the JVM took 7 seconds to respond.
>>>>> INFO   | wrapper  | 2014/12/09 23:17:09 | Wrapper Process has not received any CPU time for 5 seconds.  Extending timeouts.
>>>>> STATUS | wrapper  | 2014/12/09 23:17:09 | JVM received a signal SIGKILL (9).
>>>>> STATUS | wrapper  | 2014/12/09 23:17:09 | JVM process is gone.
>>>>> ERROR  | wrapper  | 2014/12/09 23:17:09 | JVM exited unexpectedly.
>>>>> STATUS | wrapper  | 2014/12/09 23:17:09 | Automatic JVM Restarts disabled.  Shutting down.
>>>>> STATUS | wrapper  | 2014/12/09 23:17:09 | <-- Wrapper Stopped
>>>>>
>>>>>
>>>>> and I found this in the /var/log/messages
>>>>>
>>>>> Dec  9 23:17:09 jorsek-home2 kernel: Out of memory: Kill process 10262
>>>>> (java) score 591 or sacrifice child
>>>>> Dec  9 23:17:09 jorsek-home2 kernel: Killed process 10262 (java)
>>>>> total-vm:6478728kB, anon-rss:2600344kB, file-rss:0kB
>>>>>
>>>>> So this does appear to be the kernel killing the process, but
>>>>> according to java I have plenty of memory available (>1GB).
>>>>>
>>>>> I know this is not really a wrapper issue any longer, but I would be
>>>>> very appreciative if anyone can give me some advice or point me to some
>>>>> resources that might help figure this out.
>>>>>
>>>>> For reference this is a Centos7 box with 4GB of RAM and a 1GB swap
>>>>> space, and 40GB SSD drive.
>>>>>
>>>>> Tonight I am going to add NewRelic monitoring to this server to see if
>>>>> I can get a better picture of what is going on.
>>>>>
>>>>> Thanks!
>>>>>
>>>>> On Tue, Dec 2, 2014 at 7:30 PM, Roberto Espinoza <
>>>>> rob...@ta...> wrote:
>>>>>
>>>>>> Casey,
>>>>>>
>>>>>> Have you checked the kernel logs to see if it wasn't killed because
>>>>>> your system ran out of memory?
>>>>>>
>>>>>> Certainly there is no way to know this inside the JVM because the
>>>>>> kernel out of memory killer will just score the current processes and
>>>>>> issue a kill.
>>>>>>
>>>>>> On Wed, Dec 3, 2014 at 12:39 AM, Casey Jordan <
>>>>>> cas...@jo...> wrote:
>>>>>> > Hi Leif,
>>>>>> >
>>>>>> > Thanks for the feedback. This is quite a mystery then because I am
>>>>>> 100% sure
>>>>>> > that the process wasn't killed manually or by any other systems we
>>>>>> have
>>>>>> > running. (This is just a standard CentOS 7 setup)
>>>>>> >
>>>>>> > At about the same time that this log appeared I was running a very
>>>>>> heavy
>>>>>> > build process in a separate JVM instance.
>>>>>> >
>>>>>> > It seems more than just coincidental that this happened at the same
>>>>>> time. I
>>>>>> > assume that if the JVM crashed it wouldn't have received a SIG 9?
>>>>>> Can you
>>>>>> > think of any scenario where this makes sense? Perhaps I am missing
>>>>>> some
>>>>>> > knowledge of java architecture here that is making this mysterious.
>>>>>> >
>>>>>> > Thanks
>>>>>> >
>>>>>> >
>>>>>> > On Tue, Dec 2, 2014 at 4:27 AM, Leif Mortenson
>>>>>> > <lei...@ta...> wrote:
>>>>>> >>
>>>>>> >> Casey,
>>>>>> >> The configuration you specified will disable all pings and cpu
>>>>>> warnings
>>>>>> >> and should do what you want.
>>>>>> >> The Wrapper should never timeout  in this case.  There are other
>>>>>> things
>>>>>> >> such as a JVM crash would would still result in a restart after
>>>>>> the fact.
>>>>>> >>
>>>>>> >> The log that you send shows that the JVM will killed with a kill
>>>>>> -9.  Not
>>>>>> >> sure if that was a test on your part, but if the source of the
>>>>>> signal has
>>>>>> >> permission to kill the JVM's process then there is nothing we can
>>>>>> do to
>>>>>> >> prevent that.
>>>>>> >>
>>>>>> >> Cheers,
>>>>>> >> Leif
>>>>>> >>
>>>>>> >> On Tue, Dec 2, 2014 at 12:16 PM, Casey Jordan <
>>>>>> cas...@jo...>
>>>>>> >> wrote:
>>>>>> >>>
>>>>>> >>> Hi all,
>>>>>> >>>
>>>>>> >>> I am sure this is a common question, but I am having trouble
>>>>>> extracting
>>>>>> >>> details I need from the documentation. From time to time we get a
>>>>>> situation
>>>>>> >>> where our JVM gets killed and we see something like this in the
>>>>>> logs:
>>>>>> >>>
>>>>>> >>> INFO   | wrapper  | 2014/12/02 02:23:18 | Wrapper Process has not
>>>>>> >>> received any CPU time for 1 seconds.  Extending timeouts.
>>>>>> >>> STATUS | wrapper  | 2014/12/02 02:23:37 | JVM received a signal
>>>>>> SIGKILL
>>>>>> >>> (9).
>>>>>> >>> STATUS | wrapper  | 2014/12/02 02:23:37 | JVM process is gone.
>>>>>> >>> ERROR  | wrapper  | 2014/12/02 02:23:37 | JVM exited unexpectedly.
>>>>>> >>> STATUS | wrapper  | 2014/12/02 02:23:38 | Automatic JVM Restarts
>>>>>> >>> disabled.  Shutting down.
>>>>>> >>> STATUS | wrapper  | 2014/12/02 02:23:38 | <-- Wrapper Stopped
>>>>>> >>>
>>>>>> >>>
>>>>>> >>> This always happens in the case where some other process is
>>>>>> eating up all
>>>>>> >>> the cpu for more than a few seconds.
>>>>>> >>>
>>>>>> >>> It's crucial for us that the wrapper never, ever restarts the JVM
>>>>>> as this
>>>>>> >>> is something we always want to look into manually.
>>>>>> >>>
>>>>>> >>> So my question is, what is the best way to make sure this doesn't
>>>>>> happen?
>>>>>> >>>
>>>>>> >>> I am using community version 3.5.17, and have the following
>>>>>> values in our
>>>>>> >>> wrapper.conf:
>>>>>> >>>
>>>>>> >>>
>>>>>> #********************************************************************
>>>>>> >>>    # Timeouts
>>>>>> >>>
>>>>>> #********************************************************************
>>>>>> >>>    wrapper.ping.timeout=0
>>>>>> >>>    wrapper.ping.timeout.action=DEBUG
>>>>>> >>>    wrapper.cpu.timeout=0
>>>>>> >>>    wrapper.startup.timeout=300
>>>>>> >>>
>>>>>> >>> Any help is much appreciated.
>>>>>> >>>
>>>>>> >>> Thanks!
>>>>>> >>>
>>>>>> >>
>>>>>> >>
>>>>>> >>
>>>>>> ------------------------------------------------------------------------------
>>>>>> >> Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
>>>>>> >> from Actuate! Instantly Supercharge Your Business Reports and
>>>>>> Dashboards
>>>>>> >> with Interactivity, Sharing, Native Excel Exports, App Integration
>>>>>> & more
>>>>>> >> Get technology previously reserved for billion-dollar
>>>>>> corporations, FREE
>>>>>> >>
>>>>>> >>
>>>>>> http://pubads.g.doubleclick.net/gampad/clk?id=157005751&iu=/4140/ostg.clktrk
>>>>>> >> _______________________________________________
>>>>>> >> Wrapper-user mailing list
>>>>>> >> Wra...@li...
>>>>>> >> https://lists.sourceforge.net/lists/listinfo/wrapper-user
>>>>>> >>
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> > --
>>>>>> > --
>>>>>> > Casey Jordan
>>>>>> > easyDITA a product of Jorsek LLC
>>>>>> > "CaseyDJordan" on LinkedIn, Twitter & Facebook
>>>>>> > (585) 348 7399
>>>>>> > easydita.com
>>>>>> >
>>>>>> >
>>>>>> > This message is intended only for the use of the Addressee(s) and
>>>>>> may
>>>>>> > contain information that is privileged, confidential, and/or exempt
>>>>>> from
>>>>>> > disclosure under applicable law.  If you are not the intended
>>>>>> recipient,
>>>>>> > please be advised that any disclosure  copying, distribution, or
>>>>>> use of
>>>>>> > the information contained herein is prohibited.  If you have
>>>>>> received
>>>>>> > this communication in error, please destroy all copies of the
>>>>>> message,
>>>>>> > whether in electronic or hard copy format, as well as attachments,
>>>>>> and
>>>>>> > immediately contact the sender by replying to this e-mail or by
>>>>>> phone.
>>>>>> > Thank you.
>>>>>> >
>>>>>> >
>>>>>> ------------------------------------------------------------------------------
>>>>>> > Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
>>>>>> > from Actuate! Instantly Supercharge Your Business Reports and
>>>>>> Dashboards
>>>>>> > with Interactivity, Sharing, Native Excel Exports, App Integration
>>>>>> & more
>>>>>> > Get technology previously reserved for billion-dollar corporations,
>>>>>> FREE
>>>>>> >
>>>>>> http://pubads.g.doubleclick.net/gampad/clk?id=157005751&iu=/4140/ostg.clktrk
>>>>>> > _______________________________________________
>>>>>> > Wrapper-user mailing list
>>>>>> > Wra...@li...
>>>>>> > https://lists.sourceforge.net/lists/listinfo/wrapper-user
>>>>>> >
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Roberto Espinoza
>>>>>> Tanuki Software, Ltd.
>>>>>> 6-16-7-1001 Nishi-Kasai, Edogawa-ku
>>>>>> Tokyo 134-0088 Japan
>>>>>> Tel/Fax: +81-3-3878-3211
>>>>>> http://www.tanukisoftware.com
>>>>>> rob...@ta...
>>>>>>
>>>>>>
>>>>>> ------------------------------------------------------------------------------
>>>>>> Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
>>>>>> from Actuate! Instantly Supercharge Your Business Reports and
>>>>>> Dashboards
>>>>>> with Interactivity, Sharing, Native Excel Exports, App Integration &
>>>>>> more
>>>>>> Get technology previously reserved for billion-dollar corporations,
>>>>>> FREE
>>>>>>
>>>>>> http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk
>>>>>> _______________________________________________
>>>>>> Wrapper-user mailing list
>>>>>> Wra...@li...
>>>>>> https://lists.sourceforge.net/lists/listinfo/wrapper-user
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> --
>>>>> Casey Jordan
>>>>> easyDITA a product of Jorsek LLC
>>>>> "CaseyDJordan" on LinkedIn, Twitter & Facebook
>>>>> (585) 348 7399
>>>>> easydita.com
>>>>>
>>>>>
>>>>> This message is intended only for the use of the Addressee(s) and may
>>>>> contain information that is privileged, confidential, and/or exempt
>>>>> from
>>>>> disclosure under applicable law.  If you are not the intended
>>>>> recipient,
>>>>> please be advised that any disclosure  copying, distribution, or use of
>>>>> the information contained herein is prohibited.  If you have received
>>>>> this communication in error, please destroy all copies of the message,
>>>>> whether in electronic or hard copy format, as well as attachments, and
>>>>> immediately contact the sender by replying to this e-mail or by phone.
>>>>> Thank you.
>>>>>
>>>>>
>>>>> ------------------------------------------------------------------------------
>>>>> Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
>>>>> from Actuate! Instantly Supercharge Your Business Reports and
>>>>> Dashboards
>>>>> with Interactivity, Sharing, Native Excel Exports, App Integration &
>>>>> more
>>>>> Get technology previously reserved for billion-dollar corporations,
>>>>> FREE
>>>>>
>>>>> http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk
>>>>> _______________________________________________
>>>>> Wrapper-user mailing list
>>>>> Wra...@li...
>>>>> https://lists.sourceforge.net/lists/listinfo/wrapper-user
>>>>>
>>>>>
>>>>
>>>>
>>>> ------------------------------------------------------------------------------
>>>> Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
>>>> from Actuate! Instantly Supercharge Your Business Reports and Dashboards
>>>> with Interactivity, Sharing, Native Excel Exports, App Integration &
>>>> more
>>>> Get technology previously reserved for billion-dollar corporations, FREE
>>>>
>>>> http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk
>>>> _______________________________________________
>>>> Wrapper-user mailing list
>>>> Wra...@li...
>>>> https://lists.sourceforge.net/lists/listinfo/wrapper-user
>>>>
>>>>
>>>
>>>
>>> --
>>> --
>>> Casey Jordan
>>> easyDITA a product of Jorsek LLC
>>> "CaseyDJordan" on LinkedIn, Twitter & Facebook
>>> (585) 348 7399
>>> easydita.com
>>>
>>>
>>> This message is intended only for the use of the Addressee(s) and may
>>> contain information that is privileged, confidential, and/or exempt from
>>> disclosure under applicable law.  If you are not the intended recipient,
>>> please be advised that any disclosure  copying, distribution, or use of
>>> the information contained herein is prohibited.  If you have received
>>> this communication in error, please destroy all copies of the message,
>>> whether in electronic or hard copy format, as well as attachments, and
>>> immediately contact the sender by replying to this e-mail or by phone.
>>> Thank you.
>>>
>>>
>>> ------------------------------------------------------------------------------
>>> Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
>>> from Actuate! Instantly Supercharge Your Business Reports and Dashboards
>>> with Interactivity, Sharing, Native Excel Exports, App Integration & more
>>> Get technology previously reserved for billion-dollar corporations, FREE
>>>
>>> http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk
>>> _______________________________________________
>>> Wrapper-user mailing list
>>> Wra...@li...
>>> https://lists.sourceforge.net/lists/listinfo/wrapper-user
>>>
>>>
>>
>>
>> ------------------------------------------------------------------------------
>> Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
>> from Actuate! Instantly Supercharge Your Business Reports and Dashboards
>> with Interactivity, Sharing, Native Excel Exports, App Integration & more
>> Get technology previously reserved for billion-dollar corporations, FREE
>>
>> http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk
>> _______________________________________________
>> Wrapper-user mailing list
>> Wra...@li...
>> https://lists.sourceforge.net/lists/listinfo/wrapper-user
>>
>>
>
>
> --
> --
> Casey Jordan
> easyDITA a product of Jorsek LLC
> "CaseyDJordan" on LinkedIn, Twitter & Facebook
> (585) 348 7399
> easydita.com
>
>
> This message is intended only for the use of the Addressee(s) and may
> contain information that is privileged, confidential, and/or exempt from
> disclosure under applicable law.  If you are not the intended recipient,
> please be advised that any disclosure  copying, distribution, or use of
> the information contained herein is prohibited.  If you have received
> this communication in error, please destroy all copies of the message,
> whether in electronic or hard copy format, as well as attachments, and
> immediately contact the sender by replying to this e-mail or by phone.
> Thank you.
>
>
> ------------------------------------------------------------------------------
> Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
> from Actuate! Instantly Supercharge Your Business Reports and Dashboards
> with Interactivity, Sharing, Native Excel Exports, App Integration & more
> Get technology previously reserved for billion-dollar corporations, FREE
>
> http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk
> _______________________________________________
> Wrapper-user mailing list
> Wra...@li...
> https://lists.sourceforge.net/lists/listinfo/wrapper-user
>
>