Menu
▾
▴
Re: [Wrapper-user] "JVM appears hung": Wrapper restarts webserver every week
|
From: Andreas K. <and...@ya...> - 2009-05-06 07:51:34
|
Leif,
Sure it is a memory leak somewhere. The problem is that it is not present on my
local machine. I've been running loadtests with profiler and the memory didn't
increase at all.
Thanks for your hint with the heap. I didn't know I could do that. I'll try it.
/Andreas
--- Leif Mortenson <lei...@ta...> wrote:
> Andreas,
> This sounds like your application has a memory leak someplace. The
> Wrapper is failing because the JVM has run out of memory and is unable
> to continue.
>
> Have you tried playing around with creating a heap dump to see where
> your application is consuming so much memory? You can do so by adding
> the following to your wrapper.conf (set the additional number
> correctly):
>
> wrapper.java.additional.7=-Xrunhprof:heap=sites,depth=8
> wrapper.shutdown.timeout=600
> wrapper.jvm_exit.timeout=600
>
> The long shutdown timeouts are needed to make sure the JVM has time to
> write the dump file. The JVM appears frozen while it is being
> written.
>
> Let me know if you have any questions understanding the file. Be
> aware that it will be quite large.
>
> Cheers,
> Leif
>
>
> On Mon, May 4, 2009 at 11:38 PM, Andreas Karlsson
> <and...@ya...> wrote:
> >
> > Hi again,
> > So now the server has crashed again - running longer than before thanks to the
> > extended timout limit. The app eats memory and reports about low memory for days
> > before the actual crash. The home made garbage collector thread takes 50% of
> > processor-load also for days before the crash. I saw once in the log that the
> > GC-succeeded to reduce the memory used significantly and the wrapper complained
> > about it taking long time and increasing its limits.
> > Here is a snap of the log prior to the crash. Any hints on how to debug this is
> > highly appreciated;)
> > brgds
> > Andreas
> > ************************************************************
> > ************************************************************
> > 2009/05/01 10:29:32 | send a packet PING : ping
> > INFO | jvm 1 | 2009/05/01 10:29:32 | Received a packet PING : ping
> > INFO | jvm 1 | 2009/05/01 10:29:32 | Send a packet PING : ok
> > DEBUG | wrapperp | 2009/05/01 10:29:32 | read a packet PING : ok
> > DEBUG | wrapper | 2009/05/01 10:29:32 | Got ping response from JVM
> > INFO | jvm 1 | 2009/05/01 10:29:34 | 256520976 2009-05-01 10:29:34,894
> WARN
> > com.sse.supervisor.SSESupervisor ** Low of memory [0]mb free !
> SSEGarbageCollect
> > will occur.
> > INFO | jvm 1 | 2009/05/01 10:29:36 | 2395102070 2009-05-01 10:29:36,191
> WARN
> > com.sse.supervisor.SSESupervisor ***** Real low of memory [0]mb
> free!!! Will
> > garbagecollect all singletons and throwing out old members.
> > INFO | jvm 1 | 2009/05/01 10:29:37 | 727880698 2009-05-01 10:29:37,472
> ERROR
> > com.sse.supervisor.SSESupervisor *************************** OUT
> OF MEMORY *** WILL
> > DIE *****************************
> > DEBUG | wrapperp | 2009/05/01 10:29:38 | send a packet PING : ping
> > INFO | jvm 1 | 2009/05/01 10:29:38 | Received a packet PING : ping
> > INFO | jvm 1 | 2009/05/01 10:29:38 | Send a packet PING : ok
> > INFO | jvm 1 | 2009/05/01 10:29:38 | 176404686 2009-05-01 10:29:38,769
> WARN
> > com.sse.thread.SSEThreadManager 1604 long living threads alive (warn
> limit = 25)
> > DEBUG | wrapperp | 2009/05/01 10:29:38 | read a packet PING : ok
> > DEBUG | wrapper | 2009/05/01 10:29:38 | Got ping response from JVM
> > INFO | jvm 1 | 2009/05/01 10:29:43 | 4102609922 2009-05-01 10:29:42,644
> WARN
> > com.sse.supervisor.SSESupervisor ** Low of memory [0]mb free !
> SSEGarbageCollect
> > will occur.
> > DEBUG | wrapperp | 2009/05/01 10:29:44 | send a packet PING : ping
> > INFO | jvm 1 | 2009/05/01 10:29:45 | Received a packet PING : ping
> > INFO | jvm 1 | 2009/05/01 10:29:45 | Send a packet PING : ok
> > INFO | jvm 1 | 2009/05/01 10:29:45 | 3599369368 2009-05-01 10:29:45,207
> WARN
> > com.sse.supervisor.SSESupervisor ***** Real low of memory [0]mb
> free!!! Will
> > garbagecollect all singletons and throwing out old members.
> > DEBUG | wrapperp | 2009/05/01 10:29:45 | read a packet PING : ok
> > DEBUG | wrapper | 2009/05/01 10:29:45 | Got ping response from JVM
> > INFO | jvm 1 | 2009/05/01 10:29:49 | 1881161018 2009-05-01 10:29:47,801
> ERROR
> > com.sse.supervisor.SSESupervisor *************************** OUT
> OF MEMORY *** WILL
> > DIE *****************************
> > DEBUG | wrapperp | 2009/05/01 10:29:50 | send a packet PING : ping
> > INFO | jvm 1 | 2009/05/01 10:29:50 | Received a packet PING : ping
> > INFO | jvm 1 | 2009/05/01 10:29:50 | Send a packet PING : ok
> > DEBUG | wrapperp | 2009/05/01 10:29:50 | read a packet PING : ok
> > DEBUG | wrapper | 2009/05/01 10:29:50 | Got ping response from JVM
> > DEBUG | wrapperp | 2009/05/01 10:29:56 | send a packet PING : ping
> > INFO | jvm 1 | 2009/05/01 10:29:56 | Received a packet PING : ping
> > INFO | jvm 1 | 2009/05/01 10:29:56 | Send a packet PING : ok
> > DEBUG | wrapperp | 2009/05/01 10:29:56 | read a packet PING : ok
> > DEBUG | wrapper | 2009/05/01 10:29:56 | Got ping response from JVM
> > INFO | jvm 1 | 2009/05/01 10:29:59 | 1644323367 2009-05-01 10:29:49,097
> WARN
> > com.sse.thread.SSEThreadManager 1605 long living threads alive (warn
> limit = 25)
> > DEBUG | wrapperp | 2009/05/01 10:30:02 | send a packet PING : ping
> > INFO | jvm 1 | 2009/05/01 10:30:03 | Closing socket.
> > INFO | jvm 1 | 2009/05/01 10:30:03 | 1959232901 2009-05-01 10:29:54,254
> WARN
> > com.sse.thread.SSEATThread ATWork jammed: SSEGarbageWrapper:
> > [java.lang.ref.WeakReference@1662402]
> > INFO | jvm 1 | 2009/05/01 10:30:03 | Server daemon died!
> > INFO | jvm 1 | 2009/05/01 10:30:03 | java.lang.OutOfMemoryError
> > INFO | jvm 1 | 2009/05/01 10:30:03 | Open socket to wrapper...
> > DEBUG | wrapperp | 2009/05/01 10:30:03 | socket read no code (closed?).
> > DEBUG | wrapperp | 2009/05/01 10:30:03 | accepted a socket from 127.0.0.1 on
> port
> > 2779
> > INFO | jvm 1 | 2009/05/01 10:30:07 | Opened Socket
> > INFO | jvm 1 | 2009/05/01 10:30:07 | Send a packet KEY : iRUaWR2l6u5k5Rdr
> > DEBUG | wrapperp | 2009/05/01 10:30:07 | read a packet KEY : iRUaWR2l6u5k5Rdr
> > DEBUG | wrapper | 2009/05/01 10:30:07 | Got key from JVM: iRUaWR2l6u5k5Rdr
> > DEBUG | wrapperp | 2009/05/01 10:30:08 | send a packet PING : ping
> > INFO | jvm 1 | 2009/05/01 10:30:10 |
> > INFO | jvm 1 | 2009/05/01 10:30:11 | Unexpected Signal :
> > EXCEPTION_ACCESS_VIOLATION occurred at PC=0x6D3A5947
> > INFO | jvm 1 | 2009/05/01 10:30:11 | Function=[Unknown.]
> > INFO | jvm 1 | 2009/05/01 10:30:11 | Library=C:\Program
> > Files\Singleton\knahem\jre\bin\client\jvm.dll
> > INFO | jvm 1 | 2009/05/01 10:30:11 |
> > INFO | jvm 1 | 2009/05/01 10:30:11 | NOTE: We are unable to locate the
> function
> > name symbol for the error
> > INFO | jvm 1 | 2009/05/01 10:30:11 | just occurred. Please refer to
> > release documentation for possible
> > INFO | jvm 1 | 2009/05/01 10:30:11 | reason and solutions.
> > INFO | jvm 1 | 2009/05/01 10:30:11 |
> > INFO | jvm 1 | 2009/05/01 10:30:11 |
> > INFO | jvm 1 | 2009/05/01 10:30:11 | Current Java thread:
> > INFO | jvm 1 | 2009/05/01 10:30:11 | at
> > java.net.PlainSocketImpl.socketAccept(Native Method)
> > INFO | jvm 1 | 2009/05/01 10:30:11 | at
> > java.net.PlainSocketImpl.accept(PlainSocketImpl.java:353)
> > INFO | jvm 1 | 2009/05/01 10:30:11 | - locked <03C62228> (a
> > java.net.PlainSocketImpl)
> > INFO | jvm 1 | 2009/05/01 10:30:11 | at
> > java.net.ServerSocket.implAccept(ServerSocket.java:439)
> > INFO | jvm 1 | 2009/05/01 10:30:11 | at
> > java.net.ServerSocket.accept(ServerSocket.java:410)
> > INFO | jvm 1 | 2009/05/01 10:30:11 | at
> > org.mortbay.util.ThreadedServer.acceptSocket(ThreadedServer.java:346)
> > INFO | jvm 1 | 2009/05/01 10:30:11 | at
> > org.mortbay.util.ThreadedServer$Acceptor.run(ThreadedServer.java:523)
> > INFO | jvm 1 | 2009/05/01 10:30:11 |
> > INFO | jvm 1 | 2009/05/01 10:30:11 | Dynamic libraries:
> > INFO | jvm 1 | 2009/05/01 10:30:11 | 0x00400000 - 0x00406000
> C:\Program
> > Files\Singleton\knahem\jre\bin\java.exe
> > INFO | jvm 1 | 2009/05/01 10:30:11 | 0x7C800000 - 0x7C8C2000
> > C:\WINDOWS\system32\ntdll.dll
> > INFO | jvm 1 | 2009/05/01 10:30:11 | 0x77E40000 - 0x77F42000
> > C:\WINDOWS\system32\kernel32.dll
> > INFO | jvm 1 | 2009/05/01 10:30:11 | 0x7D1E0000 - 0x7D27C000
> > C:\WINDOWS\system32\ADVAPI32.dll
> > INFO | jvm 1 | 2009/05/01 10:30:11 | 0x77C50000 - 0x77CEF000
> > C:\WINDOWS\system32\RPCRT4.dll
> > INFO | jvm 1 | 2009/05/01 10:30:11 | 0x76F50000 - 0x76F63000
> > C:\WINDOWS\system32\Secur32.dll
> > INFO | jvm 1 | 2009/05/01 10:30:11 | 0x77BA0000 - 0x77BFA000
> > C:\WINDOWS\system32\MSVCRT.dll
> > INFO | jvm 1 | 2009/05/01 10:30:11 | 0x6D330000 - 0x6D45A000
> C:\Program
> > Files\Singleton\knahem\jre\bin\client\jvm.dll
> > INFO | jvm 1 | 2009/05/01 10:30:11 | 0x77380000 - 0x77411000
> > C:\WINDOWS\system32\USER32.dll
> > INFO | jvm 1 | 2009/05/01 10:30:11 | 0x77C00000 - 0x77C49000
> > C:\WINDOWS\system32\GDI32.dll
> > INFO | jvm 1 | 2009/05/01 10:30:11 | 0x76AA0000 - 0x76ACD000
> > C:\WINDOWS\system32\WINMM.dll
> > INFO | jvm 1 | 2009/05/01 10:30:11 | 0x6D1D0000 - 0x6D1D7000
> C:\Program
> > Files\Singleton\knahem\jre\bin\hpi.dll
> > INFO | jvm 1 | 2009/05/01 10:30:11 | 0x6D300000 - 0x6D30D000
> C:\Program
> > Files\Singleton\knahem\jre\bin\verify.dll
> > INFO | jvm 1 | 2009/05/01 10:30:11 | 0x6D210000 - 0x6D229000
> C:\Program
> > Files\Singleton\knahem\jre\bin\java.dll
> > INFO | jvm 1 | 2009/05/01 10:30:11 | 0x6D320000 - 0x6D32D000
> C:\Program
> > Files\Singleton\knahem\jre\bin\zip.dll
> > INFO | jvm 1 | 2009/05/01 10:30:11 | 0x16C10000 - 0x16C1E000
> C:\Program
> > Files\Singleton\knahem\jar\Wrapper.dll
> > INFO | jvm 1 | 2009/05/01 10:30:11 | 0x6D2D0000 - 0x6D2DE000
> C:\Program
> > Files\Singleton\knahem\jre\bin\net.dll
> > INFO | jvm 1 | 2009/05/01 10:30:11 | 0x71BB0000 - 0x71BB9000
> > C:\WINDOWS\system32\WSOCK32.dll
> > INFO | jvm 1 | 2009/05/01 10:30:11 | 0x71C00000 - 0x71C17000
> > C:\WINDOWS\system32\WS2_32.dll
> > INFO | jvm 1 | 2009/05/01 10:30:11 | 0x71BF0000 - 0x71BF8000
>
=== message truncated ===
|
