Re: [Wrapper-user] Please Help - New problem just started after moving Java app to 2008 R2 box

SourceForge Headquarters 1320 Columbia Street Suite 310 San Diego, CA 92101 +1 (858) 422-6466

I think, hope, to have found the problem with the "java.lang.OutOfMemoryError: unable to
create new native thread" issues we've been experiencing and seems to be more frequent as of late.

Question, is there a way to fix this?  Any help would be appreciated ASAP.

Just an FYI, when I added that additional command suggested below, the JVM went down every 2 hours.

Here is the what I found in the log after putting in the suggested command in the wrapper.conf file:

#
# An unexpected error has been detected by Java Runtime Environment:
#
# java.lang.OutOfMemoryError: requested 655360 bytes for GrET in C:\BUILD_AREA\jdk6_11\hotspot\src\share\vm\utilities\growableArray.cpp. Out of swap space?
#
#  Internal Error (allocation.inline.hpp:42), pid=4868, tid=4516
#  Error: GrET in C:\BUILD_AREA\jdk6_11\hotspot\src\share\vm\utilities\growableArray.cpp
#
# Java VM: Java HotSpot(TM) Client VM (11.0-b16 mixed mode windows-x86)
# If you would like to submit a bug report, please visit:
#   http://java.sun.com/webapps/bugreport/crash.jsp
#

---------------  T H R E A D  ---------------

Current thread (0x01207c00):  VMThread [stack: 0x492a0000,0x492f0000] [id=4516]

Stack: [0x492a0000,0x492f0000]
[error occurred during error reporting (printing stack bounds), id 0xc0000005]

VM_Operation (0x4b48f2b0): GenCollectForAllocation, mode: safepoint, requested by thread 0x536c6000

---------------  P R O C E S S  ---------------

Java Threads: ( => current thread )

THERE WERE HUNDREDS OF APPLICATION THREADS LISTED IN THIS SECTION SO I DID NOT LIST THEM

Other Threads:
=>0x01207c00 VMThread [stack: 0x492a0000,0x492f0000] [id=4516]
  0x4964f800 WatcherThread [stack: 0x49b20000,0x49b70000] [id=8228]

VM state:at safepoint (normal execution)

VM Mutex/Monitor currently owned by a thread:  ([mutex/lock_event])
[0x01189c40] UNKNOWN - owner thread: 0x01207c00
[0x0118a0a0] UNKNOWN - owner thread: 0x536c6000

Heap
def new generation   total 13376K, used 936K [0x03290000, 0x04110000, 0x08150000)
  eden space 11904K,   0% used [0x03290000, 0x03290000, 0x03e30000)
  from space 1472K,  63% used [0x03e30000, 0x03f1a2d8, 0x03fa0000)
  to   space 1472K,   0% used [0x03fa0000, 0x03fa0000, 0x04110000)
tenured generation   total 111448K, used 111414K [0x08150000, 0x0ee26000, 0x43290000)
   the space 111448K,  99% used [0x08150000, 0x0ee1d818, 0x0ee1da00, 0x0ee26000)
compacting perm gen  total 12800K, used 12711K [0x43290000, 0x43f10000, 0x47290000)
   the space 12800K,  99% used [0x43290000, 0x43ef9c50, 0x43ef9e00, 0x43f10000)
No shared spaces configured.

Dynamic libraries:
0x00400000 - 0x00424000              C:\Program Files (x86)\Java\jre1.6.0_11\bin\javaw.exe
0x777b0000 - 0x77930000             C:\Windows\SysWOW64\ntdll.dll
0x75100000 - 0x75210000              C:\Windows\syswow64\kernel32.dll
0x76320000 - 0x76367000              C:\Windows\syswow64\KERNELBASE.dll
0x76a10000 - 0x76ab0000             C:\Windows\syswow64\ADVAPI32.dll
0x75610000 - 0x756bc000              C:\Windows\syswow64\msvcrt.dll
0x76800000 - 0x76819000              C:\Windows\SysWOW64\sechost.dll
0x75400000 - 0x754f0000              C:\Windows\syswow64\RPCRT4.dll
0x74e90000 - 0x74ef0000              C:\Windows\syswow64\SspiCli.dll
0x74e80000 - 0x74e8c000             C:\Windows\syswow64\CRYPTBASE.dll
0x76370000 - 0x76470000              C:\Windows\syswow64\USER32.dll
0x76760000 - 0x767f0000              C:\Windows\syswow64\GDI32.dll
0x75240000 - 0x7524a000              C:\Windows\syswow64\LPK.dll
0x76470000 - 0x7650e000             C:\Windows\syswow64\USP10.dll
0x76700000 - 0x76760000              C:\Windows\system32\IMM32.DLL
0x76940000 - 0x76a0c000              C:\Windows\syswow64\MSCTF.dll
0x7c340000 - 0x7c396000              C:\Program Files (x86)\Java\jre1.6.0_11\bin\msvcr71.dll
0x6d800000 - 0x6da56000             C:\Program Files (x86)\Java\jre1.6.0_11\bin\client\jvm.dll
0x74c50000 - 0x74c82000              C:\Windows\system32\WINMM.dll
0x74c90000 - 0x74cdb000              C:\Windows\system32\apphelp.dll
0x6d280000 - 0x6d288000             C:\Program Files (x86)\Java\jre1.6.0_11\bin\hpi.dll
0x76310000 - 0x76315000              C:\Windows\syswow64\PSAPI.DLL
0x6d7b0000 - 0x6d7bc000             C:\Program Files (x86)\Java\jre1.6.0_11\bin\verify.dll
0x6d320000 - 0x6d33f000              C:\Program Files (x86)\Java\jre1.6.0_11\bin\java.dll
0x6d7f0000 - 0x6d7ff000               C:\Program Files (x86)\Java\jre1.6.0_11\bin\zip.dll
0x6d610000 - 0x6d623000             C:\Program Files (x86)\Java\jre1.6.0_11\bin\net.dll
0x76c70000 - 0x76ca5000              C:\Windows\syswow64\WS2_32.dll
0x767f0000 - 0x767f6000               C:\Windows\syswow64\NSI.dll
0x74d60000 - 0x74d9c000             C:\Windows\system32\mswsock.dll
0x749e0000 - 0x749e6000             C:\Windows\System32\wship6.dll
0x74ce0000 - 0x74ce5000              C:\Windows\System32\wshtcpip.dll
0x74d10000 - 0x74d54000             C:\Windows\system32\DNSAPI.dll
0x749c0000 - 0x749dc000              C:\Windows\system32\IPHLPAPI.DLL
0x749b0000 - 0x749b7000             C:\Windows\system32\WINNSI.DLL
0x749a0000 - 0x749a6000              C:\Windows\system32\rasadhlp.dll
0x74960000 - 0x74998000              C:\Windows\System32\fwpuclnt.dll
0x6d560000 - 0x6d569000             C:\Program Files (x86)\Java\jre1.6.0_11\bin\management.dll
0x748a0000 - 0x748b6000             C:\Windows\system32\CRYPTSP.dll
0x74860000 - 0x7489b000             C:\Windows\system32\rsaenh.dll
0x74c30000 - 0x74c47000              C:\Windows\system32\USERENV.dll
0x74c20000 - 0x74c2b000              C:\Windows\system32\profapi.dll
0x74da0000 - 0x74db0000             C:\Windows\system32\NLAapi.dll
0x74d00000 - 0x74d08000             C:\Windows\System32\winrnr.dll
0x74cf0000 - 0x74d00000              C:\Windows\system32\napinsp.dll
0x49b70000 - 0x49ba8000             C:\ATMoPOStR2009\lib\wrapper-windows-x86-32.dll
0x6d630000 - 0x6d639000             C:\Program Files (x86)\Java\jre1.6.0_11\bin\nio.dll
0x73d50000 - 0x73d63000             C:\ATMoPOStR2009\lib\sqljdbc_auth.dll
0x6d790000 - 0x6d798000             C:\Program Files (x86)\Java\jre1.6.0_11\bin\sunmscapi.dll
0x754f0000 - 0x7560e000              C:\Windows\syswow64\CRYPT32.dll
0x77780000 - 0x7778c000              C:\Windows\syswow64\MSASN1.dll
0x74950000 - 0x74958000              C:\Windows\system32\secur32.dll
0x745d0000 - 0x745d8000             C:\Windows\system32\credssp.dll
0x74540000 - 0x745c8000              C:\Windows\SysWOW64\Kerberos.DLL
0x74520000 - 0x74531000              C:\Windows\system32\cryptdll.dll
0x744b0000 - 0x744f2000              C:\Windows\SysWOW64\msv1_0.DLL
0x74490000 - 0x744a8000              C:\Windows\system32\ntdsapi.dll

VM Arguments:
jvm_args: -Dcom.sun.management.jmxremote -Dorg.tanukisoftware.wrapper.WrapperManager.mbean=true -Xss128K -Xmx1024m -Djava.library.path=lib -Dwrapper.key=WyO_dGR8ZdnzaTyA -Dwrapper.port=32001 -Dwrapper.jvm.port.min=31000 -Dwrapper.jvm.port.max=31999 -Dwrapper.debug=TRUE -Dwrapper.pid=1508 -Dwrapper.version=3.3.5 -Dwrapper.native_library=wrapper -Dwrapper.service=TRUE -Dwrapper.cpu.timeout=10 -Dwrapper.jvmid=1
java_command: com.sharpbancsystems.ATM.ATMoPOSt
Launcher Type: SUN_STANDARD

Environment Variables:
JAVA_HOME=C:\Program Files (x86)\Java\jre1.6.0_11
PATH=C:\Program Files (x86)\Java\jre1.6.0_31\bin;C:\Program Files (x86)\Perl\site\bin;C:\Program Files (x86)\Perl\bin;C:\Perl\site\bin;C:\Perl\bin;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Program Files (x86)\GnuWin32\bin;C:\Applications
USERNAME=SYSTEM
OS=Windows_NT
PROCESSOR_IDENTIFIER=Intel64 Family 6 Model 44 Stepping 2, GenuineIntel

---------------  S Y S T E M  ---------------

OS: Windows NT 6.1 Build 7600

CPU:total 24 (16 cores per cpu, 2 threads per core) family 6 model 12 stepping 2, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3, ht

Memory: 4k page, physical 4194303k(4194303k free), swap 4194303k(4194303k free)

vm_info: Java HotSpot(TM) Client VM (11.0-b16) for windows-x86 JRE (1.6.0_11-b03), built on Nov 10 2008 02:15:12 by "java_re" with MS VC++ 7.1

time: Thu Nov 21 18:02:51 2013
elapsed time: 6482 seconds

----------------------------------------------------------------------------------------------------------------------------
Re: [Wrapper-user] Please Help - New problem just started after moving Java app to 2008 R2 box
From: Leif Mortenson <leif.mortenson@ta...<mailto:leif.mortenson@ta...>> - 2013-11-19 11:48:29
Attachments: Message as HTML
Melinda,
Another user sent the following:
---
I think there may be a different cause to the one you have suggested.

I'm not sure about 64bit Java using twice as much memory - that's never
been the case for me. However the stack sizes are substantially bigger on
64bit Java.

The user reports the exception "java.lang.OutOfMemoryError: unable to
create new native thread".

It could be they are out of heap but as it fails on creating a new thread
it's much more likely to be out of thread stack space.

Worth checking if the app is highly threaded - and if so reduce the thread
stack size with Java's -Xss switch.

For example, try this
wrapper.java.additional.3=-Xss128K
If this is the case, a more longer term solution would be to rewrite the
application to use ThreadPools as a huge number of threads is never going
to be efficient.
---

Could you please give this a try and let me know how it works for you?
Also, approximately how many threads does your application have?  Try
invoking a thread dump if you are unsure.

Cheers,
Leif

Melinda Savoy
Sharp BancSystems, Inc.
Senior Java Developer

* E:  me...@sh...<mailto:me...@sh...>
* O: 817.438.2161

"We understand that e-mail is a simple and cost effective method of communication, however items such as requests and inquiries needing immediate attention should not be sent via e-mail. Also, sensitive personal and account information like account numbers, Social Security numbers, passwords, or PIN numbers should not be sent by e-mail. Because an e-mail message may not be immediately reviewed by a company representative, you should contact the company by telephone, or in person for items needing immediate attention. Furthermore, this e-mail and any attached files may contain confidential and/or privileged material for the sole use of the intended recipient. Any review, use, distribution or disclosure by others is strictly prohibited. If you are not the intended recipient (or authorized to receive this e-mail for the recipient), please contact the sender by reply e-mail and delete all copies of this email"