Hello All
I am running Oracle JRockit(R) R28.2.3-13-149708-1.6.0_31-20120327-1523-windows-ia32, recently we started to see JVM crashes on our Regression environment. Everytime, When I look at the dump file, there are no OutOfMemoryErrors and no StackOverFlowErrors.
But, whenever it crashes, it is always on GC Worker Thread (thread number do vary). Spent a lot of time investigating, no solutions yet. Hence approaching this expert forum, any help would be much appreciated, Please share your opinion.
Thanks
Kannan
Here is the dump file text:
===== BEGIN DUMP =============================================================
JRockit dump produced after 0 days, 03:04:31 on Fri Oct 25 20:01:22 2013
**************************************************************************
* If you see this dump, please go to *
* http://download.oracle.com/docs/cd/E15289_01/go2troubleshooting.html *
* for troubleshooting information. *
**************************************************************************
Additional information is available in:
D:\Apps\EchoCluster\bin\jrockit.5436.dump
D:\Apps\EchoCluster\bin\jrockit.5436.mdmp
Error Message: Illegal memory access. [54]
Exception Rec: EXCEPTION_ACCESS_VIOLATION (c0000005) at 0x00D24BDF - memory at 0x00000048 could not be read.
Minidump : Wrote mdmp. Size is 2362MB
SafeDllMode : -1
Version : Oracle JRockit(R) R28.2.3-13-149708-1.6.0_31-20120327-1523-windows-ia32
CPU : Intel SSE SSE2 SSE3 SSSE3 SSE4.1 SSE4.2 Intel64
Number CPUs : 8
Tot Phys Mem : 34359271424 (32767 MB)
OS version : Microsoft Windows 2008 R2 version 6.1 Service Pack 1 (Build 7601) (64-bit)
Hypervisor : Unknown (but probably running on a hypervisor)
Thread System: Windows Threads
Java locking : Lazy unlocking enabled (class banning) (transfer banning)
State : JVM is running (Main thread has finished)
Command Line : <modified: I have removed the lengthy command line parameters>
Repository : C:\Users\SVCUKL~1\AppData\Local\Temp\2013_10_25_16_56_55_5436
java.home : C:\Program Files (x86)\Java\JRockit\jre
j.class.path : ..\\lib\wrappertest.jar;..\\lib\wrapper.jar;..\\lib\echo.jar;..\\lib\coherence.jar;..\\lib\coherence-jpa.jar;..\\lib\eclipselink.jar;..\\lib\eclipselink-jpa-modelgen_2.4.2.v20130514-5956486.jar;..\\lib\javax.persistence_1.0.0.jar;..\\lib\javax.persistence_2.0.5.v201212031355.jar;..\\lib\ojdbc6.jar;..\\lib\commons-lang3-3.1.jar;..\\lib\ojdbc5.jar;..\\config\cluster\pof;..\;..\\lib\log4j;..\\lib\log4j-1.2.17.jar;..\\lib\apache-log4j-extras-1.1.jar;..\\lib\monitoring-framework-1.10.0.jar;..\\lib\slf4j-api-1.7.2.jar;..\\lib\slf4j-simple-1.7.2.jar;..\\lib\tibrvj.jar;..\\lib\tibrvnative.jar;..\\lib\tibrv\;..\\config\cluster\server\log4j;..\\lib\coherence-rest.jar;..\\lib\jackson-all-1.8.1.jar;..\\lib\jersey-core-1.8.jar;..\\lib\jersey-json-1.8.jar;..\\lib\jersey-server-1.8.jar;..\\config\cluster\pof\coherence-rest-config.xml;..\\lib\coherence-loadbalancer.jar;..\\lib\commonj.jar;..\\lib\jmscommon.jar;..\\lib\jndi.jar;..\\lib\jta.jar;..\\lib\junit-4.10.jar;..\\lib\wrapperdemo.jar;..\\lib\vtd-xml.jar
j.lib.path : lib;..\\lib\tibrv\;..\\lib\
JAVA_HOME : C:\Program Files (x86)\Java\JRockit
_JAVA_OPTIONS: <not set>
PATH : C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Program Files (x86)\Bank\ODCReal;C:\Program Files\Python26\;C:\Program Files\Reuters\Rendez-vous\bin;C:\Windows\system32\config\systemprofile\AppData\Roaming\Python\Scripts;C:\Program Files (x86)\Bank\ODCReal\;C:\Program Files\SysinternalsSuite;C:\Program Files (x86)\Windows Resource Kits\Tools;C:\Windows\sysint;C:\ORACLE11G32\product\11.2.0\client_1\bin;C:\JAVA_637\BIN;C:\JAVA_637\jre\bin\client;C:\Program Files (x86)\Python26;
StackOverFlow: 0 StackOverFlowErrors have occured
OutOfMemory : 0 OutOfMemoryErrors have occured
C Heap : Good; no memory allocations have failed
GC Strategy : Mode: pausetime, with strategy: genconcon (basic strategy: genconcon)
GC Status : OC is not running. Last finished OC was OC#1.
: YC is currently running. This is YC#111.
YC Promotion : This YC has been able to promote all found objects so far
YC History : Ran 52 YCs before OC#1.
: Started 59 YCs since last OC.
Heap Holes : 1 hole(s), at 0x009F0000 - 0x7FFF0000
Heap : 0x009B0000 - 0xFFFB0000 (Size: 2048 MB)
Compaction : (no compaction area)
Allocation : TLA-min: 2048, TLA-preferred: 16384 TLA-waste limit: 2048
NurseryList : 0x7FFF0000 - 0xE4A95BD0
KeepArea : 0x83BEFFC0 - 0xE4A95BD0
KA Markers : [ 0x827EFFD8, 0x83BEFFC0 , 0xE4A95BD0 ]
Forbidden A : (none)
Previous KA : 0x827EFFD8 - 0x83BEFFC0
Previous FA : (none)
CompRefs : References are 32-bit.
Registers (from ThreadContext: 0x0BA2FA08:
eax = 80d813d8 ecx = 00000003 edx = 00000000 ebx = 0ba2fddc
esp = 0ba2fcf0 ebp = 7fff0000 esi = 0ba2fd6c edi = 02df0000
es = 0000002b cs = 00000023 ss = 0000002b ds = 0000002b
fs = 00000053 gs = 0000002b
eip = 00d24bdf eflags = 00010286
Loaded modules:
(* denotes the module where the exception occured)
00400000-00434fff C:\Program Files (x86)\Java\JRockit\bin\java.exe
77000000-7717ffff C:\Windows\SysWOW64\ntdll.dll
75750000-7585ffff C:\Windows\syswow64\kernel32.dll
76980000-769c6fff C:\Windows\syswow64\KERNELBASE.dll
754d0000-7556ffff C:\Windows\syswow64\ADVAPI32.dll
74d60000-74e0bfff C:\Windows\syswow64\msvcrt.dll
750b0000-750c8fff C:\Windows\SysWOW64\sechost.dll
76810000-768fffff C:\Windows\syswow64\RPCRT4.dll
74b60000-74bbffff C:\Windows\syswow64\SspiCli.dll
74b50000-74b5bfff C:\Windows\syswow64\CRYPTBASE.dll
7c340000-7c395fff C:\Program Files (x86)\Java\JRockit\jre\bin\msvcr71.dll
00bf0000-00ef2fff *C:\Program Files (x86)\Java\JRockit\jre\bin\jrockit\jvm.dll
75860000-75894fff C:\Windows\syswow64\WS2_32.dll
758a0000-758a5fff C:\Windows\syswow64\NSI.dll
74230000-74261fff C:\Windows\system32\WINMM.dll
76ad0000-76bcffff C:\Windows\syswow64\USER32.dll
75020000-750affff C:\Windows\syswow64\GDI32.dll
76fd0000-76fd9fff C:\Windows\syswow64\LPK.dll
74ee0000-74f7cfff C:\Windows\syswow64\USP10.dll
74e80000-74edffff C:\Windows\system32\IMM32.DLL
753f0000-754bbfff C:\Windows\syswow64\MSCTF.dll
73750000-7379bfff C:\Windows\system32\apphelp.dll
10000000-1003cfff C:\Program Files (x86)\Java\JRockit\jre\bin\jrosal.dll
001a0000-001d2fff C:\Program Files (x86)\Java\JRockit\jre\bin\jrutil.DLL
74300000-7433bfff C:\Windows\system32\pdh.dll
743e0000-74400fff C:\Windows\system32\ntmarta.dll
74e30000-74e74fff C:\Windows\syswow64\WLDAP32.dll
73460000-7346afff C:\Windows\System32\perfproc.dll
738e0000-738e8fff C:\Windows\System32\perfos.dll
00540000-0056afff C:\Program Files (x86)\Java\JRockit\jre\bin\jfr.dll
6d870000-6d87bfff C:\Program Files (x86)\Java\JRockit\jre\bin\verify.dll
6d3d0000-6d3eefff C:\Program Files (x86)\Java\JRockit\jre\bin\java.dll
6d330000-6d337fff C:\Program Files (x86)\Java\JRockit\jre\bin\hpi.dll
75910000-7655afff C:\Windows\system32\SHELL32.DLL
75600000-75656fff C:\Windows\syswow64\SHLWAPI.dll
76560000-766bbfff C:\Windows\syswow64\ole32.dll
70c60000-70dfdfff C:\Windows\WinSxS\x86_microsoft.windows.common-controls_6595b64144ccf1df_6.0.7601.17514_none_41e6975e2bd6f2b2\comctl32.dll
756c0000-7574efff C:\Windows\syswow64\OLEAUT32.dll
74bc0000-74d5cfff C:\Windows\syswow64\SETUPAPI.dll
75660000-75686fff C:\Windows\syswow64\CFGMGR32.dll
74e10000-74e21fff C:\Windows\syswow64\DEVOBJ.dll
74f90000-75012fff C:\Windows\syswow64\CLBCatQ.DLL
70a80000-70b74fff C:\Windows\system32\propsys.dll
6d8b0000-6d8befff C:\Program Files (x86)\Java\jrockit\jre\bin\zip.dll
6d600000-6d608fff C:\Program Files (x86)\Java\jrockit\jre\bin\management.dll
6d6b0000-6d6c2fff C:\Program Files (x86)\Java\jrockit\jre\bin\net.dll
74a20000-74a5bfff C:\Windows\system32\mswsock.dll
74820000-74824fff C:\Windows\System32\wshtcpip.dll
73960000-73965fff C:\Windows\System32\wship6.dll
74a70000-74a7ffff C:\Windows\system32\NLAapi.dll
74a60000-74a6ffff C:\Windows\system32\napinsp.dll
74850000-74893fff C:\Windows\system32\DNSAPI.dll
74840000-74847fff C:\Windows\System32\winrnr.dll
742e0000-742fbfff C:\Windows\system32\IPHLPAPI.DLL
742c0000-742c6fff C:\Windows\system32\WINNSI.DLL
74830000-74835fff C:\Windows\system32\rasadhlp.dll
733e0000-73417fff C:\Windows\System32\fwpuclnt.dll
0c900000-0c929fff C:\Program Files (x86)\Java\jrockit\jre\bin\jmapi.dll
0dc90000-0dcb1fff C:\Program Files (x86)\Java\jrockit\jre\bin\orii.dll
746c0000-746d5fff C:\Windows\system32\CRYPTSP.dll
74680000-746bafff C:\Windows\system32\rsaenh.dll
74290000-742a6fff C:\Windows\system32\USERENV.dll
74280000-7428afff C:\Windows\system32\profapi.dll
754c0000-754c4fff C:\Windows\system32\psapi.DLL
0e290000-0e2d3fff D:\APPS\EchoCluster\lib\wrapper.dll
74200000-7420cfff C:\Windows\system32\dhcpcsvc6.DLL
74210000-74221fff C:\Windows\system32\dhcpcsvc.DLL
6d6d0000-6d6d8fff C:\Program Files (x86)\Java\jrockit\jre\bin\nio.dll
71370000-71484fff C:\Program Files (x86)\Java\JRockit\jre\bin\dbghelp.dll
Stack:
(* marks the word pointed to by the stack pointer)
0ba2fcf0: 0ba2fd6c* 11558ea0 00846000 00d24e44 0ba2fd5c 0ba2fddc
0ba2fd08: 11558ea0 00000000 00d29cb0 0ba2fd6c 0ba2fddc 00000000
0ba2fd20: 00d29e5f 11558ea0 0ba2fd6c 1211f558 11558ea0 00000000
0ba2fd38: 1211f39c 00000000 851b0000 84fb0000 00000000 00ed2114
Code:
(* marks the word pointed to by the instruction pointer)
00d24bac: e1830487 a8e8d31f 5f097401 333c5689 c3595dc0 212c152b
00d24bc4: 146800ed 5200ed21 093ee855 c483fffe 0fc0850c 00008984
00d24bdc: 8b108b00* ed33484a 03447a83 c8030474 788b1ceb cfaf0f08
00d24bf4: 10014c8d 0e76ff85 8304528b 7c01447a 0001bd05 4e3b0000
Scan Dump Helper:
Processing the remset iterator at 0x0BA2FD6C.
Processing cards from start card: 0x0B1D3000 to end card: 0x0B1D4000.
The current dirty region is from heap address: 0x84FB0000 to: 0x84FB0200.
The last cleaned card is 0x84FAFE00 and lastRefArray is 0x00000000.
Processing references from object iterator at 0x0BA2FDDC.
The reference that caused the crash is unknown (broken object).
ObjRef.sourceObj = 0x0128CC22 -- ObjRef.targetObj = 0x00000000
ObjRef.ref = 0x0BA2FD50 -- ObjRef.kind = fullref
sourceObj: Seems to be bogus or broken.
The object address points outside the heap.
targetObj: Is NULL.
Trying to locate who is pointing to the sourceObj.
There seems to be no one on the heap referring to the sourceObj.
Last optimized methods:
#705 java/io/ObjectOutputStream.writeUTF(Ljava/lang/String;)V 9056.687-9056.701 0x13491B80-0x13491CFE 13.99 ms 192KB
#706 java/util/HashMap$ValueIterator.next()Ljava/lang/Object; 9374.233-9374.238 0x13491D00-0x13491E1E 4.78 ms 192KB
#707 java/io/ObjectStreamClass$FieldReflector.getObjFieldValues(Ljava/lang/Object;[Ljava/lang/Object;)V 9405.515-9405.524 0x13491E20-0x13492001 9.69 ms 192KB
#708 com/tangosol/coherence/component/net/management/model/localModel/ServiceModel.getPartitionsUnbalanced()I 9637.624-9637.773 0x13492020-0x13493E06 148.76 ms 2688KB
#709 com/tangosol/coherence/component/application/console/Coherence.getSafeCluster()Lcom/tangosol/coherence/component/util/SafeCluster; 9792.764-9792.896 0x13493E20-0x1349585C 131.83 ms 1984KB
#710 com/tangosol/run/xml/XmlHelper.trim(Ljava/lang/String;)Ljava/lang/String; 9933.140-9933.169 0x13495860-0x13495CAD 28.63 ms 384KB
#711 com/tangosol/util/SafeLinkedList.toArray([Ljava/lang/Object;)[Ljava/lang/Object; 10116.984-10117.172 0x13495CC0-0x13496D5B 188.04 ms 1280KB
#712 com/tangosol/coherence/component/net/TcpRing.getSelector()Ljava/nio/channels/Selector; 10525.859-10525.860 0x13496D60-0x13496D6E 1.29 ms 256KB
#713 java/util/WeakHashMap.getEntry(Ljava/lang/Object;)Ljava/util/WeakHashMap$Entry; 10880.654-10880.668 0x13496D80-0x134971B0 13.21 ms 448KB
#714 java/util/WeakHashMap.eq(Ljava/lang/Object;Ljava/lang/Object;)Z 10880.668-10880.668 0x134971C0-0x134971E1 0.38 ms 128KB
*#715 com/tangosol/coherence/component/util/daemon/queueProcessor/service/grid/ClusterService$MemberHeartbeat.write(Lcom/tangosol/io/WriteBuffer$BufferOutput;)V 10974.482-10974.483 0x13497200-0x13497224 0.42 ms 128KB
#700 java/util/HashMap$ValueIterator.<init>(Ljava/util/HashMap;Ljava/util/HashMap$1;)V 7609.065-7609.068 0x13491400-0x134914A8 2.84 ms 192KB
#701 com/tangosol/coherence/component/net/MemberSet.getBitSet(I)I 7660.780-7660.783 0x134914C0-0x134914EA 2.35 ms 128KB
#702 com/tangosol/util/SegmentedHashMap.getEntryInternal(Ljava/lang/Object;)Lcom/tangosol/util/SegmentedHashMap$Entry; 8745.015-8745.053 0x13491500-0x134918AE 38.58 ms 384KB
#703 com/tangosol/coherence/component/util/daemon/queueProcessor/Service.isRunning()Z 8780.124-8780.125 0x134918C0-0x134918E9 0.83 ms 64KB
#704 java/util/TreeMap.get(Ljava/lang/Object;)Ljava/lang/Object; 9028.359-9028.403 0x13491900-0x13491B7F 44.08 ms 320KB
Thread:
"(GC Worker Thread 4)" id=? idx=0x1c tid=5192 lastJavaFrame=0xFFFFFFFC
Stack 0: start=0x0B9F0000, end=0x0BA30000, guards=0x0B9F3000 (ok), forbidden=0x0B9F1000
Thread Stack Trace:
at remSetIterPrepareNewRegion+143(remsetiter.c:354+2)@0x00D24BDF
at remSetIterGetNext+228(remsetiter.c:504+11)@0x00D24E44
at ycProcessRemSetIter+15(ycgc.c:231+15)@0x00D29CB0
at ycProcessRemSet+286(ycgc.c:306+19)@0x00D29E5F
at ycWorkerProcessRoots+556(ycgc.c:387+20)@0x00D2A16D
at mmGCWorkerThread+166(gcthreads.c:828+16)@0x00D13D57
at thread_stub+254(lifecycle.c:808+6)@0x00DC1A0F
at BaseThreadInitThunk+17()@0x7576336A
at RtlInitializeExceptionChain+98()@0x77039F72
at RtlInitializeExceptionChain+53()@0x77039F45
-- Java stack --
Memory usage report:
Could not mark thread stacks regions properly.
Total mapped 2499820KB (reserved=73928KB)
- Java heap 2097152KB (reserved=0KB)
- GC tables 139956KB
- Thread stacks 0KB (#threads=0)
- Compiled code 5568KB (used=5003KB)
- Internal 776KB
- OS 41024KB
- Other 176432KB
- Java class data 37888KB (malloced=37827KB #39179 in 6926 classes)
Not tracing sites.
- Native memory tracking 1024KB (malloced=347KB #8)
Not tracing sites.
Set the env variable TRACE_ALLOC_SITES=1 or use the print_memusage switch
trace_alloc_sites=true to enable alloc site tracing.
Warning! Discrepancy detected while collecting memory usage. Numbers can be wrong.
**************************************************************************
* If you see this dump, please go to *
* http://download.oracle.com/docs/cd/E15289_01/go2troubleshooting.html *
* for troubleshooting information. *
**************************************************************************
===== END DUMP ===============================================================