Hello All
I am running Oracle JRockit(R) R28.2.3-13-149708-1.6.0_31-20120327-1523-windows-ia32, recently I have started to see JVM crashes on our Regression environment.
There are no OutOfMemoryErrors and no StackOverFlowErrors. But, whenever it crashes, it is always on GC Worker Thread (thread number do vary).
Spent a lot of time investigating, no solutions yet. Hence approaching this expert forum, any help would be much appreciated, Please share your opinion.
Thanks
Kannan
Here is the dump file text:
Error Message: Illegal memory access. [54]
Exception Rec: EXCEPTION_ACCESS_VIOLATION (c0000005) at 0x00D24BDF - memory at 0x00000048 could not be read.
Minidump : Wrote mdmp. Size is 2362MB
SafeDllMode : -1
Version : Oracle JRockit(R) R28.2.3-13-149708-1.6.0_31-20120327-1523-windows-ia32
CPU : Intel SSE SSE2 SSE3 SSSE3 SSE4.1 SSE4.2 Intel64
Number CPUs : 8
Tot Phys Mem : 34359271424 (32767 MB)
OS version : Microsoft Windows 2008 R2 version 6.1 Service Pack 1 (Build 7601) (64-bit)
Hypervisor : Unknown (but probably running on a hypervisor)
Thread System: Windows Threads
Java locking : Lazy unlocking enabled (class banning) (transfer banning)
State : JVM is running (Main thread has finished)
Command Line : <modified: I have removed the lengthy command line parameters>
Repository : C:\Users\SVCUKL~1\AppData\Local\Temp\2013_10_25_16_56_55_5436
java.home : C:\Program Files (x86)\Java\JRockit\jre
j.class.path : ..\\lib\wrappertest.jar;..\\lib\wrapper.jar;..\\lib\echo.jar;..\\lib\coherence.jar;..\\lib\coherence-jpa.jar;..\\lib\eclipselink.jar;..\\lib\eclipselink-jpa-modelgen_2.4.2.v20130514-5956486.jar;..\\lib\javax.persistence_1.0.0.jar;..\\lib\javax.persistence_2.0.5.v201212031355.jar;..\\lib\ojdbc6.jar;..\\lib\commons-lang3-3.1.jar;..\\lib\ojdbc5.jar;..\\config\cluster\pof;..\;..\\lib\log4j;..\\lib\log4j-1.2.17.jar;..\\lib\apache-log4j-extras-1.1.jar;..\\lib\monitoring-framework-1.10.0.jar;..\\lib\slf4j-api-1.7.2.jar;..\\lib\slf4j-simple-1.7.2.jar;..\\lib\tibrvj.jar;..\\lib\tibrvnative.jar;..\\lib\tibrv\;..\\config\cluster\server\log4j;..\\lib\coherence-rest.jar;..\\lib\jackson-all-1.8.1.jar;..\\lib\jersey-core-1.8.jar;..\\lib\jersey-json-1.8.jar;..\\lib\jersey-server-1.8.jar;..\\config\cluster\pof\coherence-rest-config.xml;..\\lib\coherence-loadbalancer.jar;..\\lib\commonj.jar;..\\lib\jmscommon.jar;..\\lib\jndi.jar;..\\lib\jta.jar;..\\lib\junit-4.10.jar;..\\lib\wrapperdemo.jar;..\\lib\vtd-xml.jar
j.lib.path : lib;..\\lib\tibrv\;..\\lib\
JAVA_HOME : C:\Program Files (x86)\Java\JRockit
_JAVA_OPTIONS: <not set>
PATH : C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Program Files (x86)\Bank\ODCReal;C:\Program Files\Python26\;C:\Program Files\Reuters\Rendez-vous\bin;C:\Windows\system32\config\systemprofile\AppData\Roaming\Python\Scripts;C:\Program Files (x86)\Bank\ODCReal\;C:\Program Files\SysinternalsSuite;C:\Program Files (x86)\Windows Resource Kits\Tools;C:\Windows\sysint;C:\ORACLE11G32\product\11.2.0\client_1\bin;C:\JAVA_637\BIN;C:\JAVA_637\jre\bin\client;C:\Program Files (x86)\Python26;
StackOverFlow: 0 StackOverFlowErrors have occured
OutOfMemory : 0 OutOfMemoryErrors have occured
C Heap : Good; no memory allocations have failed
GC Strategy : Mode: pausetime, with strategy: genconcon (basic strategy: genconcon)
GC Status : OC is not running. Last finished OC was OC#1.
: YC is currently running. This is YC#111.
YC Promotion : This YC has been able to promote all found objects so far
YC History : Ran 52 YCs before OC#1.
: Started 59 YCs since last OC.
Heap Holes : 1 hole(s), at 0x009F0000 - 0x7FFF0000
Heap : 0x009B0000 - 0xFFFB0000 (Size: 2048 MB)
Compaction : (no compaction area)
Allocation : TLA-min: 2048, TLA-preferred: 16384 TLA-waste limit: 2048
NurseryList : 0x7FFF0000 - 0xE4A95BD0
KeepArea : 0x83BEFFC0 - 0xE4A95BD0
KA Markers : [ 0x827EFFD8, 0x83BEFFC0 , 0xE4A95BD0 ]
Forbidden A : (none)
Previous KA : 0x827EFFD8 - 0x83BEFFC0
Previous FA : (none)
CompRefs : References are 32-bit.
Registers (from ThreadContext: 0x0BA2FA08:
eax = 80d813d8 ecx = 00000003 edx = 00000000 ebx = 0ba2fddc
esp = 0ba2fcf0 ebp = 7fff0000 esi = 0ba2fd6c edi = 02df0000
es = 0000002b cs = 00000023 ss = 0000002b ds = 0000002b
fs = 00000053 gs = 0000002b
eip = 00d24bdf eflags = 00010286
Loaded modules:
(* denotes the module where the exception occured)
00400000-00434fff C:\Program Files (x86)\Java\JRockit\bin\java.exe
77000000-7717ffff C:\Windows\SysWOW64\ntdll.dll
75750000-7585ffff C:\Windows\syswow64\kernel32.dll
76980000-769c6fff C:\Windows\syswow64\KERNELBASE.dll
754d0000-7556ffff C:\Windows\syswow64\ADVAPI32.dll
74d60000-74e0bfff C:\Windows\syswow64\msvcrt.dll
750b0000-750c8fff C:\Windows\SysWOW64\sechost.dll
76810000-768fffff C:\Windows\syswow64\RPCRT4.dll
74b60000-74bbffff C:\Windows\syswow64\SspiCli.dll
74b50000-74b5bfff C:\Windows\syswow64\CRYPTBASE.dll
7c340000-7c395fff C:\Program Files (x86)\Java\JRockit\jre\bin\msvcr71.dll
00bf0000-00ef2fff *C:\Program Files (x86)\Java\JRockit\jre\bin\jrockit\jvm.dll
75860000-75894fff C:\Windows\syswow64\WS2_32.dll
758a0000-758a5fff C:\Windows\syswow64\NSI.dll
74230000-74261fff C:\Windows\system32\WINMM.dll
76ad0000-76bcffff C:\Windows\syswow64\USER32.dll
75020000-750affff C:\Windows\syswow64\GDI32.dll
76fd0000-76fd9fff C:\Windows\syswow64\LPK.dll
74ee0000-74f7cfff C:\Windows\syswow64\USP10.dll
74e80000-74edffff C:\Windows\system32\IMM32.DLL
753f0000-754bbfff C:\Windows\syswow64\MSCTF.dll
73750000-7379bfff C:\Windows\system32\apphelp.dll
10000000-1003cfff C:\Program Files (x86)\Java\JRockit\jre\bin\jrosal.dll
001a0000-001d2fff C:\Program Files (x86)\Java\JRockit\jre\bin\jrutil.DLL
74300000-7433bfff C:\Windows\system32\pdh.dll
743e0000-74400fff C:\Windows\system32\ntmarta.dll
74e30000-74e74fff C:\Windows\syswow64\WLDAP32.dll
73460000-7346afff C:\Windows\System32\perfproc.dll
738e0000-738e8fff C:\Windows\System32\perfos.dll
00540000-0056afff C:\Program Files (x86)\Java\JRockit\jre\bin\jfr.dll
6d870000-6d87bfff C:\Program Files (x86)\Java\JRockit\jre\bin\verify.dll
6d3d0000-6d3eefff C:\Program Files (x86)\Java\JRockit\jre\bin\java.dll
6d330000-6d337fff C:\Program Files (x86)\Java\JRockit\jre\bin\hpi.dll
75910000-7655afff C:\Windows\system32\SHELL32.DLL
75600000-75656fff C:\Windows\syswow64\SHLWAPI.dll
76560000-766bbfff C:\Windows\syswow64\ole32.dll
70c60000-70dfdfff C:\Windows\WinSxS\x86_microsoft.windows.common-controls_6595b64144ccf1df_6.0.7601.17514_none_41e6975e2bd6f2b2\comctl32.dll
756c0000-7574efff C:\Windows\syswow64\OLEAUT32.dll
74bc0000-74d5cfff C:\Windows\syswow64\SETUPAPI.dll
75660000-75686fff C:\Windows\syswow64\CFGMGR32.dll
74e10000-74e21fff C:\Windows\syswow64\DEVOBJ.dll
74f90000-75012fff C:\Windows\syswow64\CLBCatQ.DLL
70a80000-70b74fff C:\Windows\system32\propsys.dll
6d8b0000-6d8befff C:\Program Files (x86)\Java\jrockit\jre\bin\zip.dll
6d600000-6d608fff C:\Program Files (x86)\Java\jrockit\jre\bin\management.dll
6d6b0000-6d6c2fff C:\Program Files (x86)\Java\jrockit\jre\bin\net.dll
74a20000-74a5bfff C:\Windows\system32\mswsock.dll
74820000-74824fff C:\Windows\System32\wshtcpip.dll
73960000-73965fff C:\Windows\System32\wship6.dll
74a70000-74a7ffff C:\Windows\system32\NLAapi.dll
74a60000-74a6ffff C:\Windows\system32\napinsp.dll
74850000-74893fff C:\Windows\system32\DNSAPI.dll
74840000-74847fff C:\Windows\System32\winrnr.dll
742e0000-742fbfff C:\Windows\system32\IPHLPAPI.DLL
742c0000-742c6fff C:\Windows\system32\WINNSI.DLL
74830000-74835fff C:\Windows\system32\rasadhlp.dll
733e0000-73417fff C:\Windows\System32\fwpuclnt.dll
0c900000-0c929fff C:\Program Files (x86)\Java\jrockit\jre\bin\jmapi.dll
0dc90000-0dcb1fff C:\Program Files (x86)\Java\jrockit\jre\bin\orii.dll
746c0000-746d5fff C:\Windows\system32\CRYPTSP.dll
74680000-746bafff C:\Windows\system32\rsaenh.dll
74290000-742a6fff C:\Windows\system32\USERENV.dll
74280000-7428afff C:\Windows\system32\profapi.dll
754c0000-754c4fff C:\Windows\system32\psapi.DLL
0e290000-0e2d3fff D:\APPS\EchoCluster\lib\wrapper.dll
74200000-7420cfff C:\Windows\system32\dhcpcsvc6.DLL
74210000-74221fff C:\Windows\system32\dhcpcsvc.DLL
6d6d0000-6d6d8fff C:\Program Files (x86)\Java\jrockit\jre\bin\nio.dll
71370000-71484fff C:\Program Files (x86)\Java\JRockit\jre\bin\dbghelp.dll
Stack:
(* marks the word pointed to by the stack pointer)
0ba2fcf0: 0ba2fd6c* 11558ea0 00846000 00d24e44 0ba2fd5c 0ba2fddc
0ba2fd08: 11558ea0 00000000 00d29cb0 0ba2fd6c 0ba2fddc 00000000
0ba2fd20: 00d29e5f 11558ea0 0ba2fd6c 1211f558 11558ea0 00000000
0ba2fd38: 1211f39c 00000000 851b0000 84fb0000 00000000 00ed2114
Code:
(* marks the word pointed to by the instruction pointer)
00d24bac: e1830487 a8e8d31f 5f097401 333c5689 c3595dc0 212c152b
00d24bc4: 146800ed 5200ed21 093ee855 c483fffe 0fc0850c 00008984
00d24bdc: 8b108b00* ed33484a 03447a83 c8030474 788b1ceb cfaf0f08
00d24bf4: 10014c8d 0e76ff85 8304528b 7c01447a 0001bd05 4e3b0000
Scan Dump Helper:
Processing the remset iterator at 0x0BA2FD6C.
Processing cards from start card: 0x0B1D3000 to end card: 0x0B1D4000.
The current dirty region is from heap address: 0x84FB0000 to: 0x84FB0200.
The last cleaned card is 0x84FAFE00 and lastRefArray is 0x00000000.
Processing references from object iterator at 0x0BA2FDDC.
The reference that caused the crash is unknown (broken object).
ObjRef.sourceObj = 0x0128CC22 -- ObjRef.targetObj = 0x00000000
ObjRef.ref = 0x0BA2FD50 -- ObjRef.kind = fullref
sourceObj: Seems to be bogus or broken.
The object address points outside the heap.
targetObj: Is NULL.
Trying to locate who is pointing to the sourceObj.
There seems to be no one on the heap referring to the sourceObj.
Last optimized methods:
#705 java/io/ObjectOutputStream.writeUTF(Ljava/lang/String;)V 9056.687-9056.701 0x13491B80-0x13491CFE 13.99 ms 192KB
#706 java/util/HashMap$ValueIterator.next()Ljava/lang/Object; 9374.233-9374.238 0x13491D00-0x13491E1E 4.78 ms 192KB
#707 java/io/ObjectStreamClass$FieldReflector.getObjFieldValues(Ljava/lang/Object;[Ljava/lang/Object;)V 9405.515-9405.524 0x13491E20-0x13492001 9.69 ms 192KB
#708 com/tangosol/coherence/component/net/management/model/localModel/ServiceModel.getPartitionsUnbalanced()I 9637.624-9637.773 0x13492020-0x13493E06 148.76 ms 2688KB
#709 com/tangosol/coherence/component/application/console/Coherence.getSafeCluster()Lcom/tangosol/coherence/component/util/SafeCluster; 9792.764-9792.896 0x13493E20-0x1349585C 131.83 ms 1984KB
#710 com/tangosol/run/xml/XmlHelper.trim(Ljava/lang/String;)Ljava/lang/String; 9933.140-9933.169 0x13495860-0x13495CAD 28.63 ms 384KB
#711 com/tangosol/util/SafeLinkedList.toArray([Ljava/lang/Object;)[Ljava/lang/Object; 10116.984-10117.172 0x13495CC0-0x13496D5B 188.04 ms 1280KB
#712 com/tangosol/coherence/component/net/TcpRing.getSelector()Ljava/nio/channels/Selector; 10525.859-10525.860 0x13496D60-0x13496D6E 1.29 ms 256KB
#713 java/util/WeakHashMap.getEntry(Ljava/lang/Object;)Ljava/util/WeakHashMap$Entry; 10880.654-10880.668 0x13496D80-0x134971B0 13.21 ms 448KB
#714 java/util/WeakHashMap.eq(Ljava/lang/Object;Ljava/lang/Object;)Z 10880.668-10880.668 0x134971C0-0x134971E1 0.38 ms 128KB
*#715 com/tangosol/coherence/component/util/daemon/queueProcessor/service/grid/ClusterService$MemberHeartbeat.write(Lcom/tangosol/io/WriteBuffer$BufferOutput;)V 10974.482-10974.483 0x13497200-0x13497224 0.42 ms 128KB
#700 java/util/HashMap$ValueIterator.<init>(Ljava/util/HashMap;Ljava/util/HashMap$1;)V 7609.065-7609.068 0x13491400-0x134914A8 2.84 ms 192KB
#701 com/tangosol/coherence/component/net/MemberSet.getBitSet(I)I 7660.780-7660.783 0x134914C0-0x134914EA 2.35 ms 128KB
#702 com/tangosol/util/SegmentedHashMap.getEntryInternal(Ljava/lang/Object;)Lcom/tangosol/util/SegmentedHashMap$Entry; 8745.015-8745.053 0x13491500-0x134918AE 38.58 ms 384KB
#703 com/tangosol/coherence/component/util/daemon/queueProcessor/Service.isRunning()Z 8780.124-8780.125 0x134918C0-0x134918E9 0.83 ms 64KB
#704 java/util/TreeMap.get(Ljava/lang/Object;)Ljava/lang/Object; 9028.359-9028.403 0x13491900-0x13491B7F 44.08 ms 320KB
Thread:
"(GC Worker Thread 4)" id=? idx=0x1c tid=5192 lastJavaFrame=0xFFFFFFFC
Stack 0: start=0x0B9F0000, end=0x0BA30000, guards=0x0B9F3000 (ok), forbidden=0x0B9F1000
Thread Stack Trace:
at remSetIterPrepareNewRegion+143(remsetiter.c:354+2)@0x00D24BDF
at remSetIterGetNext+228(remsetiter.c:504+11)@0x00D24E44
at ycProcessRemSetIter+15(ycgc.c:231+15)@0x00D29CB0
at ycProcessRemSet+286(ycgc.c:306+19)@0x00D29E5F
at ycWorkerProcessRoots+556(ycgc.c:387+20)@0x00D2A16D
at mmGCWorkerThread+166(gcthreads.c:828+16)@0x00D13D57
at thread_stub+254(lifecycle.c:808+6)@0x00DC1A0F
at BaseThreadInitThunk+17()@0x7576336A
at RtlInitializeExceptionChain+98()@0x77039F72
at RtlInitializeExceptionChain+53()@0x77039F45
-- Java stack --
Memory usage report:
Could not mark thread stacks regions properly.
Total mapped 2499820KB (reserved=73928KB)
- Java heap 2097152KB (reserved=0KB)
- GC tables 139956KB
- Thread stacks 0KB (#threads=0)
- Compiled code 5568KB (used=5003KB)
- Internal 776KB
- OS 41024KB
- Other 176432KB
- Java class data 37888KB (malloced=37827KB #39179 in 6926 classes)
Not tracing sites.
- Native memory tracking 1024KB (malloced=347KB #8)
Not tracing sites.
Set the env variable TRACE_ALLOC_SITES=1 or use the print_memusage switch
trace_alloc_sites=true to enable alloc site tracing.
=====================