JRockit R28.2 crashes under load
Hi,
we experience crashes of our Jenkins after upgrading to JRockit R28.2.0-79-14677 from R28.1.
Now:
java version "1.6.0_29"
Java(TM) SE Runtime Environment (build 1.6.0_29-b11)
Oracle JRockit(R) (build R28.2.0-79-146777-1.6.0_29-20111005-1807-linux-x86_64, compiled mode)
Before:
java version "1.6.0_20"
Java(TM) SE Runtime Environment (build 1.6.0_20-b02)
Oracle JRockit(R) (build R28.1.0-123-138454-1.6.0_20-20101014-1350-linux-x86_64, compiled mode)
Maybe this dump file already helps, I'll try to get a .core file too. I noticed there has been a similar issue with R28.0.1 (Oracle support [ID 1295504.1] / Bug 10418780) but it should have been fixed in R28.1.1. From the dump-file we have no stack overflow and different signal info (Signal info : si_signo=11, si_code=1 si_addr=0x2aaaf318a910) from the one in the bug description.
===== BEGIN DUMP =============================================================
JRockit dump produced after 0 days, 17:14:01 on Fri Jan 27 09:47:08 2012
**************************************************************************
* If you see this dump, please go to *
* http://download.oracle.com/docs/cd/E15289_01/go2troubleshooting.html *
* for troubleshooting information. *
**************************************************************************
Additional information is available in:
/opt/ci/jenkins/jenkins_slaves/slave1/jrockit.13327.dump
No snapshot file (core dump) will be created because core dumps have been
disabled. To enable core dumping, try "ulimit -c unlimited"
before starting JRockit again.
Error Message: Illegal memory access. [54]
Signal info : si_signo=11, si_code=1 si_addr=0x2aaaf318a910
Version : Oracle JRockit(R) R28.2.0-79-146777-1.6.0_29-20111005-1807-linux-x86_64
CPU : Intel Westmere (HT) SSE SSE2 SSE3 SSSE3 SSE4.1 SSE4.2 Core Intel64
Number CPUs : 24
Tot Phys Mem : 50635268096 (48289 MB)
OS version : CentOS release 5.5 (Final)
Linux version 2.6.18-194.el5 (mockbuild@builder10.centos.org) (gcc version 4.1.2 20080704 (Red Hat 4.1.2-48)) #1 SMP Fri Apr 2 14:58:14 EDT 2010 (x86_64)
Thread System: Linux NPTL
LibC release : 2.5-stable
Java locking : Lazy unlocking enabled (class banning) (transfer banning)
State : JVM is running
Command Line : -Xms256m -Xmx1280m -Djava.security.egd=file:/dev/./urandom -Duser.timezone=Europe/Berlin -Dsun.java.command=slave.jar -cp .:commons-codec.jar -jnlpCredentials xxx:xxx-jnlpUrl http://xxx/computer/slave1/slave-agent.jnlp -Dsun.java.launcher=SUN_STANDARD slave.jar -cp .:commons-codec.jar -jnlpCredentials xxx:xxx-jnlpUrl http://xxx/computer/slave1/slave-agent.jnlp
Repository :
java.home : /opt/java/jrockit1.6.0_29/jre
j.class.path : slave.jar
j.lib.path : /usr/java/packages/lib/amd64:/opt/java/jrockit1.6.0_29/jre/lib/amd64/jrockit:/opt/java/jrockit1.6.0_29/jre/lib/amd64:/opt/java/jrockit1.6.0_29/jre/../lib/amd64
JAVA_HOME : /opt/java/jrockit1.6.0_29
JAVAOPTIONS: <not set>
LD_LIBRARY_PATH: /opt/java/jrockit1.6.0_29/jre/lib/amd64/jrockit:/opt/java/jrockit1.6.0_29/jre/lib/amd64:/opt/java/jrockit1.6.0_29/jre/../lib/amd64
LD_ASSUME_KERNEL: <not set>
LD_PRELOAD : <not set>
StackOverFlow: 0 StackOverFlowErrors have occured
OutOfMemory : 0 OutOfMemoryErrors have occured
C Heap : Good; no memory allocations have failed
GC Strategy : Mode: throughput, with strategy: genparpar (basic strategy: genparpar)
GC Status : OC is not running. Last finished OC was OC#3.
: YC is not running. Last finished YC was YC#23.
YC Promotion : Last YC successfully promoted all objects
YC History : Ran 9 YCs before OC#1.
: Ran 8 YCs before OC#2.
: Ran 6 YCs before OC#3.
: Ran 0 YCs since last OC.
Heap : 0xb0000000 - 0xc0000000 (Size: 256 MB)
Compaction : (no compaction area)
Allocation : TLA-min: 2048, TLA-preferred: 65536 TLA-waste limit: 2048
NurseryList : 0xb0023e28 - 0xb81d3db0
KeepArea : 0xb6159000 - 0xb81d3db0
KA Markers : [ 0xb4198908, 0xb6159000 , 0xb81d3db0 ]
Forbidden A : (none)
Previous KA : 0xb3fdb580 - 0xb5f8dd70
Previous FA : (none)
CompRefs : References are compressed, with heap base 0x0 and shift 0.
Registers (from ThreadContext: 0x42315a30:
rax = 000000001e1e1b20 rcx = 00000000b9ff6810
rdx = 000000001ebaeed0 rbx = 00000000b9ff6810
rsp = 0000000042315e78 rbp = 00000000b8794d88
rsi = 0000000042315ea8 rdi = 000000001e1e1d40
r8 = 00000000b9f3c028 r9 = 0000000042315e88
r10 = 0000000042315e88 r11 = 00002aaaaddef400
r12 = 0000000042315e80 r13 = 00000000b0024608
r14 = 000000001e1e1b20 r15 = 000000001e1e1b20
cs = 0000000000000033 fs = 0000001400000000
gs = 0014000000000000
rip = 00002aaaf318a910 flags = 0000000000000202
Loaded modules:
(* denotes the module where the exception occured)
0000000000400000-00000000004128a3 /opt/java/jrockit1.6.0_29/bin/java
0000003a13000000-0000003a13001f93 /lib64/libdl.so.2
0000003a13400000-0000003a13415a6f /lib64/libpthread.so.0
0000003a12800000-0000003a1294d067 /lib64/libc.so.6
0000003a12400000-0000003a1241bb7f /lib64/ld-linux-x86-64.so.2
00002b462d7ed000-00002b462daf3c2b /opt/java/jrockit1.6.0_29/jre/lib/amd64/jrockit/libjvm.so
00002b462ddce000-00002b462ddecd63 /opt/java/jrockit1.6.0_29/jre/lib/amd64/libjrosal.so
00002b462dff0000-00002b462dffc553 /opt/java/jrockit1.6.0_29/jre/lib/amd64/libjrutil.so
0000003a12c00000-0000003a12c8122f /lib64/libm.so.6
0000003a17000000-0000003a17006e8b /lib64/librt.so.1
00002aaaaabd6000-00002aaaaabdf64b /lib64/libnss_files.so.2
00002aaaaade1000-00002aaaaaded003 /opt/java/jrockit1.6.0_29/jre/lib/amd64/libjfr.so
00002aaaedbd3000-00002aaaedbdf337 /opt/java/jrockit1.6.0_29/jre/lib/amd64/libverify.so
00002aaaedce2000-00002aaaedd0a15b /opt/java/jrockit1.6.0_29/jre/lib/amd64/libjava.so
0000003a1ca00000-0000003a1ca140c7 /lib64/libnsl.so.1
00002aaaede91000-00002aaaede975bf /opt/java/jrockit1.6.0_29/jre/lib/amd64/native_threads/libhpi.so
00002aaaf162b000-00002aaaf1638bc3 /opt/java/jrockit1.6.0_29/jre/lib/amd64/libzip.so
00002aaaf1cbf000-00002aaaf1cd1c87 /opt/java/jrockit1.6.0_29/jre/lib/amd64/libnet.so
00002aaaf1dff000-00002aaaf1e02d73 /lib64/libnss_dns.so.2
0000003a1ea00000-0000003a1ea10823 /lib64/libresolv.so.2
00002aaaf339a000-00002aaaf33a097b /opt/java/jrockit1.6.0_29/jre/lib/amd64/libnio.so
00002aaaf34a2000-00002aaaf34a7f63 /opt/java/jrockit1.6.0_29/jre/lib/amd64/liborii.so
00002aaaf38a9000-00002aaaf38bde33 /tmp/hudson-remoting2651934831509873941/com/sun/jna/linux-amd64/libjnidispatch.so
Stack:
(* marks the word pointed to by the stack pointer)
0000000042315e78: 00002aaaaddef405* 0000000042315e60 00002aaaf8004f70 00002aaaaddef400
0000000042315e98: 000000001e1e1e98 0000000042315e80 00000000b9f3c028 000000001e1e1b20
0000000042315eb8: 00000000b8794d88 00000000b9ff6810 00002aaaaddef34d 00000000b9ff6810
0000000042315ed8: 00002aaaaddef305 00000000b9ff6815 00000000b8794d88 0000000000000030
0000000042315ef8: 00002aaaaddef2d3 000000000000002f 00002aaaaddee9a7 00000000000000c8
0000000042315f18: 00002aaaaddee970 00000000000000c8 00002aaaadb98b15 00000000b87755d8
Code:
(* marks the word pointed to by the instruction pointer)
00002aaaf318a8b0: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
00002aaaf318a8d0: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
00002aaaf318a8f0: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
00002aaaf318a910: 0000000000000000* 0000000000000000 0000000000000000 0000000000000000
00002aaaf318a930: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
00002aaaf318a950: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
Last optimized methods:
*#1 jrockit/vm/Allocator.allocObjectOrArray(JJIZ)Ljava/lang/Object; 6252.364-6252.369 0x2aaaade23000-0x2aaaade231e0 4.57 ms 256KB
Thread:
"Finalizer" id=7 idx=0x64 tid=13353 lastJavaFrame=0x42315e88
Stack 0: start=0x422d4000, end=0x42318000, guards=0x422d9000 (ok), forbidden=0x422d7000
Thread Stack Trace:
at <unknown>(???.c)@0x2aaaf318a910
-- Java stack --
at com/sun/jna/Native.free(J)V(Native Method)
at com/sun/jna/Memory.free(Memory.java:686)
at com/sun/jna/Memory.dispose(Memory.java:144)
at com/sun/jna/Memory.finalize(Memory.java:139)
at jrockit/memory/Finalizer.doFinalize(Finalizer.java:29)
at jrockit/memory/Finalizer.access$300(Finalizer.java:12)
at jrockit/memory/Finalizer$4.run(Finalizer.java:186)
at java/lang/Thread.run(Thread.java:662)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
Memory usage report:
Total mapped 2734748KB (reserved=2283508KB)
- Java heap 1310720KB (reserved=1048576KB)
- GC tables 43852KB
- Thread stacks 19492KB (#threads=35)
- Compiled code 1048576KB (used=2934KB)
- Internal 1416KB
- OS 132764KB
- Other 146952KB
- Classblocks 1536KB (malloced=1459KB #4227)
Not tracing sites.
- Java class data 28416KB (malloced=28046KB #19675 in 4227 classes)
Not tracing sites.
- Native memory tracking 1024KB (malloced=90KB #10)
Not tracing sites.
Set the env variable TRACE_ALLOC_SITES=1 or use the print_memusage switch
trace_alloc_sites=true to enable alloc site tracing.
**************************************************************************
* If you see this dump, please go to *
* http://download.oracle.com/docs/cd/E15289_01/go2troubleshooting.html *
* for troubleshooting information. *
**************************************************************************
===== END DUMP ===============================================================
Kind regards,
Thomas