Daily JVM crashes on Solaris 9 SPARC
967707Oct 11 2012 — edited Oct 26 2012Greetings,
We have a serious problem with getting JVM crashes almost every day on Solaris 9 9/04 s9s_u7wos_09 SPARC.
The server has a lot of power: 128 processors, >60Gb memory.
The application is deployed on JBoss 4.0.4.GA (vm_info: Java HotSpot(TM) Server VM (1.5.0_20-b02) for solaris-sparc)
Typical JVM crash report*:
#
# An unexpected error has been detected by HotSpot Virtual Machine:
#
# SIGBUS (0xa) at pc=0xff147710, pid=359712, tid=135
#
# Java VM: Java HotSpot(TM) Server VM (1.5.0_20-b02 mixed mode)
# Problematic frame:
# C [libc.so.1+0x47710]
#
--------------- T H R E A D ---------------
Current thread (0x023110d8): JavaThread "http-0.0.0.0-8080-6" daemon [_thread_in_native, id=135]
siginfo:si_signo=10, si_errno=0, si_code=1, si_addr=0x696e2f69
Registers:
O0=0x696e2f69 O1=0x696e2f69 O2=0x03596a30 O3=0x02111e98
O4=0x00000038 O5=0x00000000 O6=0x5ed7e9b0 O7=0xff1475b0
G1=0xff1e0870 G2=0x023110d8 G3=0x00000066 G4=0x00007f50
G5=0xff1a0494 G6=0x00000000 G7=0xf84a4200 Y=0x00000000
PC=0xff147710 nPC=0xff1476ec
Top of Stack: (sp=0x5ed7e9b0)
0x5ed7e9b0: 00000039 035969f8 035969f8 00000000
0x5ed7e9c0: ff1c29c8 03596a30 00000000 00000000
0x5ed7e9d0: 03596a00 00000039 00002040 00000008
0x5ed7e9e0: 038ad540 8c619f10 5ed7ea10 ff147188
0x5ed7e9f0: 00000066 5ed7f284 00000000 000000be
0x5ed7ea00: 5ed7f228 f8b8cc64 e40e4500 00000000
0x5ed7ea10: 00002000 035949f0 035949f0 00000000
0x5ed7ea20: ff1c29bc 00000002 00000000 8b32b180
Instructions: (pc=0xff147710)
0xff147700: 02 80 00 05 01 00 00 00 92 10 00 08 10 bf ff f8
0xff147710: d0 02 60 00 e2 22 60 10 d2 24 60 08 d0 04 a0 00
Stack: [0x5ed00000,0x5ed80000), sp=0x5ed7e9b0, free space=506k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C [libc.so.1+0x47710]
C [libc.so.1+0x47190]
C [libc.so.1+0x46f08] malloc+0x28
C [libnet.so+0xc02c] Java_java_net_SocketInputStream_socketRead0+0xd4
J java.net.SocketInputStream.socketRead0(Ljava/io/FileDescriptor;[BIII)I
J java.net.SocketInputStream.read([BII)I
v ~I2CAdapter
j org.apache.coyote.http11.InternalInputBuffer.fill()Z+59
v ~C2IAdapter
J org.apache.coyote.http11.InternalInputBuffer.parseRequestLine()V
J org.apache.coyote.http11.Http11Processor.process(Ljava/io/InputStream;Ljava/io/OutputStream;)V
v ~I2CAdapter
j org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.processConnection(Lorg/apache/tomcat/util/net/TcpConnection;[Ljava/lang/Object;)V+113
j org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(Ljava/net/Socket;Lorg/apache/tomcat/util/net/TcpConnection;[Ljava/lang/Object;)V+45
v ~C2IAdapter
J org.apache.tomcat.util.net.MasterSlaveWorkerThread.run()V
v ~OSRAdapter
j java.lang.Thread.run()V+11
v ~StubRoutines::call_stub
V [libjvm.so+0x19fa44]
V [libjvm.so+0x2c569c]
V [libjvm.so+0x2e4fcc]
V [libjvm.so+0x2e0b68]
V [libjvm.so+0x684e04]
Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
J java.net.SocketInputStream.socketRead0(Ljava/io/FileDescriptor;[BIII)I
J java.net.SocketInputStream.read([BII)I
v ~I2CAdapter
j org.apache.coyote.http11.InternalInputBuffer.fill()Z+59
v ~C2IAdapter
J org.apache.coyote.http11.InternalInputBuffer.parseRequestLine()V
J org.apache.coyote.http11.Http11Processor.process(Ljava/io/InputStream;Ljava/io/OutputStream;)V
v ~I2CAdapter
j org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.processConnection(Lorg/apache/tomcat/util/net/TcpConnection;[Ljava/lang/Object;)V+113
j org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(Ljava/net/Socket;Lorg/apache/tomcat/util/net/TcpConnection;[Ljava/lang/Object;)V+45
v ~C2IAdapter
J org.apache.tomcat.util.net.MasterSlaveWorkerThread.run()V
v ~OSRAdapter
j java.lang.Thread.run()V+11
v ~StubRoutines::call_stub
--------------- P R O C E S S ---------------
Java Threads: ( => current thread )
0x02b4a578 JavaThread "Fork thread child of http-0.0.0.0-8080-124" daemon [_thread_in_native, id=27067]
0x02dc7040 JavaThread "pool-1-thread-7096" [_thread_blocked, id=27037]
0x0359cb98 JavaThread "pool-1-thread-7090" [_thread_blocked, id=27031]
0x02c90a48 JavaThread "Thread-10351" daemon [_thread_blocked, id=27026]
0x00e36f58 JavaThread "ConcurrentTaskBalancedExecutor.QueueProcessor" daemon [_thread_blocked, id=10922]
0x01f701d8 JavaThread "http-0.0.0.0-8080-141" daemon [_thread_blocked, id=8198]
0x028ce9f8 JavaThread "http-0.0.0.0-8080-140" daemon [_thread_blocked, id=293]
0x017665b8 JavaThread "http-0.0.0.0-8080-139" daemon [_thread_blocked, id=292]
0x03694c48 JavaThread "http-0.0.0.0-8080-138" daemon [_thread_blocked, id=291]
0x032a0d90 JavaThread "http-0.0.0.0-8080-137" daemon [_thread_blocked, id=290]
0x017ac4a0 JavaThread "http-0.0.0.0-8080-136" daemon [_thread_blocked, id=289]
0x021d6e00 JavaThread "http-0.0.0.0-8080-135" daemon [_thread_blocked, id=288]
0x01cb1fd0 JavaThread "http-0.0.0.0-8080-134" daemon [_thread_blocked, id=287]
0x03778ac8 JavaThread "http-0.0.0.0-8080-133" daemon [_thread_blocked, id=286]
0x01724c00 JavaThread "http-0.0.0.0-8080-132" daemon [_thread_blocked, id=285]
0x017aaed0 JavaThread "http-0.0.0.0-8080-131" daemon [_thread_in_native, id=284]
0x0204a3c0 JavaThread "http-0.0.0.0-8080-130" daemon [_thread_blocked, id=283]
0x036e3f20 JavaThread "http-0.0.0.0-8080-129" daemon [_thread_blocked, id=282]
0x036e4310 JavaThread "http-0.0.0.0-8080-128" daemon [_thread_blocked, id=281]
0x036e4508 JavaThread "http-0.0.0.0-8080-127" daemon [_thread_blocked, id=280]
0x036e7008 JavaThread "http-0.0.0.0-8080-126" daemon [_thread_blocked, id=279]
0x036e72d0 JavaThread "http-0.0.0.0-8080-125" daemon [_thread_blocked, id=278]
0x036e5760 JavaThread "http-0.0.0.0-8080-124" daemon [_thread_blocked, id=277]
0x01427530 JavaThread "http-0.0.0.0-8080-123" daemon [_thread_blocked, id=276]
0x01426790 JavaThread "http-0.0.0.0-8080-122" daemon [_thread_blocked, id=275]
0x021a5550 JavaThread "http-0.0.0.0-8080-121" daemon [_thread_blocked, id=274]
0x01429058 JavaThread "http-0.0.0.0-8080-120" daemon [_thread_blocked, id=273]
0x01428568 JavaThread "http-0.0.0.0-8080-119" daemon [_thread_blocked, id=272]
0x014239f8 JavaThread "http-0.0.0.0-8080-118" daemon [_thread_blocked, id=271]
0x01429be0 JavaThread "http-0.0.0.0-8080-117" daemon [_thread_blocked, id=270]
0x036e7fb8 JavaThread "http-0.0.0.0-8080-116" daemon [_thread_blocked, id=269]
0x036e1f38 JavaThread "http-0.0.0.0-8080-115" daemon [_thread_blocked, id=268]
0x01424a58 JavaThread "http-0.0.0.0-8080-114" daemon [_thread_blocked, id=267]
0x02e98a08 JavaThread "http-0.0.0.0-8080-113" daemon [_thread_blocked, id=266]
0x02e98c40 JavaThread "http-0.0.0.0-8080-112" daemon [_thread_blocked, id=265]
0x02e93b18 JavaThread "http-0.0.0.0-8080-111" daemon [_thread_blocked, id=264]
0x02e92170 JavaThread "http-0.0.0.0-8080-110" daemon [_thread_blocked, id=263]
0x0215e8e0 JavaThread "http-0.0.0.0-8080-109" daemon [_thread_blocked, id=262]
0x02164788 JavaThread "http-0.0.0.0-8080-108" daemon [_thread_blocked, id=261]
0x021649c8 JavaThread "http-0.0.0.0-8080-107" daemon [_thread_blocked, id=260]
0x02161b70 JavaThread "http-0.0.0.0-8080-106" daemon [_thread_blocked, id=259]
0x02161868 JavaThread "http-0.0.0.0-8080-105" daemon [_thread_blocked, id=258]
0x021643b0 JavaThread "http-0.0.0.0-8080-104" daemon [_thread_blocked, id=257]
0x021606c8 JavaThread "http-0.0.0.0-8080-103" daemon [_thread_blocked, id=256]
0x021614a0 JavaThread "http-0.0.0.0-8080-102" daemon [_thread_blocked, id=255]
0x021600d0 JavaThread "http-0.0.0.0-8080-101" daemon [_thread_blocked, id=254]
0x03138378 JavaThread "http-0.0.0.0-8080-100" daemon [_thread_blocked, id=253]
0x03137fe0 JavaThread "http-0.0.0.0-8080-99" daemon [_thread_blocked, id=252]
0x03132638 JavaThread "http-0.0.0.0-8080-98" daemon [_thread_blocked, id=251]
0x031333f0 JavaThread "http-0.0.0.0-8080-97" daemon [_thread_blocked, id=250]
0x02960a20 JavaThread "http-0.0.0.0-8080-96" daemon [_thread_blocked, id=249]
0x0295fb88 JavaThread "http-0.0.0.0-8080-95" daemon [_thread_blocked, id=248]
0x0295ccf0 JavaThread "http-0.0.0.0-8080-94" daemon [_thread_blocked, id=247]
0x01fcdae0 JavaThread "http-0.0.0.0-8080-93" daemon [_thread_blocked, id=246]
0x0309eba0 JavaThread "http-0.0.0.0-8080-92" daemon [_thread_blocked, id=245]
0x01476db8 JavaThread "http-0.0.0.0-8080-91" daemon [_thread_blocked, id=244]
0x01ec3640 JavaThread "http-0.0.0.0-8080-90" daemon [_thread_blocked, id=243]
0x01ec02e8 JavaThread "http-0.0.0.0-8080-89" daemon [_thread_blocked, id=242]
0x01ebf5d0 JavaThread "http-0.0.0.0-8080-88" daemon [_thread_blocked, id=241]
0x015e9008 JavaThread "http-0.0.0.0-8080-87" daemon [_thread_blocked, id=240]
0x015e7308 JavaThread "http-0.0.0.0-8080-86" daemon [_thread_blocked, id=239]
0x015e4b00 JavaThread "http-0.0.0.0-8080-85" daemon [_thread_blocked, id=238]
0x01589390 JavaThread "http-0.0.0.0-8080-84" daemon [_thread_blocked, id=237]
0x01b12608 JavaThread "http-0.0.0.0-8080-83" daemon [_thread_in_native, id=236]
0x01b11148 JavaThread "http-0.0.0.0-8080-82" daemon [_thread_blocked, id=235]
0x01192f38 JavaThread "http-0.0.0.0-8080-81" daemon [_thread_in_native, id=234]
0x038629d0 JavaThread "http-0.0.0.0-8080-80" daemon [_thread_blocked, id=233]
0x02d42c18 JavaThread "http-0.0.0.0-8080-79" daemon [_thread_in_native, id=232]
0x012bad68 JavaThread "http-0.0.0.0-8080-78" daemon [_thread_blocked, id=231]
0x02c914d0 JavaThread "http-0.0.0.0-8080-77" daemon [_thread_blocked, id=230]
0x01950438 JavaThread "http-0.0.0.0-8080-76" daemon [_thread_blocked, id=229]
0x0194f730 JavaThread "http-0.0.0.0-8080-75" daemon [_thread_blocked, id=228]
0x011350b0 JavaThread "http-0.0.0.0-8080-74" daemon [_thread_blocked, id=227]
0x01e037b8 JavaThread "http-0.0.0.0-8080-73" daemon [_thread_blocked, id=226]
0x02392a90 JavaThread "http-0.0.0.0-8080-72" daemon [_thread_blocked, id=225]
0x020e08b0 JavaThread "http-0.0.0.0-8080-71" daemon [_thread_blocked, id=224]
0x0234fa98 JavaThread "http-0.0.0.0-8080-70" daemon [_thread_in_native, id=223]
0x01244498 JavaThread "http-0.0.0.0-8080-69" daemon [_thread_blocked, id=222]
0x01242fe8 JavaThread "http-0.0.0.0-8080-68" daemon [_thread_blocked, id=221]
0x03783088 JavaThread "http-0.0.0.0-8080-67" daemon [_thread_blocked, id=220]
0x0105f738 JavaThread "http-0.0.0.0-8080-66" daemon [_thread_blocked, id=219]
0x03002f40 JavaThread "http-0.0.0.0-8080-65" daemon [_thread_blocked, id=218]
0x030dec90 JavaThread "http-0.0.0.0-8080-64" daemon [_thread_in_native, id=217]
0x037d53a8 JavaThread "http-0.0.0.0-8080-63" daemon [_thread_blocked, id=216]
0x02aad508 JavaThread "http-0.0.0.0-8080-62" daemon [_thread_blocked, id=215]
0x038053f0 JavaThread "http-0.0.0.0-8080-61" daemon [_thread_blocked, id=214]
0x03803358 JavaThread "http-0.0.0.0-8080-60" daemon [_thread_blocked, id=213]
0x01533328 JavaThread "http-0.0.0.0-8080-59" daemon [_thread_blocked, id=212]
0x00f78260 JavaThread "http-0.0.0.0-8080-58" daemon [_thread_blocked, id=211]
0x01dfe718 JavaThread "http-0.0.0.0-8080-57" daemon [_thread_blocked, id=210]
0x01822358 JavaThread "http-0.0.0.0-8080-56" daemon [_thread_blocked, id=209]
0x018204a0 JavaThread "http-0.0.0.0-8080-55" daemon [_thread_blocked, id=208]
0x03084eb0 JavaThread "http-0.0.0.0-8080-54" daemon [_thread_blocked, id=207]
0x03083d28 JavaThread "http-0.0.0.0-8080-53" daemon [_thread_blocked, id=206]
0x013a4bd8 JavaThread "http-0.0.0.0-8080-52" daemon [_thread_blocked, id=205]
0x0282d418 JavaThread "http-0.0.0.0-8080-51" daemon [_thread_blocked, id=204]
0x032fab50 JavaThread "http-0.0.0.0-8080-50" daemon [_thread_blocked, id=203]
0x0211a9c8 JavaThread "http-0.0.0.0-8080-49" daemon [_thread_blocked, id=202]
0x029b5e00 JavaThread "http-0.0.0.0-8080-48" daemon [_thread_blocked, id=201]
0x02241268 JavaThread "http-0.0.0.0-8080-47" daemon [_thread_blocked, id=200]
0x02241660 JavaThread "http-0.0.0.0-8080-46" daemon [_thread_blocked, id=199]
0x01408ed8 JavaThread "http-0.0.0.0-8080-45" daemon [_thread_blocked, id=198]
0x01408d10 JavaThread "http-0.0.0.0-8080-44" daemon [_thread_blocked, id=197]
0x0386e230 JavaThread "http-0.0.0.0-8080-43" daemon [_thread_blocked, id=196]
0x036c8a98 JavaThread "http-0.0.0.0-8080-42" daemon [_thread_blocked, id=195]
0x01c7c020 JavaThread "http-0.0.0.0-8080-41" daemon [_thread_blocked, id=194]
0x0121c3a0 JavaThread "http-0.0.0.0-8080-40" daemon [_thread_blocked, id=193]
0x0132b278 JavaThread "http-0.0.0.0-8080-39" daemon [_thread_blocked, id=192]
0x032fb980 JavaThread "http-0.0.0.0-8080-38" daemon [_thread_blocked, id=191]
0x00f6c4d0 JavaThread "http-0.0.0.0-8080-37" daemon [_thread_blocked, id=190]
0x01a6aa68 JavaThread "http-0.0.0.0-8080-36" daemon [_thread_blocked, id=189]
0x01a6b268 JavaThread "http-0.0.0.0-8080-35" daemon [_thread_blocked, id=188]
0x0303de78 JavaThread "http-0.0.0.0-8080-34" daemon [_thread_blocked, id=187]
0x01403c78 JavaThread "http-0.0.0.0-8080-33" daemon [_thread_blocked, id=186]
0x01403e70 JavaThread "http-0.0.0.0-8080-32" daemon [_thread_in_native, id=185]
0x0303ce60 JavaThread "http-0.0.0.0-8080-31" daemon [_thread_blocked, id=184]
0x012343a0 JavaThread "http-0.0.0.0-8080-30" daemon [_thread_in_native, id=183]
0x022e4510 JavaThread "http-0.0.0.0-8080-29" daemon [_thread_blocked, id=182]
0x02dd77b0 JavaThread "http-0.0.0.0-8080-28" daemon [_thread_in_native, id=181]
0x00f6c948 JavaThread "http-0.0.0.0-8080-27" daemon [_thread_blocked, id=180]
0x01235390 JavaThread "http-0.0.0.0-8080-26" daemon [_thread_blocked, id=179]
0x01cd8060 JavaThread "http-0.0.0.0-8080-25" daemon [_thread_blocked, id=178]
0x01c39ce8 JavaThread "http-0.0.0.0-8080-24" daemon [_thread_blocked, id=177]
0x02396890 JavaThread "http-0.0.0.0-8080-23" daemon [_thread_in_native, id=176]
0x021f3a50 JavaThread "http-0.0.0.0-8080-22" daemon [_thread_blocked, id=175]
0x021f3888 JavaThread "http-0.0.0.0-8080-21" daemon [_thread_blocked, id=174]
0x03884920 JavaThread "http-0.0.0.0-8080-20" daemon [_thread_blocked, id=173]
0x00ec26b0 JavaThread "http-0.0.0.0-8080-19" daemon [_thread_blocked, id=172]
0x013a5218 JavaThread "http-0.0.0.0-8080-18" daemon [_thread_blocked, id=171]
0x0285eff8 JavaThread "http-0.0.0.0-8080-17" daemon [_thread_blocked, id=170]
0x02952818 JavaThread "http-0.0.0.0-8080-16" daemon [_thread_blocked, id=169]
0x02f11580 JavaThread "http-0.0.0.0-8080-15" daemon [_thread_blocked, id=168]
0x023a7650 JavaThread "http-0.0.0.0-8080-14" daemon [_thread_in_native, id=167]
0x0285e020 JavaThread "http-0.0.0.0-8080-13" daemon [_thread_blocked, id=166]
0x02f118d8 JavaThread "http-0.0.0.0-8080-12" daemon [_thread_blocked, id=165]
0x01a76300 JavaThread "http-0.0.0.0-8080-11" daemon [_thread_blocked, id=164]
0x01a756b0 JavaThread "http-0.0.0.0-8080-10" daemon [_thread_blocked, id=163]
0x029e4868 JavaThread "TP-Monitor" daemon [_thread_blocked, id=162]
0x0298b028 JavaThread "TP-Processor25" daemon [_thread_in_native, id=161]
0x01c3d640 JavaThread "TP-Processor24" daemon [_thread_blocked, id=160]
0x02d71480 JavaThread "TP-Processor23" daemon [_thread_blocked, id=159]
0x02d70d18 JavaThread "http-0.0.0.0-8080-9" daemon [_thread_blocked, id=158]
0x03786d78 JavaThread "TP-Processor22" daemon [_thread_blocked, id=157]
0x03786bb0 JavaThread "TP-Processor21" daemon [_thread_blocked, id=156]
0x01b866c0 JavaThread "TP-Processor20" daemon [_thread_blocked, id=155]
0x01b864f8 JavaThread "TP-Processor19" daemon [_thread_blocked, id=154]
0x01ded090 JavaThread "TP-Processor18" daemon [_thread_blocked, id=153]
0x030cc4d0 JavaThread "TP-Processor17" daemon [_thread_blocked, id=152]
0x02287ff0 JavaThread "http-0.0.0.0-8080-8" daemon [_thread_blocked, id=151]
0x013e5528 JavaThread "TP-Processor16" daemon [_thread_blocked, id=150]
0x02287228 JavaThread "TP-Processor15" daemon [_thread_blocked, id=149]
0x01a30690 JavaThread "TP-Processor14" daemon [_thread_blocked, id=148]
0x032e9628 JavaThread "TP-Processor13" daemon [_thread_blocked, id=147]
0x01fd8138 JavaThread "TP-Processor12" daemon [_thread_blocked, id=146]
0x00f69d00 JavaThread "TP-Processor11" daemon [_thread_blocked, id=145]
0x030cc308 JavaThread "TP-Processor10" daemon [_thread_blocked, id=144]
0x02a512e0 JavaThread "http-0.0.0.0-8080-7" daemon [_thread_blocked, id=143]
0x01cfaed8 JavaThread "TP-Processor9" daemon [_thread_blocked, id=142]
0x0324bcb8 JavaThread "TP-Processor8" daemon [_thread_blocked, id=141]
0x011e67d8 JavaThread "TP-Processor7" daemon [_thread_blocked, id=140]
0x013ed930 JavaThread "TP-Processor6" daemon [_thread_blocked, id=139]
0x01651cb8 JavaThread "TP-Processor5" daemon [_thread_blocked, id=138]
0x015f4960 JavaThread "TP-Processor4" daemon [_thread_blocked, id=137]
0x02070d88 JavaThread "TP-Processor3" daemon [_thread_blocked, id=136]
=>0x023110d8 JavaThread "http-0.0.0.0-8080-6" daemon [_thread_in_native, id=135]
0x0185f550 JavaThread "TP-Processor2" daemon [_thread_blocked, id=134]
0x02361e90 JavaThread "TP-Processor1" daemon [_thread_blocked, id=133]
0x011ebce0 JavaThread "http-0.0.0.0-8080-5" daemon [_thread_blocked, id=132]
0x016516f0 JavaThread "http-0.0.0.0-8080-4" daemon [_thread_blocked, id=131]
0x01db8f20 JavaThread "http-0.0.0.0-8080-3" daemon [_thread_blocked, id=130]
0x01b0cf88 JavaThread "http-0.0.0.0-8080-2" daemon [_thread_blocked, id=129]
0x02130660 JavaThread "http-0.0.0.0-8080-1" daemon [_thread_blocked, id=128]
0x01defa70 JavaThread "http-0.0.0.0-8080" daemon [_thread_in_native, id=127]
0x0185a140 JavaThread "JBossLifeThread" [_thread_blocked, id=126]
0x032c8958 JavaThread "Job Manager" [_thread_blocked, id=122]
0x01799ba8 JavaThread "Bundle File Closer" daemon [_thread_blocked, id=118]
0x013aa0e8 JavaThread "Start Level Event Dispatcher" daemon [_thread_blocked, id=116]
0x01f826d8 JavaThread "Framework Event Dispatcher" daemon [_thread_blocked, id=115]
0x037e1178 JavaThread "UILServerILService Accept Thread" [_thread_in_native, id=113]
0x010d7f38 JavaThread "TimeoutFactory" daemon [_thread_blocked, id=112]
0x02dda480 JavaThread "JCA PoolFiller" [_thread_blocked, id=111]
0x00f28880 JavaThread "Timer-2" daemon [_thread_blocked, id=110]
0x02f224c0 JavaThread "HSQLDB Timer @1696452" daemon [_thread_blocked, id=109]
0x01d391d0 JavaThread "WorkManager(3)-1" daemon [_thread_blocked, id=108]
0x0220c120 JavaThread "JBossMQ Cache Reference Softner" daemon [_thread_blocked, id=107]
0x037e1390 JavaThread "ContainerBackgroundProcessor[StandardEngine[jboss.web]]" daemon [_thread_blocked, id=106]
0x03138e10 JavaThread "RMI TCP Accept-4444" daemon [_thread_in_native, id=105]
0x03170aa8 JavaThread "ClassLoadingPool(2)-1" daemon [_thread_in_native, id=104]
0x00f7cb00 JavaThread "JBoss System Threads(1)-1" daemon [_thread_in_native, id=103]
0x01871958 JavaThread "GC Daemon" daemon [_thread_blocked, id=102]
0x014882e8 JavaThread "RMI Reaper" [_thread_blocked, id=101]
0x019d02d0 JavaThread "Timer-1" daemon [_thread_blocked, id=100]
0x01c4eab0 JavaThread "RMI TCP Accept-0" daemon [_thread_in_native, id=99]
0x02a5e400 JavaThread "ScannerThread" daemon [_thread_blocked, id=98]
0x0183e658 JavaThread "Timer-0" daemon [_thread_blocked, id=97]
0x00037708 JavaThread "DestroyJavaVM" [_thread_blocked, id=1]
0x00e9be28 JavaThread "Low Memory Detector" daemon [_thread_blocked, id=94]
0x00e9ad28 JavaThread "CompilerThread1" daemon [_thread_blocked, id=93]
0x00e99fa8 JavaThread "CompilerThread0" daemon [_thread_blocked, id=92]
0x00e99188 JavaThread "AdapterThread" daemon [_thread_blocked, id=91]
0x00e98300 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=90]
0x00e97768 JavaThread "Surrogate Locker Thread (CMS)" daemon [_thread_blocked, id=89]
0x00e8c668 JavaThread "Finalizer" daemon [_thread_blocked, id=88]
0x00e8af78 JavaThread "Reference Handler" daemon [_thread_blocked, id=87]
Other Threads:
0x00e88690 VMThread [id=86]
0x00e9d030 WatcherThread [id=95]
VM state:at safepoint (normal execution)
VM Mutex/Monitor currently owned by a thread: ([mutex/lock_event])
[0x00036a48/0x00036a78] Safepoint_lock - owner thread: 0x00e88690
[0x00036b68/0x00036b98] Threads_lock - owner thread: 0x00e88690
[0x00032ab8/0x00037078] Heap_lock - owner thread: 0x01871958
Heap
par new generation total 98240K, used 84760K [0x83800000, 0x89800000, 0x89800000)
eden space 98176K, 86% used [0x83800000, 0x88ac6398, 0x897e0000)
from space 64K, 0% used [0x897f0000, 0x897f0000, 0x89800000)
to space 64K, 0% used [0x897e0000, 0x897e0000, 0x897f0000)
concurrent mark-sweep generation total 1474560K, used 150985K [0x89800000, 0xe3800000, 0xe3800000)
concurrent-mark-sweep perm gen total 83128K, used 49825K [0xe3800000, 0xe892e000, 0xf3800000)
Dynamic libraries:
0x00010000 /td01/Mercury_Interactive/Quality_Center/java/bin/java
0xff2d0000 /usr/lib/secure/s9_preload.so.1
0xff1d0000 /usr/lib/libthread.so.1
0xff2fc000 /usr/lib/libdl.so.1
0xff100000 /usr/lib/libc.so.1
0xff2f0000 /usr/platform/SUNW,T5240/lib/libc_psr.so.1
0xfe800000 /td01/Mercury_Interactive/Quality_Center/java/jre/lib/sparc/server/libjvm.so
0xff0e0000 /usr/lib/libsocket.so.1
0xff0c0000 /usr/lib/libsched.so.1
0xff090000 /usr/lib/libCrun.so.1
0xfe7b0000 /usr/lib/libm.so.1
0xfe700000 /usr/lib/libnsl.so.1
0xff070000 /usr/lib/libmp.so.2
0xfe6e0000 /td01/Mercury_Interactive/Quality_Center/java/jre/lib/sparc/native_threads/libhpi.so
0xfe6b0000 /td01/Mercury_Interactive/Quality_Center/java/jre/lib/sparc/libverify.so
0xfe670000 /td01/Mercury_Interactive/Quality_Center/java/jre/lib/sparc/libjava.so
0xfe650000 /td01/Mercury_Interactive/Quality_Center/java/jre/lib/sparc/libzip.so
0xfe240000 /usr/lib/locale/en_US.ISO8859-15/en_US.ISO8859-15.so.2
0xfb790000 /td01/Mercury_Interactive/Quality_Center/java/jre/lib/sparc/libnet.so
0xfb6e0000 /td01/Mercury_Interactive/Quality_Center/java/jre/lib/sparc/libmanagement.so
0xf7790000 /td01/Mercury_Interactive/Quality_Center/java/jre/lib/sparc/libnio.so
0xf76e0000 /usr/lib/librt.so.1
0xf76c0000 /usr/lib/libaio.so.1
0xf76a0000 /usr/lib/libmd5.so.1
0xf75e0000 /usr/lib/libsendfile.so.1
VM Arguments:
jvm_args: -Dprogram.name=run.sh -Xms1536m -Xmx1536m -XX:MaxPermSize=256m -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+PrintGCTimeStamps -XX:+HeapDumpOnOutOfMemoryError -Xloggc:/td01/Mercury_Interactive/Quality_Center/jboss/GCDetails.txt -XX:+PrintGCDetails -XX:HeapDumpPath=/td01/Mercury_Interactive/Quality_Center/jboss/heapdump -Djava.endorsed.dirs=/td01/Mercury_Interactive/Quality_Center/jboss/lib/endorsed
java_command: org.jboss.Main
Launcher Type: SUN_STANDARD
Environment Variables:
PATH=/usr/bin::/etc:/opt/EMCpower/bin/sparcv9:/etc/emc/bin
LD_LIBRARY_PATH=/td01/Mercury_Interactive/Quality_Center/java/jre/lib/sparc/server:/td01/Mercury_Interactive/Quality_Center/java/jre/lib/sparc:/td01/Mercury_Interactive/Quality_Center/java/jre/../lib/sparc
SHELL=/bin/ksh
HOSTTYPE=sparc
OSTYPE=solaris2.9
MACHTYPE=sparc-sun-solaris2.9
Signal Handlers:
SIGSEGV: [libjvm.so+0x715c44], sa_mask[0]=0x7fbffeff, sa_flags=0x00000004
SIGBUS: [libjvm.so+0x715c44], sa_mask[0]=0x7fbffeff, sa_flags=0x00000004
SIGFPE: [libjvm.so+0x278128], sa_mask[0]=0x7fbffeff, sa_flags=0x0000000c
SIGPIPE: [libjvm.so+0x278128], sa_mask[0]=0x7fbffeff, sa_flags=0x0000000c
SIGILL: [libjvm.so+0x278128], sa_mask[0]=0x7fbffeff, sa_flags=0x0000000c
SIGUSR1: [libjvm.so+0x6876ac], sa_mask[0]=0x00000000, sa_flags=0x00000008
SIGUSR2: [libjvm.so+0x278128], sa_mask[0]=0x7fbffeff, sa_flags=0x0000000c
SIGHUP: [libjvm.so+0x6862e0], sa_mask[0]=0x7fbffeff, sa_flags=0x00000004
SIGINT: SIG_IGN, sa_mask[0]=0x00000000, sa_flags=0x00000000
SIGQUIT: [libjvm.so+0x6862e0], sa_mask[0]=0x7fbffeff, sa_flags=0x00000004
SIGTERM: [libjvm.so+0x6862e0], sa_mask[0]=0x7fbffeff, sa_flags=0x00000004
--------------- S Y S T E M ---------------
OS: Solaris 9 9/04 s9s_u7wos_09 SPARC
Copyright 2004 Sun Microsystems, Inc. All Rights Reserved.
Use is subject to license terms.
Assembled 29 June 2004
uname:SunOS 5.9 Generic_Virtual sun4u (T2 libthread)
rlimit: STACK 8192k, CORE infinity, NOFILE 1024, AS infinity
load average:2.37 3.28 3.27
CPU:total 128 has_v8, has_v9, has_vis1, has_vis2, is_ultra3, is_sun4v, is_niagara1
Memory: 8k page, physical 66879488k(57508904k free)
vm_info: Java HotSpot(TM) Server VM (1.5.0_20-b02) for solaris-sparc, built on Jun 18 2009 11:46:08 by unknown with unknown Workshop:0x550
time: Wed Oct 3 11:18:29 2012
elapsed time: 511743 seconds
We've been able to obtain a java thread dump before the system was crashed.
No deadlocks found.
However we found out that there are almost 90% of threads are blocked and about 10% of threads are IN_NATIVE state trying to read/write from the socket.
Here is also GC Details:_
8355.870: [GC 8355.870: [ParNew: 98176K->0K(98240K), 0.1164923 secs] 199196K->114491K(1572800K), 0.1171990 secs]
8393.664: [GC 8393.664: [ParNew: 98176K->0K(98240K), 0.2729720 secs] 212667K->124356K(1572800K), 0.2737070 secs]
8401.385: [Full GC 8401.385: [CMS: 124356K->101787K(1474560K), 3.6843095 secs] 130753K->101787K(1572800K), [CMS Perm : 47315K->47315K(78968K)], 3.6853905 secs]
8443.244: [GC 8443.244: [ParNew: 98152K->0K(98240K), 0.1478550 secs] 199939K->109563K(1572800K), 0.1485699 secs]
8464.668: [GC 8464.668: [ParNew: 98157K->0K(98240K), 0.0728873 secs] 207721K->112895K(1572800K), 0.0735619 secs]
8465.074: [Full GC 8465.074: [CMS: 112895K->100601K(1474560K), 3.5895005 secs] 117571K->100601K(1572800K), [CMS Perm : 47319K->47315K(78968K)], 3.5905628 secs]
8486.148: [GC 8486.148: [ParNew: 98176K->0K(98240K), 0.0749352 secs] 198777K->102554K(1572800K), 0.0756663 secs]
8488.338: [GC 8488.339: [ParNew: 98176K->0K(98240K), 0.0485796 secs] 200730K->104867K(1572800K), 0.0492671 secs]
8489.852: [GC 8489.852: [ParNew: 97355K->0K(98240K), 0.0970597 secs] 202223K->113739K(1572800K), 0.0978220 secs]
8528.320: [GC 8528.321: [ParNew: 98127K->0K(98240K), 0.2305933 secs] 211867K->119583K(1572800K), 0.2312862 secs]
8528.673: [Full GC 8528.674: [CMS: 119583K->99402K(1474560K), 3.8037133 secs] 119976K->99402K(1572800K), [CMS Perm : 47316K->47316K(78968K)], 3.8047208 secs]
8574.567: [GC 8574.567: [ParNew: 97721K->0K(98240K), 0.0788791 secs] 197124K->105527K(1572800K), 0.0795659 secs]
8592.816: [Full GC 8592.821: [CMS: 105527K->99590K(1474560K), 6.3920388 secs] 151624K->99590K(1572800K), [CMS Perm : 47320K->47316K(78968K)], 6.3976484 secs]
8619.543: [GC 8619.544: [ParNew: 98176K->0K(98240K), 1.4033638 secs] 197766K->118382K(1572800K), 1.4043006 secs]
8638.976: [GC 8638.976: [ParNew: 98163K->0K(98240K), 0.7486280 secs] 216545K->133435K(1572800K), 0.7493760 secs]
8659.225: [Full GC 8659.225: [CMS: 133435K->127207K(1474560K), 6.5614305 secs] 202428K->127207K(1572800K), [CMS Perm : 47327K->47319K(78968K)], 6.5624836 secs]
8692.489: [GC 8692.489: [ParNew: 98168K->0K(98240K), 0.4273621 secs] 225376K->129320K(1572800K), 0.4283057 secs]
8725.804: [Full GC 8725.804: [CMS: 129320K->125641K(1474560K), 6.7880081 secs] 185202K->125641K(1572800K), [CMS Perm : 47320K->47320K(78968K)], 6.7891931 secs]
8775.026: [GC 8775.027: [ParNew: 98134K->0K(98240K), 0.3575806 secs] 223775K->128092K(1572800K), 0.3582847 secs]
8792.604: [Full GC 8792.605: [CMS: 128092K->126040K(1474560K), 4.9224131 secs] 163371K->126040K(1572800K), [CMS Perm : 47320K->47320K(78968K)], 4.9234914 secs]
8857.535: [Full GC 8857.535: [CMS: 126040K->92630K(1474560K), 3.8167362 secs] 142517K->92630K(1572800K), [CMS Perm : 47320K->47320K(78968K)], 3.8178372 secs]
8921.375: [Full GC 8921.377: [CMS: 92630K->93971K(1474560K), 6.0901600 secs] 181462K->93971K(1572800K), [CMS Perm : 47323K->47320K(78968K)], 6.0932058 secs]
8987.475: [Full GC 8987.475: [CMS: 93971K->91872K(1474560K), 3.4701208 secs] 156871K->91872K(1572800K), [CMS Perm : 47320K->47320K(78968K)], 3.4711992 secs]
9050.955: [Full GC 9050.955: [CMS: 91872K->91416K(1474560K), 3.4062799 secs] 114665K->91416K(1572800K), [CMS Perm : 47321K->47320K(78968K)], 3.4073394 secs]
Notice that upgrading to JDK 6 is not an option.
Any help would be really appreciated.
Let me know if any additional information is required.
Thanks in advance,
Vadim Lotar.