Skip to Main Content

Java APIs

Daily JVM crashes on Solaris 9 SPARC

967707Oct 11 2012 — edited Oct 26 2012

We have a serious problem with getting JVM crashes almost every day on Solaris 9 9/04 s9s_u7wos_09 SPARC.

The server has a lot of power: 128 processors, >60Gb memory.

The application is deployed on JBoss 4.0.4.GA (vm_info: Java HotSpot(TM) Server VM (1.5.0_20-b02) for solaris-sparc)

Typical JVM crash report*:

# An unexpected error has been detected by HotSpot Virtual Machine:
# SIGBUS (0xa) at pc=0xff147710, pid=359712, tid=135
# Java VM: Java HotSpot(TM) Server VM (1.5.0_20-b02 mixed mode)
# Problematic frame:
# C []

--------------- T H R E A D ---------------

Current thread (0x023110d8): JavaThread "http-" daemon [_thread_in_native, id=135]

siginfo:si_signo=10, si_errno=0, si_code=1, si_addr=0x696e2f69

O0=0x696e2f69 O1=0x696e2f69 O2=0x03596a30 O3=0x02111e98
O4=0x00000038 O5=0x00000000 O6=0x5ed7e9b0 O7=0xff1475b0
G1=0xff1e0870 G2=0x023110d8 G3=0x00000066 G4=0x00007f50
G5=0xff1a0494 G6=0x00000000 G7=0xf84a4200 Y=0x00000000
PC=0xff147710 nPC=0xff1476ec

Top of Stack: (sp=0x5ed7e9b0)
0x5ed7e9b0: 00000039 035969f8 035969f8 00000000
0x5ed7e9c0: ff1c29c8 03596a30 00000000 00000000
0x5ed7e9d0: 03596a00 00000039 00002040 00000008
0x5ed7e9e0: 038ad540 8c619f10 5ed7ea10 ff147188
0x5ed7e9f0: 00000066 5ed7f284 00000000 000000be
0x5ed7ea00: 5ed7f228 f8b8cc64 e40e4500 00000000
0x5ed7ea10: 00002000 035949f0 035949f0 00000000
0x5ed7ea20: ff1c29bc 00000002 00000000 8b32b180

Instructions: (pc=0xff147710)
0xff147700: 02 80 00 05 01 00 00 00 92 10 00 08 10 bf ff f8
0xff147710: d0 02 60 00 e2 22 60 10 d2 24 60 08 d0 04 a0 00

Stack: [0x5ed00000,0x5ed80000), sp=0x5ed7e9b0, free space=506k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C []
C []
C [] malloc+0x28
C [] Java_java_net_SocketInputStream_socketRead0+0xd4
v ~I2CAdapter
j org.apache.coyote.http11.InternalInputBuffer.fill()Z+59
v ~C2IAdapter
J org.apache.coyote.http11.InternalInputBuffer.parseRequestLine()V
J org.apache.coyote.http11.Http11Processor.process(Ljava/io/InputStream;Ljava/io/OutputStream;)V
v ~I2CAdapter
j org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.processConnection(Lorg/apache/tomcat/util/net/TcpConnection;[Ljava/lang/Object;)V+113
v ~C2IAdapter
v ~OSRAdapter
v ~StubRoutines::call_stub
V []
V []
V []
V []
V []

Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
v ~I2CAdapter
j org.apache.coyote.http11.InternalInputBuffer.fill()Z+59
v ~C2IAdapter
J org.apache.coyote.http11.InternalInputBuffer.parseRequestLine()V
J org.apache.coyote.http11.Http11Processor.process(Ljava/io/InputStream;Ljava/io/OutputStream;)V
v ~I2CAdapter
j org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.processConnection(Lorg/apache/tomcat/util/net/TcpConnection;[Ljava/lang/Object;)V+113
v ~C2IAdapter
v ~OSRAdapter
v ~StubRoutines::call_stub

--------------- P R O C E S S ---------------

Java Threads: ( => current thread )
0x02b4a578 JavaThread "Fork thread child of http-" daemon [_thread_in_native, id=27067]
0x02dc7040 JavaThread "pool-1-thread-7096" [_thread_blocked, id=27037]
0x0359cb98 JavaThread "pool-1-thread-7090" [_thread_blocked, id=27031]
0x02c90a48 JavaThread "Thread-10351" daemon [_thread_blocked, id=27026]
0x00e36f58 JavaThread "ConcurrentTaskBalancedExecutor.QueueProcessor" daemon [_thread_blocked, id=10922]
0x01f701d8 JavaThread "http-" daemon [_thread_blocked, id=8198]
0x028ce9f8 JavaThread "http-" daemon [_thread_blocked, id=293]
0x017665b8 JavaThread "http-" daemon [_thread_blocked, id=292]
0x03694c48 JavaThread "http-" daemon [_thread_blocked, id=291]
0x032a0d90 JavaThread "http-" daemon [_thread_blocked, id=290]
0x017ac4a0 JavaThread "http-" daemon [_thread_blocked, id=289]
0x021d6e00 JavaThread "http-" daemon [_thread_blocked, id=288]
0x01cb1fd0 JavaThread "http-" daemon [_thread_blocked, id=287]
0x03778ac8 JavaThread "http-" daemon [_thread_blocked, id=286]
0x01724c00 JavaThread "http-" daemon [_thread_blocked, id=285]
0x017aaed0 JavaThread "http-" daemon [_thread_in_native, id=284]
0x0204a3c0 JavaThread "http-" daemon [_thread_blocked, id=283]
0x036e3f20 JavaThread "http-" daemon [_thread_blocked, id=282]
0x036e4310 JavaThread "http-" daemon [_thread_blocked, id=281]
0x036e4508 JavaThread "http-" daemon [_thread_blocked, id=280]
0x036e7008 JavaThread "http-" daemon [_thread_blocked, id=279]
0x036e72d0 JavaThread "http-" daemon [_thread_blocked, id=278]
0x036e5760 JavaThread "http-" daemon [_thread_blocked, id=277]
0x01427530 JavaThread "http-" daemon [_thread_blocked, id=276]
0x01426790 JavaThread "http-" daemon [_thread_blocked, id=275]
0x021a5550 JavaThread "http-" daemon [_thread_blocked, id=274]
0x01429058 JavaThread "http-" daemon [_thread_blocked, id=273]
0x01428568 JavaThread "http-" daemon [_thread_blocked, id=272]
0x014239f8 JavaThread "http-" daemon [_thread_blocked, id=271]
0x01429be0 JavaThread "http-" daemon [_thread_blocked, id=270]
0x036e7fb8 JavaThread "http-" daemon [_thread_blocked, id=269]
0x036e1f38 JavaThread "http-" daemon [_thread_blocked, id=268]
0x01424a58 JavaThread "http-" daemon [_thread_blocked, id=267]
0x02e98a08 JavaThread "http-" daemon [_thread_blocked, id=266]
0x02e98c40 JavaThread "http-" daemon [_thread_blocked, id=265]
0x02e93b18 JavaThread "http-" daemon [_thread_blocked, id=264]
0x02e92170 JavaThread "http-" daemon [_thread_blocked, id=263]
0x0215e8e0 JavaThread "http-" daemon [_thread_blocked, id=262]
0x02164788 JavaThread "http-" daemon [_thread_blocked, id=261]
0x021649c8 JavaThread "http-" daemon [_thread_blocked, id=260]
0x02161b70 JavaThread "http-" daemon [_thread_blocked, id=259]
0x02161868 JavaThread "http-" daemon [_thread_blocked, id=258]
0x021643b0 JavaThread "http-" daemon [_thread_blocked, id=257]
0x021606c8 JavaThread "http-" daemon [_thread_blocked, id=256]
0x021614a0 JavaThread "http-" daemon [_thread_blocked, id=255]
0x021600d0 JavaThread "http-" daemon [_thread_blocked, id=254]
0x03138378 JavaThread "http-" daemon [_thread_blocked, id=253]
0x03137fe0 JavaThread "http-" daemon [_thread_blocked, id=252]
0x03132638 JavaThread "http-" daemon [_thread_blocked, id=251]
0x031333f0 JavaThread "http-" daemon [_thread_blocked, id=250]
0x02960a20 JavaThread "http-" daemon [_thread_blocked, id=249]
0x0295fb88 JavaThread "http-" daemon [_thread_blocked, id=248]
0x0295ccf0 JavaThread "http-" daemon [_thread_blocked, id=247]
0x01fcdae0 JavaThread "http-" daemon [_thread_blocked, id=246]
0x0309eba0 JavaThread "http-" daemon [_thread_blocked, id=245]
0x01476db8 JavaThread "http-" daemon [_thread_blocked, id=244]
0x01ec3640 JavaThread "http-" daemon [_thread_blocked, id=243]
0x01ec02e8 JavaThread "http-" daemon [_thread_blocked, id=242]
0x01ebf5d0 JavaThread "http-" daemon [_thread_blocked, id=241]
0x015e9008 JavaThread "http-" daemon [_thread_blocked, id=240]
0x015e7308 JavaThread "http-" daemon [_thread_blocked, id=239]
0x015e4b00 JavaThread "http-" daemon [_thread_blocked, id=238]
0x01589390 JavaThread "http-" daemon [_thread_blocked, id=237]
0x01b12608 JavaThread "http-" daemon [_thread_in_native, id=236]
0x01b11148 JavaThread "http-" daemon [_thread_blocked, id=235]
0x01192f38 JavaThread "http-" daemon [_thread_in_native, id=234]
0x038629d0 JavaThread "http-" daemon [_thread_blocked, id=233]
0x02d42c18 JavaThread "http-" daemon [_thread_in_native, id=232]
0x012bad68 JavaThread "http-" daemon [_thread_blocked, id=231]
0x02c914d0 JavaThread "http-" daemon [_thread_blocked, id=230]
0x01950438 JavaThread "http-" daemon [_thread_blocked, id=229]
0x0194f730 JavaThread "http-" daemon [_thread_blocked, id=228]
0x011350b0 JavaThread "http-" daemon [_thread_blocked, id=227]
0x01e037b8 JavaThread "http-" daemon [_thread_blocked, id=226]
0x02392a90 JavaThread "http-" daemon [_thread_blocked, id=225]
0x020e08b0 JavaThread "http-" daemon [_thread_blocked, id=224]
0x0234fa98 JavaThread "http-" daemon [_thread_in_native, id=223]
0x01244498 JavaThread "http-" daemon [_thread_blocked, id=222]
0x01242fe8 JavaThread "http-" daemon [_thread_blocked, id=221]
0x03783088 JavaThread "http-" daemon [_thread_blocked, id=220]
0x0105f738 JavaThread "http-" daemon [_thread_blocked, id=219]
0x03002f40 JavaThread "http-" daemon [_thread_blocked, id=218]
0x030dec90 JavaThread "http-" daemon [_thread_in_native, id=217]
0x037d53a8 JavaThread "http-" daemon [_thread_blocked, id=216]
0x02aad508 JavaThread "http-" daemon [_thread_blocked, id=215]
0x038053f0 JavaThread "http-" daemon [_thread_blocked, id=214]
0x03803358 JavaThread "http-" daemon [_thread_blocked, id=213]
0x01533328 JavaThread "http-" daemon [_thread_blocked, id=212]
0x00f78260 JavaThread "http-" daemon [_thread_blocked, id=211]
0x01dfe718 JavaThread "http-" daemon [_thread_blocked, id=210]
0x01822358 JavaThread "http-" daemon [_thread_blocked, id=209]
0x018204a0 JavaThread "http-" daemon [_thread_blocked, id=208]
0x03084eb0 JavaThread "http-" daemon [_thread_blocked, id=207]
0x03083d28 JavaThread "http-" daemon [_thread_blocked, id=206]
0x013a4bd8 JavaThread "http-" daemon [_thread_blocked, id=205]
0x0282d418 JavaThread "http-" daemon [_thread_blocked, id=204]
0x032fab50 JavaThread "http-" daemon [_thread_blocked, id=203]
0x0211a9c8 JavaThread "http-" daemon [_thread_blocked, id=202]
0x029b5e00 JavaThread "http-" daemon [_thread_blocked, id=201]
0x02241268 JavaThread "http-" daemon [_thread_blocked, id=200]
0x02241660 JavaThread "http-" daemon [_thread_blocked, id=199]
0x01408ed8 JavaThread "http-" daemon [_thread_blocked, id=198]
0x01408d10 JavaThread "http-" daemon [_thread_blocked, id=197]
0x0386e230 JavaThread "http-" daemon [_thread_blocked, id=196]
0x036c8a98 JavaThread "http-" daemon [_thread_blocked, id=195]
0x01c7c020 JavaThread "http-" daemon [_thread_blocked, id=194]
0x0121c3a0 JavaThread "http-" daemon [_thread_blocked, id=193]
0x0132b278 JavaThread "http-" daemon [_thread_blocked, id=192]
0x032fb980 JavaThread "http-" daemon [_thread_blocked, id=191]
0x00f6c4d0 JavaThread "http-" daemon [_thread_blocked, id=190]
0x01a6aa68 JavaThread "http-" daemon [_thread_blocked, id=189]
0x01a6b268 JavaThread "http-" daemon [_thread_blocked, id=188]
0x0303de78 JavaThread "http-" daemon [_thread_blocked, id=187]
0x01403c78 JavaThread "http-" daemon [_thread_blocked, id=186]
0x01403e70 JavaThread "http-" daemon [_thread_in_native, id=185]
0x0303ce60 JavaThread "http-" daemon [_thread_blocked, id=184]
0x012343a0 JavaThread "http-" daemon [_thread_in_native, id=183]
0x022e4510 JavaThread "http-" daemon [_thread_blocked, id=182]
0x02dd77b0 JavaThread "http-" daemon [_thread_in_native, id=181]
0x00f6c948 JavaThread "http-" daemon [_thread_blocked, id=180]
0x01235390 JavaThread "http-" daemon [_thread_blocked, id=179]
0x01cd8060 JavaThread "http-" daemon [_thread_blocked, id=178]
0x01c39ce8 JavaThread "http-" daemon [_thread_blocked, id=177]
0x02396890 JavaThread "http-" daemon [_thread_in_native, id=176]
0x021f3a50 JavaThread "http-" daemon [_thread_blocked, id=175]
0x021f3888 JavaThread "http-" daemon [_thread_blocked, id=174]
0x03884920 JavaThread "http-" daemon [_thread_blocked, id=173]
0x00ec26b0 JavaThread "http-" daemon [_thread_blocked, id=172]
0x013a5218 JavaThread "http-" daemon [_thread_blocked, id=171]
0x0285eff8 JavaThread "http-" daemon [_thread_blocked, id=170]
0x02952818 JavaThread "http-" daemon [_thread_blocked, id=169]
0x02f11580 JavaThread "http-" daemon [_thread_blocked, id=168]
0x023a7650 JavaThread "http-" daemon [_thread_in_native, id=167]
0x0285e020 JavaThread "http-" daemon [_thread_blocked, id=166]
0x02f118d8 JavaThread "http-" daemon [_thread_blocked, id=165]
0x01a76300 JavaThread "http-" daemon [_thread_blocked, id=164]
0x01a756b0 JavaThread "http-" daemon [_thread_blocked, id=163]
0x029e4868 JavaThread "TP-Monitor" daemon [_thread_blocked, id=162]
0x0298b028 JavaThread "TP-Processor25" daemon [_thread_in_native, id=161]
0x01c3d640 JavaThread "TP-Processor24" daemon [_thread_blocked, id=160]
0x02d71480 JavaThread "TP-Processor23" daemon [_thread_blocked, id=159]
0x02d70d18 JavaThread "http-" daemon [_thread_blocked, id=158]
0x03786d78 JavaThread "TP-Processor22" daemon [_thread_blocked, id=157]
0x03786bb0 JavaThread "TP-Processor21" daemon [_thread_blocked, id=156]
0x01b866c0 JavaThread "TP-Processor20" daemon [_thread_blocked, id=155]
0x01b864f8 JavaThread "TP-Processor19" daemon [_thread_blocked, id=154]
0x01ded090 JavaThread "TP-Processor18" daemon [_thread_blocked, id=153]
0x030cc4d0 JavaThread "TP-Processor17" daemon [_thread_blocked, id=152]
0x02287ff0 JavaThread "http-" daemon [_thread_blocked, id=151]
0x013e5528 JavaThread "TP-Processor16" daemon [_thread_blocked, id=150]
0x02287228 JavaThread "TP-Processor15" daemon [_thread_blocked, id=149]
0x01a30690 JavaThread "TP-Processor14" daemon [_thread_blocked, id=148]
0x032e9628 JavaThread "TP-Processor13" daemon [_thread_blocked, id=147]
0x01fd8138 JavaThread "TP-Processor12" daemon [_thread_blocked, id=146]
0x00f69d00 JavaThread "TP-Processor11" daemon [_thread_blocked, id=145]
0x030cc308 JavaThread "TP-Processor10" daemon [_thread_blocked, id=144]
0x02a512e0 JavaThread "http-" daemon [_thread_blocked, id=143]
0x01cfaed8 JavaThread "TP-Processor9" daemon [_thread_blocked, id=142]
0x0324bcb8 JavaThread "TP-Processor8" daemon [_thread_blocked, id=141]
0x011e67d8 JavaThread "TP-Processor7" daemon [_thread_blocked, id=140]
0x013ed930 JavaThread "TP-Processor6" daemon [_thread_blocked, id=139]
0x01651cb8 JavaThread "TP-Processor5" daemon [_thread_blocked, id=138]
0x015f4960 JavaThread "TP-Processor4" daemon [_thread_blocked, id=137]
0x02070d88 JavaThread "TP-Processor3" daemon [_thread_blocked, id=136]
=>0x023110d8 JavaThread "http-" daemon [_thread_in_native, id=135]
0x0185f550 JavaThread "TP-Processor2" daemon [_thread_blocked, id=134]
0x02361e90 JavaThread "TP-Processor1" daemon [_thread_blocked, id=133]
0x011ebce0 JavaThread "http-" daemon [_thread_blocked, id=132]
0x016516f0 JavaThread "http-" daemon [_thread_blocked, id=131]
0x01db8f20 JavaThread "http-" daemon [_thread_blocked, id=130]
0x01b0cf88 JavaThread "http-" daemon [_thread_blocked, id=129]
0x02130660 JavaThread "http-" daemon [_thread_blocked, id=128]
0x01defa70 JavaThread "http-" daemon [_thread_in_native, id=127]
0x0185a140 JavaThread "JBossLifeThread" [_thread_blocked, id=126]
0x032c8958 JavaThread "Job Manager" [_thread_blocked, id=122]
0x01799ba8 JavaThread "Bundle File Closer" daemon [_thread_blocked, id=118]
0x013aa0e8 JavaThread "Start Level Event Dispatcher" daemon [_thread_blocked, id=116]
0x01f826d8 JavaThread "Framework Event Dispatcher" daemon [_thread_blocked, id=115]
0x037e1178 JavaThread "UILServerILService Accept Thread" [_thread_in_native, id=113]
0x010d7f38 JavaThread "TimeoutFactory" daemon [_thread_blocked, id=112]
0x02dda480 JavaThread "JCA PoolFiller" [_thread_blocked, id=111]
0x00f28880 JavaThread "Timer-2" daemon [_thread_blocked, id=110]
0x02f224c0 JavaThread "HSQLDB Timer @1696452" daemon [_thread_blocked, id=109]
0x01d391d0 JavaThread "WorkManager(3)-1" daemon [_thread_blocked, id=108]
0x0220c120 JavaThread "JBossMQ Cache Reference Softner" daemon [_thread_blocked, id=107]
0x037e1390 JavaThread "ContainerBackgroundProcessor[StandardEngine[jboss.web]]" daemon [_thread_blocked, id=106]
0x03138e10 JavaThread "RMI TCP Accept-4444" daemon [_thread_in_native, id=105]
0x03170aa8 JavaThread "ClassLoadingPool(2)-1" daemon [_thread_in_native, id=104]
0x00f7cb00 JavaThread "JBoss System Threads(1)-1" daemon [_thread_in_native, id=103]
0x01871958 JavaThread "GC Daemon" daemon [_thread_blocked, id=102]
0x014882e8 JavaThread "RMI Reaper" [_thread_blocked, id=101]
0x019d02d0 JavaThread "Timer-1" daemon [_thread_blocked, id=100]
0x01c4eab0 JavaThread "RMI TCP Accept-0" daemon [_thread_in_native, id=99]
0x02a5e400 JavaThread "ScannerThread" daemon [_thread_blocked, id=98]
0x0183e658 JavaThread "Timer-0" daemon [_thread_blocked, id=97]
0x00037708 JavaThread "DestroyJavaVM" [_thread_blocked, id=1]
0x00e9be28 JavaThread "Low Memory Detector" daemon [_thread_blocked, id=94]
0x00e9ad28 JavaThread "CompilerThread1" daemon [_thread_blocked, id=93]
0x00e99fa8 JavaThread "CompilerThread0" daemon [_thread_blocked, id=92]
0x00e99188 JavaThread "AdapterThread" daemon [_thread_blocked, id=91]
0x00e98300 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=90]
0x00e97768 JavaThread "Surrogate Locker Thread (CMS)" daemon [_thread_blocked, id=89]
0x00e8c668 JavaThread "Finalizer" daemon [_thread_blocked, id=88]
0x00e8af78 JavaThread "Reference Handler" daemon [_thread_blocked, id=87]

Other Threads:
0x00e88690 VMThread [id=86]
0x00e9d030 WatcherThread [id=95]

VM state:at safepoint (normal execution)

VM Mutex/Monitor currently owned by a thread: ([mutex/lock_event])
[0x00036a48/0x00036a78] Safepoint_lock - owner thread: 0x00e88690
[0x00036b68/0x00036b98] Threads_lock - owner thread: 0x00e88690
[0x00032ab8/0x00037078] Heap_lock - owner thread: 0x01871958

par new generation total 98240K, used 84760K [0x83800000, 0x89800000, 0x89800000)
eden space 98176K, 86% used [0x83800000, 0x88ac6398, 0x897e0000)
from space 64K, 0% used [0x897f0000, 0x897f0000, 0x89800000)
to space 64K, 0% used [0x897e0000, 0x897e0000, 0x897f0000)
concurrent mark-sweep generation total 1474560K, used 150985K [0x89800000, 0xe3800000, 0xe3800000)
concurrent-mark-sweep perm gen total 83128K, used 49825K [0xe3800000, 0xe892e000, 0xf3800000)

Dynamic libraries:
0x00010000 /td01/Mercury_Interactive/Quality_Center/java/bin/java
0xff2d0000 /usr/lib/secure/
0xff1d0000 /usr/lib/
0xff2fc000 /usr/lib/
0xff100000 /usr/lib/
0xff2f0000 /usr/platform/SUNW,T5240/lib/
0xfe800000 /td01/Mercury_Interactive/Quality_Center/java/jre/lib/sparc/server/
0xff0e0000 /usr/lib/
0xff0c0000 /usr/lib/
0xff090000 /usr/lib/
0xfe7b0000 /usr/lib/
0xfe700000 /usr/lib/
0xff070000 /usr/lib/
0xfe6e0000 /td01/Mercury_Interactive/Quality_Center/java/jre/lib/sparc/native_threads/
0xfe6b0000 /td01/Mercury_Interactive/Quality_Center/java/jre/lib/sparc/
0xfe670000 /td01/Mercury_Interactive/Quality_Center/java/jre/lib/sparc/
0xfe650000 /td01/Mercury_Interactive/Quality_Center/java/jre/lib/sparc/
0xfe240000 /usr/lib/locale/en_US.ISO8859-15/
0xfb790000 /td01/Mercury_Interactive/Quality_Center/java/jre/lib/sparc/
0xfb6e0000 /td01/Mercury_Interactive/Quality_Center/java/jre/lib/sparc/
0xf7790000 /td01/Mercury_Interactive/Quality_Center/java/jre/lib/sparc/
0xf76e0000 /usr/lib/
0xf76c0000 /usr/lib/
0xf76a0000 /usr/lib/
0xf75e0000 /usr/lib/

VM Arguments:
jvm_args: -Xms1536m -Xmx1536m -XX:MaxPermSize=256m -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+PrintGCTimeStamps -XX:+HeapDumpOnOutOfMemoryError -Xloggc:/td01/Mercury_Interactive/Quality_Center/jboss/GCDetails.txt -XX:+PrintGCDetails -XX:HeapDumpPath=/td01/Mercury_Interactive/Quality_Center/jboss/heapdump -Djava.endorsed.dirs=/td01/Mercury_Interactive/Quality_Center/jboss/lib/endorsed
java_command: org.jboss.Main
Launcher Type: SUN_STANDARD

Environment Variables:

Signal Handlers:
SIGSEGV: [], sa_mask[0]=0x7fbffeff, sa_flags=0x00000004
SIGBUS: [], sa_mask[0]=0x7fbffeff, sa_flags=0x00000004
SIGFPE: [], sa_mask[0]=0x7fbffeff, sa_flags=0x0000000c
SIGPIPE: [], sa_mask[0]=0x7fbffeff, sa_flags=0x0000000c
SIGILL: [], sa_mask[0]=0x7fbffeff, sa_flags=0x0000000c
SIGUSR1: [], sa_mask[0]=0x00000000, sa_flags=0x00000008
SIGUSR2: [], sa_mask[0]=0x7fbffeff, sa_flags=0x0000000c
SIGHUP: [], sa_mask[0]=0x7fbffeff, sa_flags=0x00000004
SIGINT: SIG_IGN, sa_mask[0]=0x00000000, sa_flags=0x00000000
SIGQUIT: [], sa_mask[0]=0x7fbffeff, sa_flags=0x00000004
SIGTERM: [], sa_mask[0]=0x7fbffeff, sa_flags=0x00000004

--------------- S Y S T E M ---------------

OS: Solaris 9 9/04 s9s_u7wos_09 SPARC
Copyright 2004 Sun Microsystems, Inc. All Rights Reserved.
Use is subject to license terms.
Assembled 29 June 2004

uname:SunOS 5.9 Generic_Virtual sun4u (T2 libthread)
rlimit: STACK 8192k, CORE infinity, NOFILE 1024, AS infinity
load average:2.37 3.28 3.27

CPU:total 128 has_v8, has_v9, has_vis1, has_vis2, is_ultra3, is_sun4v, is_niagara1

Memory: 8k page, physical 66879488k(57508904k free)

vm_info: Java HotSpot(TM) Server VM (1.5.0_20-b02) for solaris-sparc, built on Jun 18 2009 11:46:08 by unknown with unknown Workshop:0x550

time: Wed Oct 3 11:18:29 2012
elapsed time: 511743 seconds

We've been able to obtain a java thread dump before the system was crashed.
No deadlocks found.
However we found out that there are almost 90% of threads are blocked and about 10% of threads are IN_NATIVE state trying to read/write from the socket.

Here is also GC Details:_

8355.870: [GC 8355.870: [ParNew: 98176K->0K(98240K), 0.1164923 secs] 199196K->114491K(1572800K), 0.1171990 secs]
8393.664: [GC 8393.664: [ParNew: 98176K->0K(98240K), 0.2729720 secs] 212667K->124356K(1572800K), 0.2737070 secs]
8401.385: [Full GC 8401.385: [CMS: 124356K->101787K(1474560K), 3.6843095 secs] 130753K->101787K(1572800K), [CMS Perm : 47315K->47315K(78968K)], 3.6853905 secs]
8443.244: [GC 8443.244: [ParNew: 98152K->0K(98240K), 0.1478550 secs] 199939K->109563K(1572800K), 0.1485699 secs]
8464.668: [GC 8464.668: [ParNew: 98157K->0K(98240K), 0.0728873 secs] 207721K->112895K(1572800K), 0.0735619 secs]
8465.074: [Full GC 8465.074: [CMS: 112895K->100601K(1474560K), 3.5895005 secs] 117571K->100601K(1572800K), [CMS Perm : 47319K->47315K(78968K)], 3.5905628 secs]
8486.148: [GC 8486.148: [ParNew: 98176K->0K(98240K), 0.0749352 secs] 198777K->102554K(1572800K), 0.0756663 secs]
8488.338: [GC 8488.339: [ParNew: 98176K->0K(98240K), 0.0485796 secs] 200730K->104867K(1572800K), 0.0492671 secs]
8489.852: [GC 8489.852: [ParNew: 97355K->0K(98240K), 0.0970597 secs] 202223K->113739K(1572800K), 0.0978220 secs]
8528.320: [GC 8528.321: [ParNew: 98127K->0K(98240K), 0.2305933 secs] 211867K->119583K(1572800K), 0.2312862 secs]
8528.673: [Full GC 8528.674: [CMS: 119583K->99402K(1474560K), 3.8037133 secs] 119976K->99402K(1572800K), [CMS Perm : 47316K->47316K(78968K)], 3.8047208 secs]
8574.567: [GC 8574.567: [ParNew: 97721K->0K(98240K), 0.0788791 secs] 197124K->105527K(1572800K), 0.0795659 secs]
8592.816: [Full GC 8592.821: [CMS: 105527K->99590K(1474560K), 6.3920388 secs] 151624K->99590K(1572800K), [CMS Perm : 47320K->47316K(78968K)], 6.3976484 secs]
8619.543: [GC 8619.544: [ParNew: 98176K->0K(98240K), 1.4033638 secs] 197766K->118382K(1572800K), 1.4043006 secs]
8638.976: [GC 8638.976: [ParNew: 98163K->0K(98240K), 0.7486280 secs] 216545K->133435K(1572800K), 0.7493760 secs]
8659.225: [Full GC 8659.225: [CMS: 133435K->127207K(1474560K), 6.5614305 secs] 202428K->127207K(1572800K), [CMS Perm : 47327K->47319K(78968K)], 6.5624836 secs]
8692.489: [GC 8692.489: [ParNew: 98168K->0K(98240K), 0.4273621 secs] 225376K->129320K(1572800K), 0.4283057 secs]
8725.804: [Full GC 8725.804: [CMS: 129320K->125641K(1474560K), 6.7880081 secs] 185202K->125641K(1572800K), [CMS Perm : 47320K->47320K(78968K)], 6.7891931 secs]
8775.026: [GC 8775.027: [ParNew: 98134K->0K(98240K), 0.3575806 secs] 223775K->128092K(1572800K), 0.3582847 secs]
8792.604: [Full GC 8792.605: [CMS: 128092K->126040K(1474560K), 4.9224131 secs] 163371K->126040K(1572800K), [CMS Perm : 47320K->47320K(78968K)], 4.9234914 secs]
8857.535: [Full GC 8857.535: [CMS: 126040K->92630K(1474560K), 3.8167362 secs] 142517K->92630K(1572800K), [CMS Perm : 47320K->47320K(78968K)], 3.8178372 secs]
8921.375: [Full GC 8921.377: [CMS: 92630K->93971K(1474560K), 6.0901600 secs] 181462K->93971K(1572800K), [CMS Perm : 47323K->47320K(78968K)], 6.0932058 secs]
8987.475: [Full GC 8987.475: [CMS: 93971K->91872K(1474560K), 3.4701208 secs] 156871K->91872K(1572800K), [CMS Perm : 47320K->47320K(78968K)], 3.4711992 secs]
9050.955: [Full GC 9050.955: [CMS: 91872K->91416K(1474560K), 3.4062799 secs] 114665K->91416K(1572800K), [CMS Perm : 47321K->47320K(78968K)], 3.4073394 secs]

Notice that upgrading to JDK 6 is not an option.

Any help would be really appreciated.
Let me know if any additional information is required.

Thanks in advance,
Vadim Lotar.
Locked Post
New comments cannot be posted to this locked post.
Post Details
Locked on Nov 23 2012
Added on Oct 11 2012
1 comment