Java HotSpot Virtual Machine

'Fatal error' re libjvm.so or GCTaskThread or ... ?

I'm getting jvm crashes from a largish app and I could use any advise about what might be the source of the problem. One of logs is a attached below.

I think I've eliminate the obvious culprit:

(1) My app does not use the jni, nor do any of the supporting package.
(2) The problem occurs on two different machines (one with a Newhalem Lynnfield processor, the other an 8-core AWS ec2 instance), so defective hardware is not the problem.
(3) The app uses only about half the system's memory (and nothing else is running), so memory is not the problem (and running out of memory should just throw an exception, not crash the jvm).
(4) I've tried several builds of Java 1.6_18, including the most recent official release. A corrupted binary is not the problem.

The crash logs all identify the 'problematic frame' as libjvm.so, and the 'current thread' as GCTaskThread.

The app itself does computations in either 3 or 7 threads simultaneously (one less than the number of available cores). It usually runs 15 minutes to an hour before crashing. Each computation thread typically does it's job in around 5-10 minutes, so there is a thread creation and destruction going on, but not a particularly large amount.

The OS is Centos 5.

The app itself is fairly large and I have not been able to isolate the problem, so submitting a bug report to Sun won't get me anywhere. I really don't know what would cause this sort of problem and would appreciate any comments.

thanks

-----------------


# A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0x00002b44b1da83ac, pid=2705, tid=1081280832
#
# JRE version: 6.0_18-b05
# Java VM: Java HotSpot(TM) 64-Bit Server VM (16.0-b12 mixed mode linux-amd64 )
# Problematic frame:
# V [libjvm.so+0x6223ac]
#
# If you would like to submit a bug report, please visit:
# http://java.sun.com/webapps/bugreport/crash.jsp
#

--------------- T H R E A D ---------------

Current thread (0x00002aac751bd800): GCTaskThread [stack: 0x0000000040630000,0x0000000040731000] [id=2711]

siginfo:si_signo=SIGSEGV: si_errno=0, si_code=1 (SEGV_MAPERR), si_addr=0x000000000000001d

Registers:
RAX=0x0000000000000001, RBX=0x00002aaac8a0a920, RCX=0x0000000000000001, RDX=0x0000000000000005
RSP=0x000000004072fbd0, RBP=0x000000004072fc40, RSI=0x00002b44b2097530, RDI=0x0000000000000015
R8 =0x0000000000000001, R9 =0x0000000000004ce2, R10=0x00002aaac8a0a920, R11=0x00002aac7590f9c8
R12=0x00002aaac6769cc0, R13=0x0000000000000000, R14=0x00002aac45df6078, R15=0x00002aac7525edc0
RIP=0x00002b44b1da83ac, EFL=0x0000000000010246, CSGSFS=0x000000000000e033, ERR=0x0000000000000004
TRAPNO=0x000000000000000e

Top of Stack: (sp=0x000000004072fbd0)
0x000000004072fbd0: 00002aac751bec28 00002aac751bd800
0x000000004072fbe0: 0000000000000000 01002b44b0f183fe
...

Instructions: (pc=0x00002b44b1da83ac)
0x00002b44b1da839c: 8b 56 08 8b 4b 08 48 d3 e2 48 03 13 48 8d 7a 10
0x00002b44b1da83ac: 8b 5f 08 89 d8 c1 f8 03 85 db 0f 8e cc 05 00 00

Stack: [0x0000000040630000,0x0000000040731000], sp=0x000000004072fbd0, free space=3fe0000000000000018k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V [libjvm.so+0x6223ac]
V [libjvm.so+0x622e20]
...

--------------- P R O C E S S ---------------

Java Threads: ( => current thread )
0x00002aac84168000 JavaThread "Thread-36" [_thread_blocked, id=2768, stack(0x000000004143e000,0x000000004153f000)]
0x00002aac84167000 JavaThread "Thread-35" [_thread_blocked, id=2767, stack(0x0000000041640000,0x0000000041741000)]
...
Other Threads:
0x00002aac75260800 VMThread [stack: 0x0000000040a34000,0x0000000040b35000] [id=2715]
0x00002aac75296000 WatcherThread [stack: 0x000000004113b000,0x000000004123c000] [id=2722]

=>0x00002aac751bd800 (exited) GCTaskThread [stack: 0x0000000040630000,0x0000000040731000] [id=2711]

VM state:at safepoint (normal execution)

VM Mutex/Monitor currently owned by a thread: ([mutex/lock_event])
[0x0000000040110d80] Threads_lock - owner thread: 0x00002aac75260800
[0x0000000040111280] Heap_lock - owner thread: 0x00002aac84167000

Heap
PSYoungGen total 2411008K, used 2393801K [0x00002aabde4c0000, 0x00002aac73a10000, 0x00002aac73a10000)
eden space 2375360K, 100% used [0x00002aabde4c0000,0x00002aac6f470000,0x00002aac6f470000)
from space 35648K, 51% used [0x00002aac6f470000,0x00002aac70672460,0x00002aac71740000)
to space 35584K, 11% used [0x00002aac71750000,0x00002aac71b61550,0x00002aac73a10000)
PSOldGen total 417152K, used 365653K [0x00002aaab3a10000, 0x00002aaacd170000, 0x00002aabde4c0000)
object space 417152K, 87% used [0x00002aaab3a10000,0x00002aaac9f27580,0x00002aaacd170000)
PSPermGen total 21248K, used 16146K [0x00002aaaae610000, 0x00002aaaafad0000, 0x00002aaab3a10000)
object space 21248K, 75% used [0x00002aaaae610000,0x00002aaaaf5d4888,0x00002aaaafad0000)

Dynamic libraries:
40000000-40009000 r-xp 00000000 08:01 999548 /home/apps/jre1.6.0_18/bin/java
40108000-4010a000 rwxp 00008000 08:01 999548 /home/apps/jre1.6.0_18/bin/java
4010a000-4012b000 rwxp 4010a000 00:00 0 [heap]
4012b000-4012e000 ---p 4012b000 00:00 0
...

VM Arguments:
jvm_args: -XX:-UseGCOverheadLimit -Xmx7g
java_command: com.norbl.runjrun.RunnerNodeEc2
Launcher Type: SUN_STANDARD

Environment Variables:
JAVA_HOME=/usr/java/default
PATH=/usr/kerberos/sbin:/usr/kerberos/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/home/ec2/bin/:/root/bin
LD_LIBRARY_PATH=/home/apps/jre1.6.0_18/lib/amd64/server:/home/apps/jre1.6.0_18/lib/amd64:/home/apps/jre1.6.0_18/../lib/amd64
SHELL=/bin/bash

Signal Handlers:
SIGSEGV: [libjvm.so+0x70ef10], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGBUS: [libjvm.so+0x70ef10], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
...


--------------- S Y S T E M ---------------

OS:CentOS release 5.0 (Final)

uname:Linux 2.6.18-xenU-ec2-v1.0 #2 SMP Mon Feb 18 14:28:43 UTC 2008 x86_64
libc:glibc 2.5 NPTL 2.5
rlimit: STACK 8192k, CORE 0k, NPROC 57408, NOFILE 1024, AS infinity
load average:6.29 6.13 5.29

CPU:total 8 (4 cores per cpu, 1 threads per core) family 6 model 23 stepping 10, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3, sse4.1

Memory: 4k page, physical 7340172k(3785612k free), swap 0k(0k free)

vm_info: Java HotSpot(TM) 64-Bit Server VM (16.0-b12) for linux-amd64 JRE (1.6.0_18-ea-b05), built on Nov 18 2009 01:10:07 by "java_re" with gcc 3.2.2 (SuSE Linux)

time: Sun Feb 21 03:56:05 2010
elapsed time: 1891 seconds