Java HotSpot Virtual Machine

Hi,

I am seeing JVM crashing in a rmi thread when its marshalling an object. Here is the information from the log file:

#
# An unexpected error has been detected by HotSpot Virtual Machine:
#
# SIGBUS (0xa) at pc=0xf6c0a9dc, pid=7187, tid=2090
#
# Java VM: Java HotSpot(TM) Server VM (1.5.0_04-b05 mixed mode)
# Problematic frame:
# j sun.rmi.server.MarshalOutputStream$1.run()Ljava/lang/Object;+0
#

--------------- T H R E A D ---------------

Current thread (0x1da0f580): JavaThread "RMI TCP Connection(904)-10.1.11.20" daemon [_thread_in_Java, id=2090]

siginfo:si_signo=10, si_errno=12, si_code=3, si_addr=0xb19f9d90

Registers:
O0=0x000029c1 O1=0xb19ffe68 O2=0xb19ffe6c O3=0xff292a10
O4=0xfe147f8c O5=0x00640f50 O6=0xb19ffd90 O7=0xfe13f128
G1=0xffffff80 G2=0x1da0f580 G3=0xffffa000 G4=0xb19ffe70
G5=0xb5878d40 G6=0xfe7d25bc G7=0xb1a01d78 Y=0x00000000
PC=0xf6c0a9dc nPC=0xf6c0a9e0


Top of Stack: (sp=0xb19ffd90)
0xb19ffd90: b19ffdf4 b5878d30 b5878d40 b19ffe6c
0xb19ffda0: b19ffdf8 b1a01d78 b5878f30 fe7d1d18
0xb19ffdb0: 00000000 fe7d0d80 00000000 00002000
0xb19ffdc0: 000078c8 b19ffe10 b19ffe10 f6c00218
0xb19ffdd0: 00000000 00000000 00000000 00000000
0xb19ffde0: 00000000 00000000 00000000 00000000
0xb19ffdf0: 00000000 00000000 00000000 00000000
0xb19ffe00: 00000000 00000000 00000000 00000000

Instructions: (pc=0xf6c0a9dc)
0xf6c0a9cc: c0 23 80 03 07 3f ff f0 c0 23 80 03 07 3f ff e8
0xf6c0a9dc: c0 23 80 03 c0 20 a1 00 1b 3f 9f 8f da 03 60 5c

Stack: [0xb1982000,0xb1a02000), sp=0xb19ffd90, free space=503k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
j sun.rmi.server.MarshalOutputStream$1.run()Ljava/lang/Object;+0
v ~StubRoutines::call_stub
V [libjvm.so+0x1924c8]
V [libjvm.so+0x220964]
C [libjava.so+0x9be4] Java_java_security_AccessController_doPrivileged__Ljava_security_PrivilegedAction_2+0x1c
j java.security.AccessController.doPrivileged(Ljava/security/PrivilegedAction;)Ljava/lang/Object;+4109746
j java.security.AccessController.doPrivileged(Ljava/security/PrivilegedAction;)Ljava/lang/Object;+0
j sun.rmi.server.MarshalOutputStream.<init>(Ljava/io/OutputStream;I)V+18
j sun.rmi.server.MarshalOutputStream.<init>(Ljava/io/OutputStream;)V+3
j sun.rmi.transport.ConnectionOutputStream.<init>(Lsun/rmi/transport/Connection;Z)V+7
j sun.rmi.transport.StreamRemoteCall.getOutputStream(Z)Ljava/io/ObjectOutput;+28
j sun.rmi.transport.StreamRemoteCall.getOutputStream()Ljava/io/ObjectOutput;+2
j sun.rmi.transport.StreamRemoteCall.<init>(Lsun/rmi/transport/Connection;Ljava/rmi/server/ObjID;IJ)V+55
j sun.rmi.server.UnicastRef.newCall(Ljava/rmi/server/RemoteObject;[Ljava/rmi/server/Operation;IJ)Ljava/rmi/server/RemoteCall;+72
j sun.rmi.transport.DGCImpl_Stub.dirty([Ljava/rmi/server/ObjID;JLjava/rmi/dgc/Lease;)Ljava/rmi/dgc/Lease;+12
j sun.rmi.transport.DGCClient$EndpointEntry.makeDirtyCall(Ljava/util/Set;J)V+43
j sun.rmi.transport.DGCClient$EndpointEntry.registerRefs(Ljava/util/List;)Z+188
j sun.rmi.transport.DGCClient.registerRefs(Lsun/rmi/transport/Endpoint;Ljava/util/List;)V+7
j sun.rmi.transport.ConnectionInputStream.registerRefs()V+74
j sun.rmi.transport.StreamRemoteCall.releaseInputStream()V+22
j sun.rmi.server.UnicastServerRef.dispatch(Ljava/rmi/Remote;Ljava/rmi/server/RemoteCall;)V+194
j sun.rmi.transport.Transport$1.run()Ljava/lang/Object;+23
v ~StubRoutines::call_stub
V [libjvm.so+0x1924c8]
V [libjvm.so+0x220964]
C [libjava.so+0xee44] Java_java_security_AccessController_doPrivileged__Ljava_security_PrivilegedExceptionAction_2Ljava_security_AccessControlContext_2+0x1c
j java.security.AccessController.doPrivileged(Ljava/security/PrivilegedExceptionAction;Ljava/security/AccessControlContext;)Ljava/lang/Object;+658582
j java.security.AccessController.doPrivileged(Ljava/security/PrivilegedExceptionAction;Ljava/security/AccessControlContext;)Ljava/lang/Object;+0
j sun.rmi.transport.Transport.serviceCall(Ljava/rmi/server/RemoteCall;)Z+163
v ~C2IAdapter
J sun.rmi.transport.tcp.TCPTransport.handleMessages(Lsun/rmi/transport/Connection;Z)V
v ~I2CAdapter
j sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run()V+685
j java.lang.Thread.run()V+11
v ~StubRoutines::call_stub
V [libjvm.so+0x1924c8]
V [libjvm.so+0x2babfc]
V [libjvm.so+0x2d9d30]
V [libjvm.so+0x2d58cc]
V [libjvm.so+0x64a5bc]


--------------- P R O C E S S ---------------

Java Threads: ( => current thread )
=>0x1da0f580 JavaThread "RMI TCP Connection(904)-10.1.11.20" daemon [_thread_in_Java, id=2090]
0x1dcdfa40 JavaThread "RMI ConnectionExpiration-[10.1.11.2:3010]" daemon [_thread_blocked, id=2089]
0x1c319c20 JavaThread "RMI TCP Connection(903)-10.1.11.2" daemon [_thread_in_native, id=2088]
0x00f34020 JavaThread "RMI RenewClean-[10.1.11.2:51153]" daemon [_thread_in_native, id=1035]
0x007e4830 JavaThread "TrapSeqUpdater" daemon [_thread_blocked, id=130]
0x00851b30 JavaThread "Session(): sending thread" daemon [_thread_blocked, id=129]
0x007a75f0 JavaThread "UDP Transport receiving thread" daemon [_thread_in_native, id=128]
0x00336c08 JavaThread "DcDispatcher-3" daemon [_thread_blocked, id=126]
0x00249e50 JavaThread "RMI RenewClean-[10.1.11.20:1200]" daemon [_thread_blocked, id=122]
0x006858f0 JavaThread "pool-1-thread-10" [_thread_blocked, id=116]
0x003ee418 JavaThread "pool-1-thread-9" [_thread_blocked, id=113]
0x00b3a840 JavaThread "pool-1-thread-8" [_thread_blocked, id=111]
0x007a4600 JavaThread "pool-1-thread-7" [_thread_blocked, id=108]
0x00513cc8 JavaThread "pool-1-thread-6" [_thread_blocked, id=106]
0x003ee218 JavaThread "pool-1-thread-5" [_thread_blocked, id=102]
0x003f1ba8 JavaThread "pool-1-thread-4" [_thread_blocked, id=101]
0x007eb1a8 JavaThread "pool-1-thread-3" [_thread_blocked, id=98]
0x007a47c0 JavaThread "DcDispatcher-2" daemon [_thread_blocked, id=97]
0x00708ca0 JavaThread "RMI RenewClean-[10.1.11.2:50779]" daemon [_thread_blocked, id=93]
0x00f29b30 JavaThread "RMI-AccessManager-getActiveUserSession" daemon [_thread_in_native, id=92]
0x00505b30 JavaThread "pool-1-thread-2" [_thread_blocked, id=88]
0x00037fb0 JavaThread "DestroyJavaVM" [_thread_blocked, id=1]
0x017a67c8 JavaThread "pool-1-thread-1" [_thread_blocked, id=87]
0x017a5e08 JavaThread "DcDispatcher-1" [_thread_blocked, id=86]
0x007081d8 JavaThread "com.hh.zm.general.datacache.util.RemoteListenerImpl" [_thread_blocked, id=85]
0x00e34090 JavaThread "TrapProcessor-1" [_thread_blocked, id=84]
0x00b445a8 JavaThread "SocketReader" [_thread_in_native, id=83]
0x004e1310 JavaThread "NodeOperStateSetter" [_thread_blocked, id=82]
0x00709cd0 JavaThread "Session(): sending thread" daemon [_thread_blocked, id=81]
0x00867638 JavaThread "UDP Transport receiving thread" daemon [_thread_in_native, id=80]
0x003442c0 JavaThread "AuditLog" [_thread_blocked, id=79]
0x00484788 JavaThread "EventHandler" [_thread_blocked, id=78]
0x00b43fb8 JavaThread "EventManagerImpl" [_thread_blocked, id=77]
0x004e1150 JavaThread "FtpServer" [_thread_in_native, id=76]
0x00a19218 JavaThread "JMSServer" [_thread_blocked, id=75]
0x0049c0e0 JavaThread "RMI LeaseChecker" daemon [_thread_blocked, id=47]
0x007097d8 JavaThread "RMI RenewClean-[10.1.11.2:3010]" daemon [_thread_blocked, id=45]
0x00280710 JavaThread "com.hh.zm.server.util.JWNaming" [_thread_in_native, id=43]
0x0049f370 JavaThread "GC Daemon" daemon [_thread_blocked, id=42]
0x0049e670 JavaThread "RMI Reaper" [_thread_blocked, id=41]
0x0031c938 JavaThread "Timer-1" daemon [_thread_blocked, id=40]
0x00516e70 JavaThread "RMI TCP Accept-0" daemon [_thread_in_native, id=39]
0x00219cc8 JavaThread "SeedGenerator Thread" daemon [_thread_blocked, id=17]
0x002505f0 JavaThread "Timer-0" daemon [_thread_blocked, id=16]
0x0028d1b8 JavaThread "Thread-0" daemon [_thread_blocked, id=15]
0x001488b8 JavaThread "Low Memory Detector" daemon [_thread_blocked, id=13]
0x00147810 JavaThread "CompilerThread1" daemon [_thread_blocked, id=12]
0x001469a8 JavaThread "CompilerThread0" daemon [_thread_blocked, id=11]
0x00145718 JavaThread "AdapterThread" daemon [_thread_blocked, id=10]
0x0013cd48 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=9]
0x0013ae18 JavaThread "Finalizer" daemon [_thread_blocked, id=8]
0x00138ed0 JavaThread "Reference Handler" daemon [_thread_blocked, id=7]

Other Threads:
0x001352a8 VMThread [id=6]
0x0014a340 WatcherThread [id=14]

VM state:not at safepoint (normal execution)

VM Mutex/Monitor currently owned by a thread: None

Heap
PSYoungGen total 283072K, used 78803K [0xe2160000, 0xf4a40000, 0xf6800000)
eden space 262336K, 30% used [0xe2160000,0xe6e54cd8,0xf2190000)
from space 20736K, 0% used [0xf35e0000,0xf35e0000,0xf4a20000)
to space 20800K, 0% used [0xf2190000,0xf2190000,0xf35e0000)
PSOldGen total 174784K, used 33532K [0xb9400000, 0xc3eb0000, 0xe2160000)
object space 174784K, 19% used [0xb9400000,0xbb4bf340,0xc3eb0000)
PSPermGen total 16384K, used 16138K [0xb5400000, 0xb6400000, 0xb9400000)
object space 16384K, 98% used [0xb5400000,0xb63c29a8,0xb6400000)

Dynamic libraries:
0x00010000 /home/shark/pg/bin/../jre/bin/java
0xff350000 /usr/lib/libthread.so.1
0xff390000 /usr/lib/libdl.so.1
0xff200000 /usr/lib/libc.so.1
0xff340000 /usr/platform/SUNW,Sun-Blade-1000/lib/libc_psr.so.1
0xfe000000 /home/shark/pg/jre/lib/sparc/server/libjvm.so
0xff2e0000 /usr/lib/libsocket.so.1
0xff1e0000 /usr/lib/libsched.so.1
0xff1b0000 /usr/lib/libCrun.so.1
0xff180000 /usr/lib/libm.so.1
0xff080000 /usr/lib/libnsl.so.1
0xff300000 /usr/lib/libw.so.1
0xff160000 /usr/lib/libmp.so.2
0xff060000 /home/shark/pg/jre/lib/sparc/native_threads/libhpi.so
0xfdfd0000 /home/shark/pg/jre/lib/sparc/libverify.so
0xfdf90000 /home/shark/pg/jre/lib/sparc/libjava.so
0xff020000 /home/shark/pg/jre/lib/sparc/libzip.so
0xfdea0000 /usr/lib/locale/en_US.ISO8859-1/en_US.ISO8859-1.so.2
0xfb850000 /home/shark/pg/jre/lib/sparc/libnet.so
0xfafe0000 /home/shark/pg/jre/lib/sparc/librmi.so
0xb2d80000 /home/shark/pg/jre/lib/sparc/libawt.so
0xb2c00000 /home/shark/pg/jre/lib/sparc/libmlib_image.so
0xf6880000 /home/shark/pg/jre/lib/sparc/motif21/libmawt.so
0xb2980000 /usr/dt/lib/libXm.so.4
0xfafc0000 /usr/openwin/lib/libXp.so.1
0xfad10000 /usr/openwin/lib/libXt.so.4
0xfac50000 /usr/openwin/lib/libXext.so.0
0xfafa0000 /usr/openwin/lib/libXtst.so.1
0xb3100000 /usr/openwin/lib/libX11.so.4
0xfac30000 /usr/openwin/lib/libSM.so.6
0xf6bd0000 /usr/openwin/lib/libICE.so.6
0xf6ba0000 /usr/openwin/lib/libdga.so.1

VM Arguments:
jvm_args: -Xmx1024000000 -Xms268435456 -XX:CodeCacheMinimumFreeSpace=2M -XX:ReservedCodeCacheSize=64M
java_command: com.zerog.lax.LAX /home/shark/pg/./bin/pgserver.lax /tmp/env.properties.7187

Environment Variables:
JAVA_HOME=/home/shark/pg/bin/../jre
CLASSPATH=/home/shark/pg/./bin/../lib/zmgeneral.jar:/home/shark/pg/./bin/../lib/zmserver.jar:/home/shark/pg/./bin/../lib/j2eeapi.jar:/home/shark/pg/./bin/../lib/xerces.jar:/home/shark/pg/./bin/../lib/log4j.jar:/home/shark/pg/./bin/../properties:/home/shark/pg/./bin/../lib/manager.jar:/home/shark/pg/./bin/../lib/MIBCompiler.jar:/home/shark/pg/./bin/../lib/mysql.jar:/home/shark/pg/./bin/../lib/jwsdp/relaxngDatatype.jar:/home/shark/pg/./bin/../lib/jwsdp/namespace.jar:/home/shark/pg/./bin/../lib/jwsdp/jaxb-libs.jar:/home/shark/pg/./bin/../lib/jwsdp/jaxb-impl.jar:/home/shark/pg/./bin/../lib/jwsdp/jaxb-api.jar:/home/shark/pg/./bin/lax.jar:/home/shark/./pg/bin/../properties:/home/shark/./pg/bin/../lib/zmgeneral.jar:/home/shark/./pg/bin/../lib/zmserver.jar:/home/shark/./pg/bin/../lib/log4j.jar:/home/shark/./pg/bin/../lib/j2eeapi.jar:/home/shark/./pg/bin/../lib/xerces.jar:/home/shark/./pg/bin/../lib/jbossall-client.jar:/home/shark/./pg/bin/lax.jar:
PATH=/usr/bin:/usr/bin:/usr/bin::/usr/dt/bin:/usr/openwin/bin:/bin:/usr/ucb:/usr/sbin:/usr/bin:/etc:/opt/Pegador/db/dbms/bin:/opt/mysql/bin:/home/oracle/OraHome1/bin:/shark/home/expect-5.43:/usr/sbin:/usr/bin:/etc:/opt/Pegador/db/dbms/bin:/opt/mysql/bin:/home/oracle/OraHome1/bin:/shark/home/expect-5.43
LD_LIBRARY_PATH=/home/shark/pg/jre/lib/sparc/server:/home/shark/pg/jre/lib/sparc:/home/shark/pg/jre/../lib/sparc
SHELL=/bin/tcsh
DISPLAY=cthai-dt1.headquarters.hh.com:0.0
HOSTTYPE=sun4
OSTYPE=solaris
MACHTYPE=sparc

Signal Handlers:
SIGSEGV: [libjvm.so+0x6c66dc], sa_mask[0]=0xffbffeff, sa_flags=0x00000004
SIGBUS: [libjvm.so+0x6c66dc], sa_mask[0]=0xffbffeff, sa_flags=0x00000004
SIGFPE: [libjvm.so+0x26def0], sa_mask[0]=0xffbffeff, sa_flags=0x0000000c
SIGPIPE: [libjvm.so+0x26def0], sa_mask[0]=0xffbffeff, sa_flags=0x0000000c
SIGILL: [libjvm.so+0x26def0], sa_mask[0]=0xffbffeff, sa_flags=0x0000000c
SIGUSR1: [libjvm.so+0x64c470], sa_mask[0]=0x00008000, sa_flags=0x00000008
SIGUSR2: [libjvm.so+0x26def0], sa_mask[0]=0xffbffeff, sa_flags=0x0000000c
SIGHUP: [libjvm.so+0x64b580], sa_mask[0]=0xffbffeff, sa_flags=0x00000004
SIGINT: [libjvm.so+0x64b580], sa_mask[0]=0xffbffeff, sa_flags=0x00000004
SIGQUIT: [libjvm.so+0x64b580], sa_mask[0]=0xffbffeff, sa_flags=0x00000004
SIGTERM: [libjvm.so+0x64b580], sa_mask[0]=0xffbffeff, sa_flags=0x00000004


--------------- S Y S T E M ---------------

OS: Solaris 8 1/01 s28s_u3wos_08 SPARC
Copyright 2000 Sun Microsystems, Inc. All Rights Reserved.
Assembled 28 November 2000

uname:SunOS 5.8 Generic_111433-02 sun4u (T1 libthread)
rlimit: STACK 8192k, CORE infinity, NOFILE 1024, AS infinity
load average:1.06 1.05 1.05

CPU:total 2 has_v8, has_v9, has_vis1, has_vis2, is_ultra3

Memory: 8k page, physical 2097152k(385536k free)

vm_info: Java HotSpot(TM) Server VM (1.5.0_04-b05) for solaris-sparc, built on Jun 3 2005 03:32:53 by unknown with unknown Workshop:0x550

Any help is appreciated.

Thanks,
Anand.