Skip to Main Content

How to analyze JRockit dump

3c511242-6534-40ed-904a-946171a8b4a5Oct 30 2018 — edited Nov 1 2018

wcTask:Mon Oct 29 09:01:40 CST 2018

serverTask:Mon Oct 29 09:01:40 CST 2018

Clients:24

{"Fun":"02","CmdNo":515,"Data":"{}"}

{"Fun":"02","CmdNo":515,"Data":"{\"Status\":1,\"SessionID\":\"\",\"ErrorMessage\":\"\"}"}

[JRockit] ERROR: The JVM has crashed. Writing crash information to /bea11/user_projects/domains/zhdt/jrockit.11729.dump.

{"CmdNo":9,"Data":"{}","Fun":"02"}

===== BEGIN DUMP =============================================================

JRockit dump produced after 2 days, 18:56:15 on Mon Oct 29 09:01:40 2018

    **************************************************************************

    *  If you see this dump, please go to                                    *

    *  http://download.oracle.com/docs/cd/E15289_01/go2troubleshooting.html  *

    *  for troubleshooting information.                                      *

    **************************************************************************

Additional information is available in:

  /bea11/user_projects/domains/zhdt/jrockit.11729.dump

No snapshot file (core dump) will be created because core dumps have been

disabled. To enable core dumping, try "ulimit -c unlimited"

before starting JRockit again.

Error Message: Illegal memory access. [54]

Signal info  : si_signo=11, si_code=2 si_addr=0xa79ff000

Version      : Oracle JRockit(R) R28.2.0-79-146777-1.6.0_29-20111005-1807-linux-ia32

CPU          : AMD Opteron QC/Phenom SSE SSE2 SSE3 SSE4A AMD64

Number CPUs  : 32

Tot Phys Mem : 67594600448 (64463 MB)

OS version   : Red Hat Enterprise Linux Server release 5.5 (Tikanga)

Linux version 2.6.18-194.el5 (mockbuild@x86-005.build.bos.redhat.com) (gcc version 4.1.2 20080704 (Red Hat 4.1.2-48)) #1 SMP Tue Mar 16 21:52:39 EDT 2010 (x86_64)

Thread System: Linux NPTL

LibC release : 2.5-stable

Java locking : Lazy unlocking enabled (class banning) (transfer banning)

State        : JVM is running

Command Line : -Xms512m -Xmx1024m -Dweblogic.Name=zswyHMS -Djava.security.policy=/bea11/wlserver_10.3/server/lib/weblogic.policy -Dweblogic.ProductionModeEnabled=true -Dweblogic.security.SSL.trustedCAKeyStore=/bea11/wlserver_10.3/server/lib/cacerts -Djava.security.egd=file:/dev/./urandom -da -Dplatform.home=/bea11/wlserver_10.3 -Dwls.home=/bea11/wlserver_10.3/server -Dweblogic.home=/bea11/wlserver_10.3/server -Dweblogic.management.discover=false -Dweblogic.management.server=http://10.1.8.21:7004 -Dwlw.iterativeDev=false -Dwlw.testConsole=false -Dwlw.logErrorsToConsole=false -Dweblogic.ext.dirs=/bea11/patch_wls1036/profiles/default/sysext_manifest_classpath:/bea11/patch_ocp371/profiles/default/sysext_manifest_classpath -Dweblogic.management.username=zhdt -Dweblogic.management.password=opic2009 -Dsun.java.command=weblogic.Server -Dsun.java.launcher=SUN_STANDARD weblogic.Server

Repository   : /tmp/2018_10_26_14_05_32_11729

java.home    : /bea11/jrockit_160_29_D1.2.0-10/jre

j.class.path : /bea11/patch_wls1036/profiles/default/sys_manifest_classpath/weblogic_patch.jar:/bea11/patch_ocp371/profiles/default/sys_manifest_classpath/weblogic_patch.jar:/bea11/jrockit_160_29_D1.2.0-10/lib/tools.jar:/bea11/wlserver_10.3/server/lib/weblogic_sp.jar:/bea11/wlserver_10.3/server/lib/weblogic.jar:/bea11/modules/features/weblogic.server.modules_10.3.6.0.jar:/bea11/wlserver_10.3/server/lib/webservices.jar:/bea11/modules/org.apache.ant_1.7.1/lib/ant-all.jar:/bea11/modules/net.sf.antcontrib_1.1.0.0_1-0b2/lib/ant-contrib.jar:/bea11/wlserver_10.3/common/derby/lib/derbyclient.jar:/bea11/wlserver_10.3/server/lib/xqrl.jar

j.lib.path   : /usr/java/packages/lib/i386:/bea11/jrockit_160_29_D1.2.0-10/jre/lib/i386/jrockit:/bea11/jrockit_160_29_D1.2.0-10/jre/lib/i386:/bea11/jrockit_160_29_D1.2.0-10/jre/../lib/i386:/bea11/patch_wls1036/profiles/default/native:/bea11/patch_ocp371/profiles/default/native:/bea11/wlserver_10.3/server/native/linux/i686:/bea11/wlserver_10.3/server/native/linux/i686/oci920_8

JAVA_HOME    : /bea11/jrockit_160_29_D1.2.0-10

_JAVA_OPTIONS: <not set>

LD_LIBRARY_PATH: /bea11/jrockit_160_29_D1.2.0-10/jre/lib/i386/jrockit:/bea11/jrockit_160_29_D1.2.0-10/jre/lib/i386:/bea11/jrockit_160_29_D1.2.0-10/jre/../lib/i386:/bea11/patch_wls1036/profiles/default/native:/bea11/patch_ocp371/profiles/default/native:/bea11/wlserver_10.3/server/native/linux/i686:/bea11/wlserver_10.3/server/native/linux/i686/oci920_8

LD_ASSUME_KERNEL: <not set>

LD_PRELOAD   : <not set>

StackOverFlow: 0 StackOverFlowErrors have occured

OutOfMemory  : 0 OutOfMemoryErrors have occured

C Heap       : Good; no memory allocations have failed

GC Strategy  : Mode: throughput, with strategy: genparpar (basic strategy: genparpar)

GC Status    : OC is not running. Last finished OC was OC#28.

             : YC is not running. Last finished YC was YC#5112.

YC Promotion : Last YC successfully promoted all objects

YC History   : Ran 189 YCs before OC#24.

             : Ran 169 YCs before OC#25.

             : Ran 0 YCs before OC#26.

             : Ran 177 YCs before OC#27.

             : Ran 88 YCs before OC#28.

             : Ran 3 YCs since last OC.

Heap         : 0xb66d3000 - 0xd66d3000  (Size: 512 MB)

Compaction   : (no compaction area)

Allocation   : TLA-min: 2048, TLA-preferred: 65536 TLA-waste limit: 2048

NurseryList  : 0xb6f53918 - 0xc79d2700

KeepArea     : 0xc0bc2250 - 0xc430bd80

KA Markers   : [ 0xc0bc2250,  0xc430bd80 , 0xc79d2700 ]

Forbidden A  : (none)

Previous KA  : 0xc430bd80 - 0xc79d2700

Previous FA  : (none)

CompRefs     : References are 32-bit.

Registers (from ThreadContext: 0xa7225a9c:

   eax = a79fef80    ecx = 00000002    edx = a79fef9c    ebx = 00000014

   esp = a7225d94    ebp = a7225e60    esi = a79feffd    edi = a7714316

    es = 0000002b     cs = a7220023     ss = bb86002b     ds = 0000002b

    fs = 00000003     gs = 00000063

   eip = 007428ec eflags = 00000216

Loaded modules:

(* denotes the module where the exception occured)

08048000-08057313  /bea11/jrockit_160_29_D1.2.0-10/bin/java

ffffe000-ffffe7c7  /bea11/jrockit_160_29_D1.2.0-10/bin/java

0085c000-0085dd0b  /lib/libdl.so.2

00819000-0082c33b  /lib/libpthread.so.0

006d1000-0080f9af */lib/libc.so.6

006b3000-006cc113  /lib/ld-linux.so.2

f7c7a000-f7f3da17  /bea11/jrockit_160_29_D1.2.0-10/jre/lib/i386/jrockit/libjvm.so

f7c5d000-f7c777cf  /bea11/jrockit_160_29_D1.2.0-10/jre/lib/i386/libjrosal.so

f7fbe000-f7fc8607  /bea11/jrockit_160_29_D1.2.0-10/jre/lib/i386/libjrutil.so

00833000-00857167  /lib/libm.so.6

00862000-0086848b  /lib/librt.so.1

f713c000-f71447cb  /lib/libnss_files.so.2

f714a000-f715686b  /bea11/jrockit_160_29_D1.2.0-10/jre/lib/i386/libjfr.so

b43a4000-b43ae9bb  /bea11/jrockit_160_29_D1.2.0-10/jre/lib/i386/libverify.so

b437f000-b43a1f07  /bea11/jrockit_160_29_D1.2.0-10/jre/lib/i386/libjava.so

0094e000-00960c1f  /lib/libnsl.so.1

b4318000-b431de50  /bea11/jrockit_160_29_D1.2.0-10/jre/lib/i386/native_threads/libhpi.so

b3c99000-b3ca76e4  /bea11/jrockit_160_29_D1.2.0-10/jre/lib/i386/libzip.so

b3b0a000-b3b1d150  /bea11/jrockit_160_29_D1.2.0-10/jre/lib/i386/libnet.so

b3b03000-b3b08013  /bea11/jrockit_160_29_D1.2.0-10/jre/lib/i386/liborii.so

b3142000-b31484e4  /bea11/jrockit_160_29_D1.2.0-10/jre/lib/i386/libnio.so

b2825000-b282aa06  /bea11/jrockit_160_29_D1.2.0-10/jre/lib/i386/libmanagement.so

b21ea000-b21f2f93  /bea11/jrockit_160_29_D1.2.0-10/jre/lib/i386/libjmapi.so

b29fe000-b2a002c3  /bea11/wlserver_10.3/server/native/linux/i686/libwlfileio3.so

a906f000-a90f34a6  /bea11/jrockit_160_29_D1.2.0-10/jre/lib/i386/libawt.so

b3b29000-b3b2d228  /bea11/jrockit_160_29_D1.2.0-10/jre/lib/i386/headless/libmawt.so

a8ea7000-a8f1f72f  /bea11/jrockit_160_29_D1.2.0-10/jre/lib/i386/libfontmanager.so

a86bd000-a86eac2f  /bea11/jrockit_160_29_D1.2.0-10/jre/lib/i386/libjpeg.so

Stack:

(* marks the word pointed to by the stack pointer)

a7225d94: f7ed5198* a771430a  a79feff1  00000014  00000008  a8c8987c 

a7225dac: 00000000  a7724010  0000002f  a8c8987c  00812ff4  a7714300 

a7225dc4: a7225df2  a7225e8c  0b8426fc  0b83b3a0  00000053  00000000 

a7225ddc: 00000053  a78282c8  a79fef9b  a79fef9a  a79fef9d  00000001 

Code:

(* marks the word pointed to by the instruction pointer)

007428bc: 90909090  0c24448b  10244439  49c2820f  90660007  0c244c8b 

007428d4: 7c8bf889  f2890424  0824748b  73e9d1fc  e9d1a401  a5660273 

007428ec: c789a5f3* 448bd689  90c30424  90909090  90909090  83e58955 

00742904: 74890cec  758b0424  241c8910  e8084d8b  fffa43c8  06dcc381 

Last optimized methods:

  #1249 weblogic/common/resourcepool/ResourcePoolImpl$ResourcePoolMaintanenceTask.timerExpired(Lweblogic/timers/Timer;)V  206737.272-206737.874 0xa8b17000-0xa8b18e0c 601.90 ms 2368KB 

  #1250 weblogic/socket/BaseAbstractMuxableSocket.getCompleteMessageTimeoutMillis()I  209908.554-209908.554 0xa8b18e20-0xa8b18e45 0.62 ms 256KB 

  #1251 weblogic/server/channels/BasicServerChannelImpl.getCompleteMessageTimeout()I  212868.545-212868.545 0xa8b18e60-0xa8b18e7f 0.59 ms 256KB 

  #1252 java/lang/ref/ReferenceQueue.poll()Ljava/lang/ref/Reference;  214394.569-214394.573 0xa8b18e80-0xa8b18fc9 4.02 ms 256KB 

  #1253 com/fasterxml/jackson/databind/ser/std/StringSerializer.serialize(Ljava/lang/Object;Lcom/fasterxml/jackson/core/JsonGenerator;Lcom/fasterxml/jackson/databind/SerializerProvider;)V  219664.947-219664.948 0xa8b18fe0-0xa8b19010 0.94 ms 128KB 

  #1254 weblogic/kernel/ThreadLocalStack$StackInitialValue.resetValue(Ljava/lang/Object;)Ljava/lang/Object;  224878.406-224878.408 0xa8b19020-0xa8b19092 1.83 ms 256KB 

  #1255 weblogic/utils/collections/Stack.clear()V  224878.408-224878.409 0xa8b190a0-0xa8b190f4 1.18 ms 128KB 

  #1256 weblogic/work/ServiceClassesStats.reset()V  224980.219-224980.220 0xa8b19100-0xa8b19197 1.45 ms 128KB 

  #1257 com/zswy/callsocket/CallSocketListener$4.run()V  235548.959-235548.959 0xa8b191a0-0xa8b191b5 0.59 ms 128KB 

  #1258 weblogic/servlet/internal/CharsetChunkOutput.<init>(Lweblogic/servlet/internal/ChunkOutput;Ljava/nio/charset/Charset;)V  239837.094-239837.130 0xa8b191c0-0xa8b19b83 36.29 ms 640KB 

  #1259 org/springframework/core/annotation/AnnotationUtils.findAnnotation(Ljava/lang/Class;Ljava/lang/Class;)Ljava/lang/annotation/Annotation;  240348.646-240348.659 0xa8b19ba0-0xa8b19e0e 12.75 ms 256KB 

  #1260 javax/naming/InitialContext.init(Ljava/util/Hashtable;)V  240526.983-240526.985 0xa8b19e20-0xa8b19ea8 1.68 ms 64KB 

  #1261 java/text/SimpleDateFormat.zeroPaddingNumber(IIILjava/lang/StringBuffer;)V  240631.554-240631.938 0xa8b19ec0-0xa8b1b7c2 384.26 ms 2240KB 

*#1262 java/util/concurrent/locks/ReentrantLock$Sync.isHeldExclusively()Z  240641.241-240641.242 0xa8b1b7e0-0xa8b1b803 0.74 ms 256KB 

  #1247 weblogic/diagnostics/instrumentation/EventQueue.timerExpired(Lweblogic/timers/Timer;)V  197182.701-197183.222 0xa8b14b60-0xa8b16c2f 520.63 ms 3072KB 

  #1248 com/sun/naming/internal/ResourceManager.getInitialEnvironment(Ljava/util/Hashtable;)Ljava/util/Hashtable;  198465.725-198465.738 0xa8b16c40-0xa8b16fee 12.87 ms 256KB 

Thread:

"Javelin Worker-0" id=1055 idx=0x284 tid=24534 lastJavaFrame=0xa7225ea8

Stack 0: start=0xa7205000, end=0xa7227000, guards=0xa720a000 (ok), forbidden=0xa7208000

Thread Stack Trace:

    at memcpy+28()@0x7428ec

    at RJNI_java_util_zip_ZipFile_getNextEntry+37(rnizipfile.c:151)@0xf7e98be6

    -- Java stack --

    at java/util/zip/ZipFile.getNextEntry(II)I(Native Method)

    at java/util/zip/ZipFile.access$500(ZipFile.java:36)

    at java/util/zip/ZipFile$2.nextElement(ZipFile.java:316)

    at java/util/zip/ZipFile$2.nextElement(ZipFile.java:304)

    at com/bea/core/repackaged/jdt/internal/compiler/batch/ClasspathJar.isPackage(ClasspathJar.java:154)

    at com/bea/core/repackaged/jdt/internal/compiler/batch/ClasspathJar.findClass(ClasspathJar.java:93)

    at com/bea/core/repackaged/jdt/internal/compiler/batch/FileSystem.findClass(FileSystem.java:242)

    at com/bea/core/repackaged/jdt/internal/compiler/batch/FileSystem.findType(FileSystem.java:334)

    at weblogic/utils/compiler/jdt/JSPFileSystem.findType(JSPFileSystem.java:30)

    at com/bea/core/repackaged/jdt/internal/compiler/lookup/LookupEnvironment.createPackage(LookupEnvironment.java:700)

    at com/bea/core/repackaged/jdt/internal/compiler/lookup/CompilationUnitScope.buildTypeBindings(CompilationUnitScope.java:89)

    at com/bea/core/repackaged/jdt/internal/compiler/lookup/LookupEnvironment.buildTypeBindings(LookupEnvironment.java:152)

    at com/bea/core/repackaged/jdt/internal/compiler/Compiler.internalBeginToCompile(Compiler.java:717)

    at com/bea/core/repackaged/jdt/internal/compiler/Compiler.beginToCompile(Compiler.java:377)

    at com/bea/core/repackaged/jdt/internal/compiler/Compiler.compile(Compiler.java:422)

    at weblogic/jsp/internal/java/JDTJavaCompiler.generateByteCode(JDTJavaCompiler.java:104)

    at weblogic/jsp/internal/java/JavaSourceFile._codeGen(JavaSourceFile.java:211)

    at weblogic/jsp/internal/java/JavaSourceFile.codeGen(JavaSourceFile.java:201)

    at weblogic/jsp/internal/ProxySourceFile.compileGeneratedFiles(ProxySourceFile.java:314)

    at weblogic/jsp/internal/ProxySourceFile.codeGen(ProxySourceFile.java:248)

    at weblogic/jsp/internal/SourceFile.codeGen(SourceFile.java:327)

    at weblogic/jsp/internal/client/ClientUtilsImpl$CodeGenJob.run(ClientUtilsImpl.java:620)

    at weblogic/jsp/internal/client/Job.performJob(Job.java:83)

    at weblogic/jsp/internal/client/ThreadPool$WorkerThread.run(ThreadPool.java:217)

    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)

    -- end of trace

Memory usage report:

Total mapped                  1423408KB           (reserved=544412KB)

-              Java heap      1048576KB           (reserved=524288KB)

-              GC tables        35084KB         

-          Thread stacks        28156KB           (#threads=128)

-          Compiled code        16064KB           (used=14761KB)

-               Internal          776KB         

-                     OS        83876KB         

-                  Other       106684KB         

-        Java class data       103168KB           (malloced=103141KB #141593 in 25389 classes)

                                                  Not tracing sites.

- Native memory tracking         1024KB           (malloced=137KB #8)

                                                  Not tracing sites.

Set the env variable TRACE_ALLOC_SITES=1 or use the print_memusage switch

trace_alloc_sites=true to enable alloc site tracing.

    **************************************************************************

    *  If you see this dump, please go to                                    *

    *  http://download.oracle.com/docs/cd/E15289_01/go2troubleshooting.html  *

    *  for troubleshooting information.                                      *

    **************************************************************************

===== END DUMP ===============================================================

/bea11/user_projects/domains/zhdt/bin/startWebLogic.sh: line 180: 11729 已放弃               ${JAVA_HOME}/bin/java ${JAVA_VM} ${MEM_ARGS} -Dweblogic.Name=${SERVER_NAME} -Djava.security.policy=${WL_HOME}/server/lib/weblogic.policy ${JAVA_OPTIONS} ${PROXY_SETTINGS} ${SERVER_CLASS}

Comments
Post Details
Added on Oct 30 2018
1 comment
719 views