Java EE (Java Enterprise Edition) General Discussion

Announcement

For appeals, questions and feedback about Oracle Forums, please email oracle-forums-moderators_us@oracle.com. Technical questions should be asked in the appropriate category. Thank you!

We are facing jvm hung issue. Please share your observations/root cause/solutions to resolve this is

3548348Nov 20 2017 — edited Nov 20 2017

We are facing jvm hung issue. Please share your observations/root cause/solutions to resolve this issue.

Observed that one thread in JVM utilizing CPU 99.9%.

top -H -p <jvm processId>

top - 13:12:59 up 119 days, 19:49, 1 user, load average: 1.02, 1.18, 1.28

Threads: 3363 total, 1 running, 3362 sleeping, 0 stopped, 0 zombie

%Cpu(s): 2.9 us, 0.3 sy, 0.0 ni, 96.8 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st

KiB Mem : 11741086+total, 51379164 free, 10710248 used, 55321448 buff/cache

KiB Swap: 0 total, 0 free, 0 used. 10082680+avail Mem

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND

12428 root 20 0 28.938g 1.531g 15620 R 99.9 1.4 73:40.88 java

12340 root 20 0 28.938g 1.531g 15620 S 0.0 1.4 0:00.00 java

12346 root 20 0 28.938g 1.531g 15620 S 0.0 1.4 0:02.07 java

If we take core dump for JVM process(when it was in hung state) and using gdb if we observe the stacktrace for high cpu usage thread showing as below.

we are sure that this is not application thread. Suspecting jvm deamon thread causing issue.

#0 0x00007f454a92647e in InstanceKlass::find_instance_method(Symbol*, Symbol*) ()

from /usr/java/jdk1.8.0_121/jre/lib/amd64/server/libjvm.so

#1 0x00007f454a7f6ed2 in ClassHierarchyWalker::find_witness_anywhere(Klass*, bool, bool) ()

from /usr/java/jdk1.8.0_121/jre/lib/amd64/server/libjvm.so

#2 0x00007f454a7fa416 in Dependencies::find_unique_concrete_method(Klass*, Method*) ()

from /usr/java/jdk1.8.0_121/jre/lib/amd64/server/libjvm.so

#3 0x00007f454a70dd78 in ciMethod::find_monomorphic_target(ciInstanceKlass*, ciInstanceKlass*, ciInstanceKlass*, bool) ()

from /usr/java/jdk1.8.0_121/jre/lib/amd64/server/libjvm.so

#4 0x00007f454a653c45 in GraphBuilder::invoke(Bytecodes::Code) () from /usr/java/jdk1.8.0_121/jre/lib/amd64/server/libjvm.so

#5 0x00007f454a6544fd in GraphBuilder::iterate_bytecodes_for_block(int) () from /usr/java/jdk1.8.0_121/jre/lib/amd64/server/libjvm.so

#6 0x00007f454a65608b in GraphBuilder::iterate_all_blocks(bool) () from /usr/java/jdk1.8.0_121/jre/lib/amd64/server/libjvm.so

#7 0x00007f454a656a36 in GraphBuilder::try_inline_full(ciMethod*, bool, Bytecodes::Code, Instruction*) ()

from /usr/java/jdk1.8.0_121/jre/lib/amd64/server/libjvm.so

#8 0x00007f454a652a3f in GraphBuilder::try_inline(ciMethod*, bool, Bytecodes::Code, Instruction*) ()

from /usr/java/jdk1.8.0_121/jre/lib/amd64/server/libjvm.so

#9 0x00007f454a653b82 in GraphBuilder::invoke(Bytecodes::Code) () from /usr/java/jdk1.8.0_121/jre/lib/amd64/server/libjvm.so

#10 0x00007f454a6544fd in GraphBuilder::iterate_bytecodes_for_block(int) () from /usr/java/jdk1.8.0_121/jre/lib/amd64/server/libjvm.so

#11 0x00007f454a656032 in GraphBuilder::iterate_all_blocks(bool) () from /usr/java/jdk1.8.0_121/jre/lib/amd64/server/libjvm.so

#12 0x00007f454a656a36 in GraphBuilder::try_inline_full(ciMethod*, bool, Bytecodes::Code, Instruction*) ()

from /usr/java/jdk1.8.0_121/jre/lib/amd64/server/libjvm.so

#13 0x00007f454a652a3f in GraphBuilder::try_inline(ciMethod*, bool, Bytecodes::Code, Instruction*) ()

from /usr/java/jdk1.8.0_121/jre/lib/amd64/server/libjvm.so

#14 0x00007f454a653b82 in GraphBuilder::invoke(Bytecodes::Code) () from /usr/java/jdk1.8.0_121/jre/lib/amd64/server/libjvm.so

#15 0x00007f454a6544fd in GraphBuilder::iterate_bytecodes_for_block(int) () from /usr/java/jdk1.8.0_121/jre/lib/amd64/server/libjvm.so

#16 0x00007f454a65608b in GraphBuilder::iterate_all_blocks(bool) () from /usr/java/jdk1.8.0_121/jre/lib/amd64/server/libjvm.so

#17 0x00007f454a656a36 in GraphBuilder::try_inline_full(ciMethod*, bool, Bytecodes::Code, Instruction*) ()

from /usr/java/jdk1.8.0_121/jre/lib/amd64/server/libjvm.so

#18 0x00007f454a652a3f in GraphBuilder::try_inline(ciMethod*, bool, Bytecodes::Code, Instruction*) ()

from /usr/java/jdk1.8.0_121/jre/lib/amd64/server/libjvm.so

#19 0x00007f454a653b82 in GraphBuilder::invoke(Bytecodes::Code) () from /usr/java/jdk1.8.0_121/jre/lib/amd64/server/libjvm.so

#20 0x00007f454a6544fd in GraphBuilder::iterate_bytecodes_for_block(int) () from /usr/java/jdk1.8.0_121/jre/lib/amd64/server/libjvm.so

#21 0x00007f454a656032 in GraphBuilder::iterate_all_blocks(bool) () from /usr/java/jdk1.8.0_121/jre/lib/amd64/server/libjvm.so

#22 0x00007f454a657c67 in GraphBuilder::GraphBuilder(Compilation*, IRScope*) () from /usr/java/jdk1.8.0_121/jre/lib/amd64/server/libjvm.so

#23 0x00007f454a65ff97 in IRScope::IRScope(Compilation*, IRScope*, int, ciMethod*, int, bool) ()

from /usr/java/jdk1.8.0_121/jre/lib/amd64/server/libjvm.so

#24 0x00007f454a6600af in IR::IR(Compilation*, ciMethod*, int) () from /usr/java/jdk1.8.0_121/jre/lib/amd64/server/libjvm.so

#25 0x00007f454a63e64b in Compilation::build_hir() () from /usr/java/jdk1.8.0_121/jre/lib/amd64/server/libjvm.so

#26 0x00007f454a63ea0e in Compilation::compile_java_method() () from /usr/java/jdk1.8.0_121/jre/lib/amd64/server/libjvm.so

---Type <return> to continue, or q <return> to quit---

#27 0x00007f454a63eb3e in Compilation::compile_method() () from /usr/java/jdk1.8.0_121/jre/lib/amd64/server/libjvm.so

#28 0x00007f454a63eeae in Compilation::Compilation(AbstractCompiler*, ciEnv*, ciMethod*, int, BufferBlob*) ()

from /usr/java/jdk1.8.0_121/jre/lib/amd64/server/libjvm.so

#29 0x00007f454a63fc59 in Compiler::compile_method(ciEnv*, ciMethod*, int) () from /usr/java/jdk1.8.0_121/jre/lib/amd64/server/libjvm.so

#30 0x00007f454a794eaa in CompileBroker::invoke_compiler_on_method(CompileTask*) ()

from /usr/java/jdk1.8.0_121/jre/lib/amd64/server/libjvm.so

#31 0x00007f454a795e56 in CompileBroker::compiler_thread_loop() () from /usr/java/jdk1.8.0_121/jre/lib/amd64/server/libjvm.so

#32 0x00007f454ad5a543 in JavaThread::thread_main_inner() () from /usr/java/jdk1.8.0_121/jre/lib/amd64/server/libjvm.so

#33 0x00007f454ad5a68c in JavaThread::run() () from /usr/java/jdk1.8.0_121/jre/lib/amd64/server/libjvm.so

#34 0x00007f454ac0b268 in java_start(Thread*) () from /usr/java/jdk1.8.0_121/jre/lib/amd64/server/libjvm.so

#35 0x00007f454babadc5 in start_thread () from /lib64/libpthread.so.0

#36 0x00007f454b3cdced in clone () from /lib64/libc.so.6

We are running jvm as -server flag with below details

Operating System

-----------------------

uname -a

Linux EGSL2C1 3.10.0-327.36.3.el7.x86_64 #1 SMP Thu Oct 20 04:56:07 EDT 2016 x86_64 x86_64 x86_64 GNU/Linux

[root@EGSL2C1 kodiak]# rpm -q glibc

glibc-2.17-106.el7_2.8.x86_64

[root@EGSL2C1 kodiak]# getconf GNU_LIBPTHREAD_VERSION

NPTL 2.17

Command-Line Options

------------------------

$JAVA_HOME/bin/java -server -XX:MaxJavaStackTraceDepth=10 -Xms4096m -Xmx4096m -XX:+UseG1GC -XX:+DisableExplicitGC -Xnoclassgc -verbose:gc

Environment Variables

------------------------

[root@EGSL2C1 kodiak]# echo $JAVA_HOME

/usr/java/jdk1.8.0_121

[root@EGSL2C1 kodiak]# echo $LD_ASSUME_KERNEL

[root@EGSL2C1 kodiak]# echo $_JAVA_SR_SIGNUM

[root@EGSL2C1 kodiak]# echo $LD_LIBRARY_PATH

/DG/activeRelease/lib:/opt/ACE_wrappers:/usr/local/BWSIfact/host/linux/lib/release:/usr/local/BWSIgoip/host_libs/lib:/opt/TimesTen/kodiak/lib

[root@EGSL2C1 kodiak]# echo $LD_PRELOAD

[root@EGSL2C1 kodiak]# echo $SHELL

/bin/bash

[root@EGSL2C1 kodiak]# echo $DISPLAY

[root@EGSL2C1 kodiak]# echo $HOSTTYPE

x86_64

[root@EGSL2C1 kodiak]# echo $OSTYPE

linux-gnu

[root@EGSL2C1 kodiak]# echo $ARCH

[root@EGSL2C1 kodiak]# echo $MACHTYPE

x86_64-redhat-linux-gnu

Added on Nov 20 2017

#consolidate-jcp, #java, #java-community-process-discussions, #linux

0 comments

1,049 views