I'm using Tomcat to run Jenkins. For the last few days, i'm noticing that Jenkins goes down anytime. I could not find anything in the Tomcat log. There are many articles about troubleshooting such issues and i went through many of them. One of them suggested to check JVM log for clues. I did see the log file but i'm not sure about what the exact cause is. Since i could not find any OutOfMemoryError so i'm not sure whether i need to add some memory to Tomcat. Please suggest.
Server version: Apache Tomcat/7.0.2
Server built: Aug 4 2010 12:23:47
Server number: 7.0.2.0
OS Name: Linux
OS Version: 3.14.20-20.44.amzn1.x86_64
Architecture: amd64
JVM Version: 1.7.0_65-mockbuild_2014_09_18_00_28-b00
JVM Vendor: Oracle Corporation
Here is the JVM log:
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0x00007f2d286512c5, pid=28177, tid=139831173089024
#
# JRE version: OpenJDK Runtime Environment (7.0_65-b17) (build 1.7.0_65-mockbuild_2014_09_18_00_28-b00)
# Java VM: OpenJDK 64-Bit Server VM (24.65-b04 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# C [libc.so.6+0x1482c5] __strlen_sse2_pminub+0x35
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# If you would like to submit a bug report, please include
# instructions on how to reproduce the bug and visit:
# http://icedtea.classpath.org/bugzilla
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#
--------------- P R O C E S S ---------------
Java Threads: ( => current thread )
0x00007f2c5ce92000 JavaThread "Computer.threadPoolForRemoting [#655]" daemon [_thread_blocked, id=16602, stack(0x00007f2cf82fd000,0x00007f2cf83fe000)]
0x00007f2c38003000 JavaThread "Computer.threadPoolForRemoting [#654]" daemon [_thread_blocked, id=16601, stack(0x00007f2ce90f1000,0x00007f2ce91f2000)]
0x00007f2c5cf7e800 JavaThread "Proc.executor [#969]" daemon [_thread_blocked, id=16599, stack(0x00007f2cfb1f2000,0x00007f2cfb2f3000)]
0x00007f2c5d77e800 JavaThread "/usr/bin/git -c core.askpass=true ls-remote -h git@git.XYZ.com:XYZ: stderr copier" [_thread_in_native, id=16598, stack(0x00007f2cf81fc000,0x00007f2cf82fd000)]
...
...
Heap
PSYoungGen total 678912K, used 390987K [0x00000007aaa80000, 0x00000007d5580000, 0x0000000800000000)
eden space 658432K, 57% used [0x00000007aaa80000,0x00000007c1e32e40,0x00000007d2d80000)
from space 20480K, 50% used [0x00000007d2d80000,0x00000007d37a0010,0x00000007d4180000)
to space 20480K, 0% used [0x00000007d4180000,0x00000007d4180000,0x00000007d5580000)
ParOldGen total 1398272K, used 185368K [0x0000000700000000, 0x0000000755580000, 0x00000007aaa80000)
object space 1398272K, 13% used [0x0000000700000000,0x000000070b506090,0x0000000755580000)
PSPermGen total 132608K, used 115513K [0x00000006c0000000, 0x00000006c8180000, 0x0000000700000000)
object space 132608K, 87% used [0x00000006c0000000,0x00000006c70ce758,0x00000006c8180000)
GC Heap History (10 events):
...
...
Event: 59356.217 GC heap before
{Heap before GC invocations=1240 (full 18):
PSYoungGen total 676864K, used 661824K [0x00000007aaa80000, 0x00000007d5580000, 0x0000000800000000)
eden space 655360K, 100% used [0x00000007aaa80000,0x00000007d2a80000,0x00000007d2a80000)
from space 21504K, 30% used [0x00000007d2a80000,0x00000007d30d0010,0x00000007d3f80000)
to space 20480K, 0% used [0x00000007d4180000,0x00000007d4180000,0x00000007d5580000)
ParOldGen total 1398272K, used 180244K [0x0000000700000000, 0x0000000755580000, 0x00000007aaa80000)
object space 1398272K, 12% used [0x0000000700000000,0x000000070b005080,0x0000000755580000)
PSPermGen total 132608K, used 115511K [0x00000006c0000000, 0x00000006c8180000, 0x0000000700000000)
object space 132608K, 87% used [0x00000006c0000000,0x00000006c70cdfa8,0x00000006c8180000)
Event: 59356.225 GC heap after
Heap after GC invocations=1240 (full 18):
PSYoungGen total 678912K, used 1024K [0x00000007aaa80000, 0x00000007d5580000, 0x0000000800000000)
eden space 658432K, 0% used [0x00000007aaa80000,0x00000007aaa80000,0x00000007d2d80000)
from space 20480K, 5% used [0x00000007d4180000,0x00000007d4280020,0x00000007d5580000)
to space 20480K, 0% used [0x00000007d2d80000,0x00000007d2d80000,0x00000007d4180000)
ParOldGen total 1398272K, used 185336K [0x0000000700000000, 0x0000000755580000, 0x00000007aaa80000)
object space 1398272K, 13% used [0x0000000700000000,0x000000070b4fe090,0x0000000755580000)
PSPermGen total 132608K, used 115511K [0x00000006c0000000, 0x00000006c8180000, 0x0000000700000000)
object space 132608K, 87% used [0x00000006c0000000,0x00000006c70cdfa8,0x00000006c8180000)
}
Event: 60382.794 GC heap before
{Heap before GC invocations=1241 (full 18):
PSYoungGen total 678912K, used 659456K [0x00000007aaa80000, 0x00000007d5580000, 0x0000000800000000)
eden space 658432K, 100% used [0x00000007aaa80000,0x00000007d2d80000,0x00000007d2d80000)
from space 20480K, 5% used [0x00000007d4180000,0x00000007d4280020,0x00000007d5580000)
to space 20480K, 0% used [0x00000007d2d80000,0x00000007d2d80000,0x00000007d4180000)
ParOldGen total 1398272K, used 185336K [0x0000000700000000, 0x0000000755580000, 0x00000007aaa80000)
object space 1398272K, 13% used [0x0000000700000000,0x000000070b4fe090,0x0000000755580000)
PSPermGen total 132608K, used 115511K [0x00000006c0000000, 0x00000006c8180000, 0x0000000700000000)
object space 132608K, 87% used [0x00000006c0000000,0x00000006c70cdfa8,0x00000006c8180000)
Event: 60382.802 GC heap after
Heap after GC invocations=1241 (full 18):
PSYoungGen total 678912K, used 10368K [0x00000007aaa80000, 0x00000007d5580000, 0x0000000800000000)
eden space 658432K, 0% used [0x00000007aaa80000,0x00000007aaa80000,0x00000007d2d80000)
from space 20480K, 50% used [0x00000007d2d80000,0x00000007d37a0010,0x00000007d4180000)
to space 20480K, 0% used [0x00000007d4180000,0x00000007d4180000,0x00000007d5580000)
ParOldGen total 1398272K, used 185368K [0x0000000700000000, 0x0000000755580000, 0x00000007aaa80000)
object space 1398272K, 13% used [0x0000000700000000,0x000000070b506090,0x0000000755580000)
PSPermGen total 132608K, used 115511K [0x00000006c0000000, 0x00000006c8180000, 0x0000000700000000)
object space 132608K, 87% used [0x00000006c0000000,0x00000006c70cdfa8,0x00000006c8180000)
}
Deoptimization events (10 events):
Event: 37770.210 Thread 0x00007f2cb01ab800 Uncommon trap: reason=unreached action=reinterpret pc=0x00007f2d1d9eea3c method=java.util.concurrent.locks.AbstractQueuedSynchronizer.unparkSuccessor(Ljava/util/concurrent/locks/AbstractQueuedSynchronizer$Node;)V @ 22
Event: 37793.304 Thread 0x00007f2c5cf22000 Uncommon trap: reason=bimorphic action=maybe_recompile pc=0x00007f2d1de867a8 method=java.util.AbstractSet.removeAll(Ljava/util/Collection;)Z @ 51
Event: 41148.828 Thread 0x00007f2c5cde6800 Uncommon trap: reason=unreached action=reinterpret pc=0x00007f2d1e557e20 method=org.apache.velocity.runtime.parser.node.ASTMethod.execute(Ljava/lang/Object;Lorg/apache/velocity/context/InternalContextAdapter;)Ljava/lang/Object; @ 124
...
...
Internal exceptions (10 events):
Event: 60200.167 Thread 0x00007f2c5d613800 Threw 0x00000007ccd571a8 at /builddir/build/BUILD/java-1.7.0-openjdk/openjdk/hotspot/src/share/vm/prims/jni.cpp:721
Event: 60200.167 Thread 0x00007f2c5d613800 Threw 0x00000007ccd5db60 at /builddir/build/BUILD/java-1.7.0-openjdk/openjdk/hotspot/src/share/vm/prims/jni.cpp:721
...
...
Events (10 events):
Event: 61096.236 Thread 0x00007f2c5c0b1000 Thread exited: 0x00007f2c5c0b1000
Event: 61096.237 Executing VM operation: RevokeBias
Event: 61096.238 Thread 0x00007f2c5c0b1000 Thread added: 0x00007f2c5c0b1000
...
...
Dynamic libraries:
00400000-00401000 r-xp 00000000 ca:01 136731 /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.65.x86_64/jre/bin/java
00600000-00601000 rw-p 00000000 ca:01 136731 /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.65.x86_64/jre/bin/java
006f0000-00732000 rw-p 00000000 00:00 0 [heap]
6c0000000-6c8180000 rw-p 00000000 00:00 0
6c8180000-6cbc00000 ---p 00000000 00:00 0
...
...
VM Arguments:
jvm_args: -Djava.util.logging.config.file=/home/rel-user/jen-tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Xms2048m -Xmx4096m -XX:MaxPermSize=1024m -Dfile.encoding=UTF8 -Dhudson.plugins.openid.impl.TeamsExtensiondisable=true -Djava.endorsed.dirs=/home/rel-user/jen-tomcat/endorsed -Dcatalina.base=/home/rel-user/jen-tomcat -Dcatalina.home=/home/rel-user/jen-tomcat -Djava.io.tmpdir=/home/rel-user/jen-tomcat/temp
java_command: org.apache.catalina.startup.Bootstrap start
Launcher Type: SUN_STANDARD
Environment Variables:
JAVA_HOME=/usr/lib/jvm/jre
PATH=/home/ec2-user/perl/5.18.2/bin:/home/ec2-user/perl/5.18.2/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/opt/aws/bin:/home/ec2-user/bin
SHELL=/bin/bash
--------------- S Y S T E M ---------------
OS:Linux
uname:Linux 3.14.20-20.44.amzn1.x86_64 #1 SMP Mon Oct 6 22:52:46 UTC 2014 x86_64
libc:glibc 2.17 NPTL 2.17
rlimit: STACK 8192k, CORE 0k, NPROC 120223, NOFILE 4096, AS infinity
load average:2.20 3.37 3.55
Memory: 4k page, physical 15404752k(1735452k free), swap 8191996k(8171656k free)
vm_info: OpenJDK 64-Bit Server VM (24.65-b04) for linux-amd64 JRE (1.7.0_65-b17), built on Sep 18 2014 00:37:14 by "mockbuild" with gcc 4.8.2 20140120 (Red Hat 4.8.2-16)
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
I've removed certain portions that i thought are not relevant. In case you want more details, i can furnish it.