Our Tomcat processes (in two different servers) which are usually very stable started crashing everyday since last week. Prior to last week, we never had any issue with this process and it used to be up and running for months at a time without any need to restart. We have not made any changes to our application which could be causing this. Our Unix team also has not applied any patches lately. We are puzzled that this is happening on two different servers. JVM is producing hs_err_pid log file prior to crash and it says out of memory error. We have enough memory available in our server and we are on 64-bit Solaris and using 64-bit version of Java. What's puzzling is that this issue started happening simultaneously in two different servers.
Following are the errors in different hs_err_pid files:
hs_err_pid13798.log:# Native memory allocation (malloc) failed to allocate 280 bytes for AllocateHeap
hs_err_pid2354.log:# Native memory allocation (malloc) failed to allocate 280 bytes for AllocateHeap
hs_err_pid23886.log:# Native memory allocation (malloc) failed to allocate 280 bytes for AllocateHeap
hs_err_pid4302.log:# Native memory allocation (malloc) failed to allocate 280 bytes for AllocateHeap
hs_err_pid3766.log:# Native memory allocation (malloc) failed to allocate 32744 bytes for ChunkPool::allocate
GC details from one of the log files:
Event: 72408.229 GC heap before
{Heap before GC invocations=281 (full 22):
PSYoungGen total 98304K, used 98303K [0x00000005d5700000, 0x00000005e0900000, 0x0000000800000000)
eden space 96768K, 100% used [0x00000005d5700000,0x00000005db580000,0x00000005db580000)
from space 1536K, 99% used [0x00000005e0780000,0x00000005e08fff70,0x00000005e0900000)
to space 2560K, 0% used [0x00000005e0400000,0x00000005e0400000,0x00000005e0680000)
ParOldGen total 848896K, used 17162K [0x0000000180400000, 0x00000001b4100000, 0x00000005d5700000)
object space 848896K, 2% used [0x0000000180400000,0x00000001814c2bb0,0x00000001b4100000)
Metaspace used 27860K, capacity 28290K, committed 28800K, reserved 1075200K
class space used 2982K, capacity 3134K, committed 3200K, reserved 1048576K
Event: 72408.234 GC heap after
Heap after GC invocations=281 (full 22):
PSYoungGen total 99328K, used 2153K [0x00000005d5700000, 0x00000005e0800000, 0x0000000800000000)
eden space 96768K, 0% used [0x00000005d5700000,0x00000005d5700000,0x00000005db580000)
from space 2560K, 84% used [0x00000005e0400000,0x00000005e061a6d0,0x00000005e0680000)
to space 1536K, 0% used [0x00000005e0680000,0x00000005e0680000,0x00000005e0800000)
ParOldGen total 848896K, used 17162K [0x0000000180400000, 0x00000001b4100000, 0x00000005d5700000)
object space 848896K, 2% used [0x0000000180400000,0x00000001814c2bb0,0x00000001b4100000)
Metaspace used 27860K, capacity 28290K, committed 28800K, reserved 1075200K
class space used 2982K, capacity 3134K, committed 3200K, reserved 1048576K
}