Hi
In the enterprise where i work we've been experiencing some problems. The cpu usage grows up to 100% and keeps so until we restart it. We are running a zk framework based web app on a tomcat server running on linux with jdk 1.6.0_12 and postgres as dbms.
We made many java dumps and used Thread Dump Analyzer and confirmed it's not just an infinite loop in our code (that was the first thing we thought).
We saw that each time that happened, a connection with the postgres was kept open and appeared idle in transaction:
postgres 6833 19172 0 08:00 ? 00:00:17 postgres: hco hco 10.10.10.13(51460) idle in transaction
I guess that should be related somehow, but i really don't know if it's related with the reason of the problem or is a consequence of it.
This problem usually starts at 9 o clock in the morning, that's when clients start connecting.
Something that called our attention too was that there were 75 sockets (java.io, not java.nio) open and they are always RUNNABLE. I guess socket reading threads should be blocked waiting, not runnable.
"TP-Processor219" daemon prio=10 tid=0x0a854400 nid=0x30a7 runnable [0x271fe000..0x271fefb0]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
- locked <0xaf3633d0> (a java.io.BufferedInputStream)
at org.apache.jk.common.ChannelSocket.read(ChannelSocket.java:621)
at org.apache.jk.common.ChannelSocket.receive(ChannelSocket.java:559)
at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:686)
at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:891)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:690)
at java.lang.Thread.run(Thread.java:619)
Is that normal?. I read http://stackoverflow.com/questions/2978644/why-does-javas-socket-connect-consume-100-cpu-resources about a similar situation, but I don't know how to know what's the thread that is consuming most of the processing. How could i know it?.
I'm sure we are not seeing something. I have some thread dumps taken during these failures if needed (i didn't include them here because they are too big).