Stopped threads waiting on DriverManager.getConnection()?
843811May 19 2004 — edited May 19 2004Solaris 5.8, Sun JDK 1.3.1-b24
RMI based application with the server piece running as a Java app on Solaris. After a few hours of use, with 100-150 users, the server app appears to hang from the client. If I do a kill -3 on the server process, I see a whole buttload of threads (over 100) stuck on DriverManager.getConnection() method (see below). The trace doesn't show anything deeper than that, but it is Oracle thin client being used. Our DBAs don't see any connections being attempted in the listeners log, and in fact, the number of the connections I see to the DB box via netstat jives exactly with the number of sessions I see on the Oracle side (in V$SESSION). While this is going on, I can connect to the DB box from the app server box all day long using a small class I wrote that just does a connection open and close.
This app was working fairly flawlessly until about a week ago. Nothing in terms of drivers or JVM version has changed, but there has been slightly more user load lately. It seems that every new user request that comes in after a certain point ends up with a thread blocked like this. What is the thread stopped in getConnection() indicative of? Has anyone seen anything like this?
Much appreciated,
Mike
Tons of these show in the thread dump (kill -3):
Wed May 19 11:40:04 EDT 2004:ExecGroup-0:out:"RMI TCP Connection(11307)-10.160.96.220" daemon prio=5 tid=0x4946d0 nid=0x2e7b waiting for monitor entry [0xd4480000..0xd44819e0]
Wed May 19 11:40:04 EDT 2004:ExecGroup-0:out: at java.sql.DriverManager.getConnection(DriverManager.java:193)
Wed May 19 11:40:04 EDT 2004:ExecGroup-0:out: at sched_MT.rmiSched_MTImpl.getLocation(rmiSched_MTImpl.java:556)
Wed May 19 11:40:04 EDT 2004:ExecGroup-0:out: at java.lang.reflect.Method.invoke(Native Method)
Wed May 19 11:40:04 EDT 2004:ExecGroup-0:out: at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:241)
Wed May 19 11:40:04 EDT 2004:ExecGroup-0:out: at sun.rmi.transport.Transport$1.run(Transport.java:152)
Wed May 19 11:40:04 EDT 2004:ExecGroup-0:out: at java.security.AccessController.doPrivileged(Native Method)
Wed May 19 11:40:04 EDT 2004:ExecGroup-0:out: at sun.rmi.transport.Transport.serviceCall(Transport.java:148)
Wed May 19 11:40:04 EDT 2004:ExecGroup-0:out: at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:465)
Wed May 19 11:40:04 EDT 2004:ExecGroup-0:out: at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:706)