Hi,
I am seeing an issue using 2 clustered WebLogic 10.3 servers in that one of the connection pools is constantly giving errors and we are unsure as to why. It has been reported by others to happen during larger load on the application running on the app server, however I seem to see it happening constantly for one connection pool configured. Just to give some system information:
java.version = 1.6.0_05
java.vm.vendor = Sun Microsystems Inc.
Two of our pools:
<Connection Pool "ThunderheadDS" using Driver: "Oracle JDBC driver", Version: "11.1.0.6.0-Production+".>
<Created "50" resources for pool "ThunderheadDS", out of which "50" are available and "0" are unavailable.>
<Created Connection Pool named ThunderheadDS.>
<Creating Data Source named ThunderheadDS, JNDI Name = jdbc/thunderheadDS.>
<Data Source ThunderheadDS has been successfully created.>
<Creating Connection Pool named ThunderheadDSECF, URL = jdbc:oracle:thin:@lndengu1.uk.db.com:1544:LNDENGU1, Properties = appName=weblogicDataSourceReadOnly;user=DSER_OWNER;portNumber=1544;databaseName=LNDENGU1;progName=weblogicDataSourceReadOnly;serverName=lndengu1.uk.db.com;.>
<Free resources in pool "ThunderheadDSECF" will be tested every "120" seconds.>
<Connection Pool "ThunderheadDSECF" connected to Database: "Oracle", Version: "Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - 64bit Production
<Connection Pool "ThunderheadDSECF" using Driver: "Oracle JDBC driver", Version: "11.1.0.6.0-Production+".>
<Created "1" resources for pool "ThunderheadDSECF", out of which "1" are available and "0" are unavailable.>
<Created Connection Pool named ThunderheadDSECF.>
<Creating Data Source named ThunderheadDSECF, JNDI Name = jdbc/thunderheadDSECF.>
<Data Source ThunderheadDSECF has been successfully created.>
<Creating Connection Pool named ThunderheadDSNoTX, URL = jdbc:oracle:thin:@lndengu1.uk.db.com:1544:LNDENGU1, Properties = appName=weblogicDataSourceNoTX;user=DSER_OWNER;portNumber=1544;databaseName=LNDENGU1;progName=weblogicDataSourceNoTX;serverName=lndengu1.uk.db.com;.>
<Free resources in pool "ThunderheadDSNoTX" will be tested every "120" seconds.>
<Connection Pool "ThunderheadDSNoTX" connected to Database: "Oracle", Version: "Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - 64bit Production
As you can see, these pools are similarly configured. They are using the WebLogic option of 'Test connections on reserve'.
After starting the servers we see this:
<[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1274160842702> <BEA-001128> <Connection for pool "ThunderheadDSECF" closed.>
<[ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'> <<anonymous>> <BEA1-0388216376ED6AB99A2C> <> <1274160876823> <BEA-000628> <Created "1" resources for pool "ThunderheadDSECF", out of which "1" are available and "0" are unavailable.>
<weblogic.GCMonitor> <<anonymous>> <> <> <1274160889484> <BEA-310002> <88% of the total memory in the server is free>
<[ACTIVE] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)'> <<anonymous>> <BEA1-0488216376ED6AB99A2C> <> <1274160890664> <BEA-000628> <Created "1" resources for pool "ThunderheadDSECF", out of which "1" are available and "0" are unavailable.>
<[ACTIVE] ExecuteThread: '30' for queue: 'weblogic.kernel.Default (self-tuning)'> <<anonymous>> <> <> <1274160890698> <BEA-000628> <Created "1" resources for pool "ThunderheadDSECF", out of which "1" are available and "0" are unavailable.>
<[ACTIVE] ExecuteThread: '52' for queue: 'weblogic.kernel.Default (self-tuning)'> <<anonymous>> <BEA1-07F5216376ED6AB99A2C> <> <1274160921528> <BEA-000628> <Created "1" resources for pool "ThunderheadDSECF", out of which "1" are available and "0" are unavailable.>
<[ACTIVE] ExecuteThread: '66' for queue: 'weblogic.kernel.Default (self-tuning)'> <<anonymous>> <BEA1-09C6216376ED6AB99A2C> <> <1274160933327> <BEA-001129> <Received exception while creating connection for pool "ThunderheadDS": Io exception: Got minus one from a read call>
<[ACTIVE] ExecuteThread: '66' for queue: 'weblogic.kernel.Default (self-tuning)'> <<anonymous>> <BEA1-09C6216376ED6AB99A2C> <> <1274160933330> <BEA-001156> <Stack trace associated with message 001129 follows:
java.sql.SQLRecoverableException: Io exception: Got minus one from a read call
at oracle.jdbc.driver.SQLStateMapping.newSQLException(SQLStateMapping.java:101)
at oracle.jdbc.driver.DatabaseError.newSQLException(DatabaseError.java:112)
at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:173)
at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:229)
at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:458)
at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:411)
at oracle.jdbc.driver.PhysicalConnection.<init>(PhysicalConnection.java:490)
at oracle.jdbc.driver.T4CConnection.<init>(T4CConnection.java:202)
at oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:33)
at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:474)
at oracle.jdbc.pool.OracleDataSource.getPhysicalConnection(OracleDataSource.java:275)
at oracle.jdbc.xa.client.OracleXADataSource.getPooledConnection(OracleXADataSource.java:454)
...
This is followed by a series of the same errors regarding the pool 'ThunderheadDS'. However it seems that the pool 'ThunderheadDSECF' does not give the same issue.
Eventually we get errors using ThunderheadDS of the format:
Internal error: Cannot obtain XAConnection weblogic.common.resourcepool.ResourceDisabledException: Pool ThunderheadDS is disabled, cannot allocate resources to applications..
I'm guessing this is disabled as a result of failing so many times already. I've searched the forums to understand why this 'Got minus one from a read call' error occurs. Thread (http://forums.sun.com/thread.jspa?forumID=48&threadID=5397831) suggests it is down to a bad connection url. However you can see from the other datasources that the URL is valid. Thread (
433151 suggests there is some form of leak for ORA sessions however there are no reported errors on the Oracle side.
I've also found many other posts on this, however it seems this message is quite generic and I can't find conclusive answers. Some point at JDBC drivers, others come to no conclusion (http://kr.forums.oracle.com/forums/thread.jspa?threadID=591056).
Does anyone know how to determine the root cause of the problem or solve this issue? Any help with this issue would be appreciated.
Thanks,
Louis