Oracle Database Discussions

Hi All,

My system is basically a vertically clustered. Please find below the configuration details of the system.

1. IHS web servers - 4 servers
2. WAS 6.1 - 4 servers - Each one has 3 vertically clustered JVM instances.
3. Oracle 10g DB - 1

We are using a Content Management Product called 'Fatwire 7.0.2' which runs over each WAS instance.

Issue Description: We are getting blank pages(no html in the view source) intermittently in our production environment.

Technical Description: We are encountering the following exception in the websphere log trace.

StaleConnectionException:
[5/19/09 13:10:57:894 GMT] 00000a31 db E COM.FutureTense.Servlet.e6 b Exception getting a connection for data source jndi/engine_cs
com.ibm.websphere.ce.cm.StaleConnectionException: Io exception: Got minus one from a read callDSRA0010E: SQL State = null, Error Code = 17,002
at sun.reflect.GeneratedConstructorAccessor89.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:522)
at com.ibm.websphere.rsadapter.GenericDataStoreHelper.mapExceptionHelper(GenericDataStoreHelper.java:523)
at com.ibm.websphere.rsadapter.GenericDataStoreHelper.mapException(GenericDataStoreHelper.java:578)
at com.ibm.ws.rsadapter.AdapterUtil.mapException(AdapterUtil.java:2124)
at com.ibm.ws.rsadapter.spi.WSRdbDataSource.getPooledConnection(WSRdbDataSource.java:1441)
at com.ibm.ws.rsadapter.spi.WSManagedConnectionFactoryImpl.createManagedConnection(WSManagedConnectionFactoryImpl.java:1089)
at com.ibm.ejs.j2c.FreePool.createManagedConnectionWithMCWrapper(FreePool.java:1821)
at com.ibm.ejs.j2c.FreePool.createOrWaitForConnection(FreePool.java:1552)
at com.ibm.ejs.j2c.PoolManager.reserve(PoolManager.java:2289)
at com.ibm.ejs.j2c.ConnectionManager.allocateMCWrapper(ConnectionManager.java:909)
at com.ibm.ejs.j2c.ConnectionManager.allocateConnection(ConnectionManager.java:599)
at com.ibm.ws.rsadapter.jdbc.WSJdbcDataSource.getConnection(WSJdbcDataSource.java:439)
at com.ibm.ws.rsadapter.jdbc.WSJdbcDataSource.getConnection(WSJdbcDataSource.java:408)
at COM.FutureTense.Servlet.e6.b(DashoA19*..)
at COM.FutureTense.Servlet.e5.a(DashoA19*..)
at COM.FutureTense.Servlet.g8.a(DashoA19*..)
at COM.FutureTense.Servlet.g8.<init>(DashoA19*..)
at COM.FutureTense.Servlet.e5.a(DashoA19*..)
at COM.FutureTense.Common.b6.e(DashoA19*..)
at COM.FutureTense.Common.b6.aj(DashoA19*..)
at COM.FutureTense.CatalogManager.do.a(DashoA19*..)
at COM.FutureTense.CatalogManager.do.a(DashoA19*..)
at COM.FutureTense.Common.ez.ci(DashoA19*..)
at COM.FutureTense.Common.g1$nu.doWork(DashoA19*..)
at com.fatwire.cs.core.util.KeyThrottle.execute(DashoA19*..)
at COM.FutureTense.Common.g1.a(DashoA19*..)
at COM.FutureTense.Common.g1.a(DashoA19*..)
at COM.FutureTense.Common.ez.o(DashoA19*..)
at COM.FutureTense.Servlet.FTServlet.a(DashoA19*..)
at COM.FutureTense.Servlet.FTServlet.doGet(DashoA19*..)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:743)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:856)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1075)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1016)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:118)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain._doFilter(WebAppFilterChain.java:87)
at com.ibm.ws.webcontainer.filter.WebAppFilterManager.doFilter(WebAppFilterManager.java:771)
at com.ibm.ws.webcontainer.filter.WebAppFilterManager.doFilter(WebAppFilterManager.java:679)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:546)
at com.ibm.ws.wswebcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:478)
at com.ibm.ws.webcontainer.servlet.CacheServletWrapper.handleRequest(CacheServletWrapper.java:90)
at com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:744)
at com.ibm.ws.wswebcontainer.WebContainer.handleRequest(WebContainer.java:1455)
at com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java:115)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:458)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewInformation(HttpInboundLink.java:387)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.ready(HttpInboundLink.java:267)
at com.ibm.ws.tcp.channel.impl.NewConnectionInitialReadCallback.sendToDiscriminators(NewConnectionInitialReadCallback.java:214)
at com.ibm.ws.tcp.channel.impl.NewConnectionInitialReadCallback.complete(NewConnectionInitialReadCallback.java:113)
at com.ibm.ws.tcp.channel.impl.AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:165)
at com.ibm.io.async.AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217)
at com.ibm.io.async.AsyncChannelFuture$1.run(AsyncChannelFuture.java:205)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1473)
---- Begin backtrace for Nested Throwables
java.sql.SQLException: Io exception: Got minus one from a read callDSRA0010E: SQL State = null, Error Code = 17,002
at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:112)
at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:146)
at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:255)
at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:387)
at oracle.jdbc.driver.PhysicalConnection.<init>(PhysicalConnection.java:439)
at oracle.jdbc.driver.T4CConnection.<init>(T4CConnection.java:165)
at oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:35)
at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:801)
at oracle.jdbc.pool.OracleDataSource.getPhysicalConnection(OracleDataSource.java:297)
at oracle.jdbc.pool.OracleDataSource.getConnection(OracleDataSource.java:221)
at oracle.jdbc.pool.OracleConnectionPoolDataSource.getPhysicalConnection(OracleConnectionPoolDataSource.java:157)
at oracle.jdbc.pool.OracleConnectionPoolDataSource.getPooledConnection(OracleConnectionPoolDataSource.java:94)
at oracle.jdbc.pool.OracleConnectionPoolDataSource.getPooledConnection(OracleConnectionPoolDataSource.java:75)
at com.ibm.ws.rsadapter.spi.InternalGenericDataStoreHelper$1.run(InternalGenericDataStoreHelper.java:918)
at java.security.AccessController.doPrivileged(AccessController.java:246)
at com.ibm.ws.security.util.AccessController.doPrivileged(AccessController.java:125)
at com.ibm.ws.rsadapter.spi.InternalGenericDataStoreHelper.getPooledConnection(InternalGenericDataStoreHelper.java:955)
at com.ibm.ws.rsadapter.spi.WSRdbDataSource.getPooledConnection(WSRdbDataSource.java:1437)
at com.ibm.ws.rsadapter.spi.WSManagedConnectionFactoryImpl.createManagedConnection(WSManagedConnectionFactoryImpl.java:1089)
at com.ibm.ejs.j2c.FreePool.createManagedConnectionWithMCWrapper(FreePool.java:1821)
at com.ibm.ejs.j2c.FreePool.createOrWaitForConnection(FreePool.java:1552)
at com.ibm.ejs.j2c.PoolManager.reserve(PoolManager.java:2289)
at com.ibm.ejs.j2c.ConnectionManager.allocateMCWrapper(ConnectionManager.java:909)
at com.ibm.ejs.j2c.ConnectionManager.allocateConnection(ConnectionManager.java:599)
at com.ibm.ws.rsadapter.jdbc.WSJdbcDataSource.getConnection(WSJdbcDataSource.java:439)
at com.ibm.ws.rsadapter.jdbc.WSJdbcDataSource.getConnection(WSJdbcDataSource.java:408)
at COM.FutureTense.Servlet.e6.b(DashoA19*..)
at COM.FutureTense.Servlet.e5.a(DashoA19*..)
at COM.FutureTense.Servlet.g8.a(DashoA19*..)
at COM.FutureTense.Servlet.g8.<init>(DashoA19*..)
at COM.FutureTense.Servlet.e5.a(DashoA19*..)
at COM.FutureTense.Common.b6.e(DashoA19*..)
at COM.FutureTense.Common.b6.aj(DashoA19*..)
at COM.FutureTense.CatalogManager.do.a(DashoA19*..)
at COM.FutureTense.CatalogManager.do.a(DashoA19*..)
at COM.FutureTense.Common.ez.ci(DashoA19*..)
at COM.FutureTense.Common.g1$nu.doWork(DashoA19*..)
at com.fatwire.cs.core.util.KeyThrottle.execute(DashoA19*..)
at COM.FutureTense.Common.g1.a(DashoA19*..)
at COM.FutureTense.Common.g1.a(DashoA19*..)
at COM.FutureTense.Common.ez.o(DashoA19*..)
at COM.FutureTense.Servlet.FTServlet.a(DashoA19*..)
at COM.FutureTense.Servlet.FTServlet.doGet(DashoA19*..)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:743)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:856)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1075)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1016)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:118)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain._doFilter(WebAppFilterChain.java:87)
at com.ibm.ws.webcontainer.filter.WebAppFilterManager.doFilter(WebAppFilterManager.java:771)
at com.ibm.ws.webcontainer.filter.WebAppFilterManager.doFilter(WebAppFilterManager.java:679)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:546)
at com.ibm.ws.wswebcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:478)
at com.ibm.ws.webcontainer.servlet.CacheServletWrapper.handleRequest(CacheServletWrapper.java:90)
at com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:744)
at com.ibm.ws.wswebcontainer.WebContainer.handleRequest(WebContainer.java:1455)
at com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java:115)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:458)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewInformation(HttpInboundLink.java:387)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.ready(HttpInboundLink.java:267)
at com.ibm.ws.tcp.channel.impl.NewConnectionInitialReadCallback.sendToDiscriminators(NewConnectionInitialReadCallback.java:214)
at com.ibm.ws.tcp.channel.impl.NewConnectionInitialReadCallback.complete(NewConnectionInitialReadCallback.java:113)
at com.ibm.ws.tcp.channel.impl.AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:165)
at com.ibm.io.async.AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217)
at com.ibm.io.async.AsyncChannelFuture$1.run(AsyncChannelFuture.java:205)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1473)
I herewith attached the full trace of the exception.

It looks like the the WAS instance tries to use stale connections sometimes. As stale connections cannot able to retrieve any information from DB, we are getting blank pages.

FYI:
DB Connection Pool Size: 700

I really don’t know whether the issue with the DB or WAS. Could you please help me in resolving this issue?

Your help on this is much appreciated.

Kind Regards,
Selvandhan M