Problem in the database at 10pm everyday
974867Nov 18 2012 — edited Dec 27 2012Hi,
I'm using Oracle 11gR2 (11.2.0.1.0) with default installation, running on SLES 11 SP2.
Another SLES 11SP2 server hosts a web application with JBoss, it's using the corresponding JDBC connector (ojdbc6 11.2.0.1.0) to connect and update some tables of the database.
The problem is that every day the jboss is unable to connect to the database for 1 hour exactly. This happens every day from 22:00 to 23:00 pm.
The servers don't have any job scheduled at Linux level, JBboss, backup server or network infrastructure.
For that reason I think the problem is in the database, I was looking for predefined jobs or maintenance tasks that may be running at 22 pm and making the database inaccessible, but still could not solve the problem.
This is the error log from JBoss that i see every time that the JDBC driver want to connect to the database.
After exactly one hour the connection returns automatically without problems.
---------------------------------------------------
2012-11-13 22:00:26,847 WARN [org.jboss.resource.connectionmanager.TxConnectionManager] (Thread-45) Connection error occured: org.jboss.resource.connectionm
anager.TxConnectionManager$TxConnectionEventListener@1ac1a45[state=NORMAL mc=org.jboss.resource.adapter.jdbc.local.LocalManagedConnection@563fa9 handles=1 la
stUse=1352854823458 permit=true trackByTx=false mcp=org.jboss.resource.connectionmanager.JBossManagedConnectionPool$PoolBySubject@458eab context=org.jboss.re
source.connectionmanager.InternalManagedConnectionPool@99b936 xaResource=org.jboss.resource.connectionmanager.TxConnectionManager$LocalXAResource@6d8e2a txSy
nc=null]
java.lang.RuntimeException: Assertion botch: negative time
at oracle.jdbc.driver.DateCommonBinder.setOracleHMS(OraclePreparedStatement.java:18740)
at oracle.jdbc.driver.TimestampBinder.bind(OraclePreparedStatement.java:19245)
at oracle.jdbc.driver.OraclePreparedStatement.setupBindBuffers(OraclePreparedStatement.java:3014)
at oracle.jdbc.driver.OraclePreparedStatement.executeBatch(OraclePreparedStatement.java:9973)
at oracle.jdbc.driver.OracleStatementWrapper.executeBatch(OracleStatementWrapper.java:213)
at org.jboss.resource.adapter.jdbc.WrappedStatement.executeBatch(WrappedStatement.java:774)
at org.hibernate.jdbc.BatchingBatcher.doExecuteBatch(BatchingBatcher.java:70)
at org.hibernate.jdbc.AbstractBatcher.executeBatch(AbstractBatcher.java:268)
at org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:268)
at org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:185)
at org.hibernate.event.def.AbstractFlushingEventListener.performExecutions(AbstractFlushingEventListener.java:321)
at org.hibernate.event.def.DefaultFlushEventListener.onFlush(DefaultFlushEventListener.java:51)
at org.hibernate.impl.SessionImpl.flush(SessionImpl.java:1216)
at org.hibernate.impl.SessionImpl.managedFlush(SessionImpl.java:383)
at org.hibernate.transaction.JDBCTransaction.commit(JDBCTransaction.java:133)
at com.novell.soa.persist.HibernateUtil.commit(HibernateUtil.java:211)
at com.novell.soa.persist.HibernateDAO.commit(HibernateDAO.java:450)
at com.novell.soa.af.impl.persist.EngineStateDAO.updateHeartbeat(EngineStateDAO.java:151)
at com.novell.soa.af.impl.core.EngineImpl$HeartbeatTimer.run(EngineImpl.java:1459)
at java.lang.Thread.run(Unknown Source)
2012-11-13 22:00:27,086 INFO [STDOUT] (Thread-45) ERROR [RBPM] [com.novell.soa.persist.HibernateUtil:rollback] Error rolling back transaction
org.hibernate.TransactionException: JDBC rollback failed
at org.hibernate.transaction.JDBCTransaction.rollback(JDBCTransaction.java:200)
at com.novell.soa.persist.HibernateUtil.rollback(HibernateUtil.java:237)
at com.novell.soa.persist.HibernateDAO.rollback(HibernateDAO.java:463)
at com.novell.soa.af.impl.persist.EngineStateDAO.updateHeartbeat(EngineStateDAO.java:154)
at com.novell.soa.af.impl.core.EngineImpl$HeartbeatTimer.run(EngineImpl.java:1459)
at java.lang.Thread.run(Unknown Source)
Caused by: java.sql.SQLException: You cannot rollback with autocommit set!
at org.jboss.resource.adapter.jdbc.BaseWrapperManagedConnection.jdbcRollback(BaseWrapperManagedConnection.java:695)
at org.jboss.resource.adapter.jdbc.WrappedConnection.rollback(WrappedConnection.java:500)
at org.hibernate.transaction.JDBCTransaction.rollbackAndResetAutoCommit(JDBCTransaction.java:213)
at org.hibernate.transaction.JDBCTransaction.rollback(JDBCTransaction.java:192)
... 5 more
2012-11-13 22:00:27,088 INFO [STDOUT] (Thread-45) WARN [RBPM] [com.novell.soa.af.impl.core.EngineImpl:run] Failed to update heartbeat for engine: Engine01
com.novell.soa.persist.PersistenceException: org.hibernate.exception.GenericJDBCException: Could not execute JDBC batch update
at com.novell.soa.persist.HibernateUtil.commit(HibernateUtil.java:213)
at com.novell.soa.persist.HibernateDAO.commit(HibernateDAO.java:450)
at com.novell.soa.af.impl.persist.EngineStateDAO.updateHeartbeat(EngineStateDAO.java:151)
at com.novell.soa.af.impl.core.EngineImpl$HeartbeatTimer.run(EngineImpl.java:1459)
at java.lang.Thread.run(Unknown Source)
Caused by: org.hibernate.exception.GenericJDBCException: Could not execute JDBC batch update
at org.hibernate.exception.SQLStateConverter.handledNonSpecificException(SQLStateConverter.java:140)
at org.hibernate.exception.SQLStateConverter.convert(SQLStateConverter.java:128)
at org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:66)
at org.hibernate.jdbc.AbstractBatcher.executeBatch(AbstractBatcher.java:275)
at org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:268)
at org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:185)
at org.hibernate.event.def.AbstractFlushingEventListener.performExecutions(AbstractFlushingEventListener.java:321)
at org.hibernate.event.def.DefaultFlushEventListener.onFlush(DefaultFlushEventListener.java:51)
at org.hibernate.impl.SessionImpl.flush(SessionImpl.java:1216)
at org.hibernate.impl.SessionImpl.managedFlush(SessionImpl.java:383)
at org.hibernate.transaction.JDBCTransaction.commit(JDBCTransaction.java:133)
at com.novell.soa.persist.HibernateUtil.commit(HibernateUtil.java:211)
... 4 more
Caused by: org.jboss.util.NestedSQLException: Error; - nested throwable: (java.lang.RuntimeException: Assertion botch: negative time)
at org.jboss.resource.adapter.jdbc.WrappedConnection.checkException(WrappedConnection.java:873)
at org.jboss.resource.adapter.jdbc.WrappedStatement.checkException(WrappedStatement.java:852)
at org.jboss.resource.adapter.jdbc.WrappedStatement.executeBatch(WrappedStatement.java:778)
at org.hibernate.jdbc.BatchingBatcher.doExecuteBatch(BatchingBatcher.java:70)
at org.hibernate.jdbc.AbstractBatcher.executeBatch(AbstractBatcher.java:268)
... 12 more
Caused by: java.lang.RuntimeException: Assertion botch: negative time
at oracle.jdbc.driver.DateCommonBinder.setOracleHMS(OraclePreparedStatement.java:18740)
at oracle.jdbc.driver.TimestampBinder.bind(OraclePreparedStatement.java:19245)
at oracle.jdbc.driver.OraclePreparedStatement.setupBindBuffers(OraclePreparedStatement.java:3014)
at oracle.jdbc.driver.OraclePreparedStatement.executeBatch(OraclePreparedStatement.java:9973)
at oracle.jdbc.driver.OracleStatementWrapper.executeBatch(OracleStatementWrapper.java:213)
at org.jboss.resource.adapter.jdbc.WrappedStatement.executeBatch(WrappedStatement.java:774)
... 14 more
---------------------------------------------------
Any ideas?
Thanks.
Regards.