Hi,
Whilst running performance tests with JE, java.nio.BufferOverflowException was thrown from JE cleaner thread. We have been running similar tests for couple of years on regular basis almost every day and never seen such sort of exception before. It looks like a JE bug.
Can anyone confirm that it a JE issue?
The environment was started with configuration
EnvironmentConfig envConfig = new EnvironmentConfig();
envConfig.setAllowCreate(true);
envConfig.setTransactional(true);
envConfig.setCacheMode(CacheMode.EVICT_LN);
envConfig.setConfigParam(EnvironmentConfig.FILE_LOGGING_LEVEL, "OFF");
envConfig.setConfigParam(EnvironmentConfig.CONSOLE_LOGGING_LEVEL, "OFF");
envConfig.setLoggingHandler(new Slf4jLoggingHandler("["+configuration.getName()+"]"));
EnvironmentConfig.LOCK_N_LOCK_TABLES = "7"
EnvironmentConfig.STATS_COLLECT = "false"
I can provide the store files if required.
The stack trace is below.
2016-01-10 23:04:44,672 ERROR [Cleaner-1] (c.s.j.d.EnvironmentImpl) - [default]
com.sleepycat.je.EnvironmentFailureException: Environment invalid because of previous exception: (JE 5.0.104) /path/to/store java.nio.BufferOverflowException UNEXPECTED_EXCEPTION_FATAL: Unexpected internal Exception, unable to continue. Environment is invalid and must be closed.
at com.sleepycat.je.EnvironmentFailureException.unexpectedException(EnvironmentFailureException.java:351)
at com.sleepycat.je.log.LogManager.serialLog(LogManager.java:574)
at com.sleepycat.je.log.LogManager.logItems(LogManager.java:508)
at com.sleepycat.je.log.LogManager.multiLog(LogManager.java:410)
at com.sleepycat.je.log.LogManager.log(LogManager.java:350)
at com.sleepycat.je.tree.LN.logInternal(LN.java:556)
at com.sleepycat.je.tree.LN.log(LN.java:411)
at com.sleepycat.je.cleaner.FileProcessor.processFoundLN(FileProcessor.java:1084)
at com.sleepycat.je.cleaner.FileProcessor.processLN(FileProcessor.java:881)
at com.sleepycat.je.cleaner.FileProcessor.processFile(FileProcessor.java:677)
at com.sleepycat.je.cleaner.FileProcessor.doClean(FileProcessor.java:290)
at com.sleepycat.je.cleaner.FileProcessor.onWakeup(FileProcessor.java:148)
at com.sleepycat.je.utilint.DaemonThread.run(DaemonThread.java:148)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.nio.BufferOverflowException
at java.nio.HeapByteBuffer.put(HeapByteBuffer.java:183)
at java.nio.ByteBuffer.put(ByteBuffer.java:832)
at com.sleepycat.je.log.LogUtils.writeBytesNoLength(LogUtils.java:349)
at com.sleepycat.je.log.entry.LNLogEntry.writeBaseLNEntry(LNLogEntry.java:403)
at com.sleepycat.je.log.entry.LNLogEntry.writeEntry(LNLogEntry.java:365)
at com.sleepycat.je.log.entry.BaseReplicableEntry.writeEntry(BaseReplicableEntry.java:51)
at com.sleepycat.je.log.entry.LNLogEntry.writeEntry(LNLogEntry.java:52)
at com.sleepycat.je.log.LogManager.marshallIntoBuffer(LogManager.java:841)
at com.sleepycat.je.log.LogManager.serialLogWork(LogManager.java:708)
at com.sleepycat.je.log.LogManager.serialLog(LogManager.java:555)
... 12 more
2016-01-10 23:04:44,678 ERROR [Checkpointer] (c.s.j.d.EnvironmentImpl) - [default] checkpointId=129
com.sleepycat.je.EnvironmentFailureException: (JE 5.0.104) Environment must be closed, caused by: com.sleepycat.je.EnvironmentFailureException: Environment invalid because of previous exception: (JE 5.0.104) /path/to/store java.nio.BufferOverflowException UNEXPECTED_EXCEPTION_FATAL: Unexpected internal Exception, unable to continue. Environment is invalid and must be closed.
at com.sleepycat.je.EnvironmentFailureException.wrapSelf(EnvironmentFailureException.java:246)
at com.sleepycat.je.dbi.EnvironmentImpl.checkIfInvalid(EnvironmentImpl.java:1618)
at com.sleepycat.je.log.LogManager.serialLog(LogManager.java:552)
at com.sleepycat.je.log.LogManager.logItems(LogManager.java:508)
at com.sleepycat.je.log.LogManager.multiLog(LogManager.java:410)
at com.sleepycat.je.log.LogManager.log(LogManager.java:350)
at com.sleepycat.je.tree.LN.logInternal(LN.java:556)
at com.sleepycat.je.tree.LN.optionalLog(LN.java:372)
at com.sleepycat.je.dbi.CursorImpl.putCurrentAlreadyLatchedAndLocked(CursorImpl.java:1290)
at com.sleepycat.je.dbi.CursorImpl.putCurrent(CursorImpl.java:1186)
at com.sleepycat.je.dbi.DbTree$RewriteMapLN.doWork(DbTree.java:804)
at com.sleepycat.je.tree.Tree.withRootLatchedExclusive(Tree.java:415)
at com.sleepycat.je.dbi.DbTree.modifyDbRoot(DbTree.java:773)
at com.sleepycat.je.dbi.DbTree.modifyDbRoot(DbTree.java:698)
at com.sleepycat.je.recovery.Checkpointer.flushIN(Checkpointer.java:1079)
at com.sleepycat.je.recovery.Checkpointer.flushIN(Checkpointer.java:1341)
at com.sleepycat.je.recovery.Checkpointer.flushDirtyNodes(Checkpointer.java:951)
at com.sleepycat.je.recovery.Checkpointer.doCheckpoint(Checkpointer.java:735)
at com.sleepycat.je.recovery.Checkpointer.onWakeup(Checkpointer.java:508)
at com.sleepycat.je.utilint.DaemonThread.run(DaemonThread.java:148)
at java.lang.Thread.run(Thread.java:745)
Caused by: com.sleepycat.je.EnvironmentFailureException: Environment invalid because of previous exception: (JE 5.0.104) /path/to/store java.nio.BufferOverflowException UNEXPECTED_EXCEPTION_FATAL: Unexpected internal Exception, unable to continue. Environment is invalid and must be closed.
at com.sleepycat.je.EnvironmentFailureException.unexpectedException(EnvironmentFailureException.java:351)
at com.sleepycat.je.log.LogManager.serialLog(LogManager.java:574)
at com.sleepycat.je.log.LogManager.logItems(LogManager.java:508)
at com.sleepycat.je.log.LogManager.multiLog(LogManager.java:410)
at com.sleepycat.je.log.LogManager.log(LogManager.java:350)
at com.sleepycat.je.tree.LN.logInternal(LN.java:556)
at com.sleepycat.je.tree.LN.log(LN.java:411)
at com.sleepycat.je.cleaner.FileProcessor.processFoundLN(FileProcessor.java:1084)
at com.sleepycat.je.cleaner.FileProcessor.processLN(FileProcessor.java:881)
at com.sleepycat.je.cleaner.FileProcessor.processFile(FileProcessor.java:677)
at com.sleepycat.je.cleaner.FileProcessor.doClean(FileProcessor.java:290)
at com.sleepycat.je.cleaner.FileProcessor.onWakeup(FileProcessor.java:148)
... 2 more
Caused by: java.nio.BufferOverflowException
at java.nio.HeapByteBuffer.put(HeapByteBuffer.java:183)
at java.nio.ByteBuffer.put(ByteBuffer.java:832)
at com.sleepycat.je.log.LogUtils.writeBytesNoLength(LogUtils.java:349)
at com.sleepycat.je.log.entry.LNLogEntry.writeBaseLNEntry(LNLogEntry.java:403)
at com.sleepycat.je.log.entry.LNLogEntry.writeEntry(LNLogEntry.java:365)
at com.sleepycat.je.log.entry.BaseReplicableEntry.writeEntry(BaseReplicableEntry.java:51)
at com.sleepycat.je.log.entry.LNLogEntry.writeEntry(LNLogEntry.java:52)
at com.sleepycat.je.log.LogManager.marshallIntoBuffer(LogManager.java:841)
at com.sleepycat.je.log.LogManager.serialLogWork(LogManager.java:708)
at com.sleepycat.je.log.LogManager.serialLog(LogManager.java:555)
... 12 more
2016-01-10 23:04:44,688 ERROR [Cleaner-1] (o.a.q.s.s.b.LoggingAsyncExceptionListener) - Asynchronous exception thrown by BDB thread 'Cleaner-1'
com.sleepycat.je.EnvironmentFailureException: Environment invalid because of previous exception: (JE 5.0.104) /path/to/store java.nio.BufferOverflowException UNEXPECTED_EXCEPTION_FATAL: Unexpected internal Exception, unable to continue. Environment is invalid and must be closed.
at com.sleepycat.je.EnvironmentFailureException.unexpectedException(EnvironmentFailureException.java:351) ~[je-5.0.104.jar:5.0.104]
at com.sleepycat.je.log.LogManager.serialLog(LogManager.java:574) ~[je-5.0.104.jar:5.0.104]
at com.sleepycat.je.log.LogManager.logItems(LogManager.java:508) ~[je-5.0.104.jar:5.0.104]
at com.sleepycat.je.log.LogManager.multiLog(LogManager.java:410) ~[je-5.0.104.jar:5.0.104]
at com.sleepycat.je.log.LogManager.log(LogManager.java:350) ~[je-5.0.104.jar:5.0.104]
at com.sleepycat.je.tree.LN.logInternal(LN.java:556) ~[je-5.0.104.jar:5.0.104]
at com.sleepycat.je.tree.LN.log(LN.java:411) ~[je-5.0.104.jar:5.0.104]
at com.sleepycat.je.cleaner.FileProcessor.processFoundLN(FileProcessor.java:1084) ~[je-5.0.104.jar:5.0.104]
at com.sleepycat.je.cleaner.FileProcessor.processLN(FileProcessor.java:881) ~[je-5.0.104.jar:5.0.104]
at com.sleepycat.je.cleaner.FileProcessor.processFile(FileProcessor.java:677) ~[je-5.0.104.jar:5.0.104]
at com.sleepycat.je.cleaner.FileProcessor.doClean(FileProcessor.java:290) ~[je-5.0.104.jar:5.0.104]
at com.sleepycat.je.cleaner.FileProcessor.onWakeup(FileProcessor.java:148) ~[je-5.0.104.jar:5.0.104]
at com.sleepycat.je.utilint.DaemonThread.run(DaemonThread.java:148) ~[je-5.0.104.jar:5.0.104]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_67]
Caused by: java.nio.BufferOverflowException: null
at java.nio.HeapByteBuffer.put(HeapByteBuffer.java:183) ~[na:1.7.0_67]
at java.nio.ByteBuffer.put(ByteBuffer.java:832) ~[na:1.7.0_67]
at com.sleepycat.je.log.LogUtils.writeBytesNoLength(LogUtils.java:349) ~[je-5.0.104.jar:5.0.104]
at com.sleepycat.je.log.entry.LNLogEntry.writeBaseLNEntry(LNLogEntry.java:403) ~[je-5.0.104.jar:5.0.104]
at com.sleepycat.je.log.entry.LNLogEntry.writeEntry(LNLogEntry.java:365) ~[je-5.0.104.jar:5.0.104]
at com.sleepycat.je.log.entry.BaseReplicableEntry.writeEntry(BaseReplicableEntry.java:51) ~[je-5.0.104.jar:5.0.104]
at com.sleepycat.je.log.entry.LNLogEntry.writeEntry(LNLogEntry.java:52) ~[je-5.0.104.jar:5.0.104]
at com.sleepycat.je.log.LogManager.marshallIntoBuffer(LogManager.java:841) ~[je-5.0.104.jar:5.0.104]
at com.sleepycat.je.log.LogManager.serialLogWork(LogManager.java:708) ~[je-5.0.104.jar:5.0.104]
at com.sleepycat.je.log.LogManager.serialLog(LogManager.java:555) ~[je-5.0.104.jar:5.0.104]
... 12 common frames omitted
2016-01-10 23:04:44,688 ERROR [Commit-Thread-default] (o.a.q.s.u.ServerScopedRuntimeException) - Exception whilst syncing data to disk
com.sleepycat.je.EnvironmentFailureException: (JE 5.0.104) Environment must be closed, caused by: com.sleepycat.je.EnvironmentFailureException: Environment invalid because of previous exception: (JE 5.0.104)/path/to/store java.nio.BufferOverflowException UNEXPECTED_EXCEPTION_FATAL: Unexpected internal Exception, unable to continue. Environment is invalid and must be closed.
at com.sleepycat.je.EnvironmentFailureException.wrapSelf(EnvironmentFailureException.java:246) ~[je-5.0.104.jar:5.0.104]
at com.sleepycat.je.dbi.EnvironmentImpl.checkIfInvalid(EnvironmentImpl.java:1618) ~[je-5.0.104.jar:5.0.104]
at com.sleepycat.je.log.FileManager.writeLogBuffer(FileManager.java:1589) ~[je-5.0.104.jar:5.0.104]
at com.sleepycat.je.log.LogBufferPool.writeBufferToFile(LogBufferPool.java:411) ~[je-5.0.104.jar:5.0.104]
at com.sleepycat.je.log.LogBufferPool.writeDirty(LogBufferPool.java:370) ~[je-5.0.104.jar:5.0.104]
at com.sleepycat.je.log.LogManager.flushInternal(LogManager.java:1306) ~[je-5.0.104.jar:5.0.104]
at com.sleepycat.je.log.LogManager.flush(LogManager.java:1256) ~[je-5.0.104.jar:5.0.104]
at com.sleepycat.je.dbi.EnvironmentImpl.flushLog(EnvironmentImpl.java:2084) ~[je-5.0.104.jar:5.0.104]
at com.sleepycat.je.Environment.flushLog(Environment.java:1593) ~[je-5.0.104.jar:5.0.104]
at org.apache.qpid.server.store.berkeleydb.StandardEnvironmentFacade.flushLog(StandardEnvironmentFacade.java:212) ~[qpid-bdbstore-6.0.0.jar:6.0.0]
at org.apache.qpid.server.store.berkeleydb.CoalescingCommiter$CommitThread.processJobs(CoalescingCommiter.java:198) [qpid-bdbstore-6.0.0.jar:6.0.0]
at org.apache.qpid.server.store.berkeleydb.CoalescingCommiter$CommitThread.run(CoalescingCommiter.java:177) [qpid-bdbstore-6.0.0.jar:6.0.0]
Caused by: com.sleepycat.je.EnvironmentFailureException: Environment invalid because of previous exception: (JE 5.0.104) /path/to/store java.nio.BufferOverflowException UNEXPECTED_EXCEPTION_FATAL: Unexpected internal Exception, unable to continue. Environment is invalid and must be closed.
at com.sleepycat.je.EnvironmentFailureException.unexpectedException(EnvironmentFailureException.java:351) ~[je-5.0.104.jar:5.0.104]
at com.sleepycat.je.log.LogManager.serialLog(LogManager.java:574) ~[je-5.0.104.jar:5.0.104]
at com.sleepycat.je.log.LogManager.logItems(LogManager.java:508) ~[je-5.0.104.jar:5.0.104]
at com.sleepycat.je.log.LogManager.multiLog(LogManager.java:410) ~[je-5.0.104.jar:5.0.104]
at com.sleepycat.je.log.LogManager.log(LogManager.java:350) ~[je-5.0.104.jar:5.0.104]
at com.sleepycat.je.tree.LN.logInternal(LN.java:556) ~[je-5.0.104.jar:5.0.104]
at com.sleepycat.je.tree.LN.log(LN.java:411) ~[je-5.0.104.jar:5.0.104]
at com.sleepycat.je.cleaner.FileProcessor.processFoundLN(FileProcessor.java:1084) ~[je-5.0.104.jar:5.0.104]
at com.sleepycat.je.cleaner.FileProcessor.processLN(FileProcessor.java:881) ~[je-5.0.104.jar:5.0.104]
at com.sleepycat.je.cleaner.FileProcessor.processFile(FileProcessor.java:677) ~[je-5.0.104.jar:5.0.104]
at com.sleepycat.je.cleaner.FileProcessor.doClean(FileProcessor.java:290) ~[je-5.0.104.jar:5.0.104]