Berkeley DB Family

Hi,

We've seen what looks like a deadlock in je-5.0.48. Here are the offending threads:


_"RMI TCP Connection(5358)-10.119.152.33" - Thread t@41433_
java.lang.Thread.State: WAITING
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <469f6330> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:842)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1178)
at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:807)
at com.sleepycat.je.latch.SharedLatch.acquireExclusive(SharedLatch.java:95)
at com.sleepycat.je.tree.IN.latch(IN.java:473)
at com.sleepycat.je.tree.IN.findParent(IN.java:3149)
at com.sleepycat.je.tree.Tree.getParentINForChildIN(Tree.java:773)
at com.sleepycat.je.tree.Tree.getParentINForChildIN(Tree.java:704)
at com.sleepycat.je.tree.Tree.getParentINForChildIN(Tree.java:656)
at com.sleepycat.je.tree.Tree.getNextBinInternal(Tree.java:993)
at com.sleepycat.je.tree.Tree.getNextBin(Tree.java:924)
at com.sleepycat.je.dbi.CursorImpl.getNext(CursorImpl.java:1621)
at com.sleepycat.je.Cursor.retrieveNextAllowPhantoms(Cursor.java:2924)
at com.sleepycat.je.Cursor.retrieveNextNoDups(Cursor.java:2801)
at com.sleepycat.je.Cursor.retrieveNext(Cursor.java:2775)
at com.sleepycat.je.Cursor.getNext(Cursor.java:1128)
at org.apache.qpid.server.store.berkeleydb.BDBMessageStore.getContent(BDBMessageStore.java:1965)
.....
at java.lang.Thread.run(Thread.java:662)

Locked ownable synchronizers:
- locked <5e37baac> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)

- locked <1788a0f5> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)


_"RMI TCP Connection(5062)-10.119.128.156" - Thread t@41101_
java.lang.Thread.State: WAITING
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <148e36dd> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:842)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1178)
at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:807)
at com.sleepycat.je.latch.SharedLatch.acquireExclusive(SharedLatch.java:95)
at com.sleepycat.je.latch.SharedLatch.acquireShared(SharedLatch.java:135)
at com.sleepycat.je.tree.IN.latchShared(IN.java:484)
at com.sleepycat.je.tree.Tree.searchSubTreeInternal(Tree.java:1534)
at com.sleepycat.je.tree.Tree.searchSubTree(Tree.java:1358)
at com.sleepycat.je.tree.Tree.search(Tree.java:1218)
at com.sleepycat.je.dbi.CursorImpl.searchAndPosition(CursorImpl.java:2069)
at com.sleepycat.je.Cursor.searchInternal(Cursor.java:2666)
at com.sleepycat.je.Cursor.searchAllowPhantoms(Cursor.java:2576)
at com.sleepycat.je.Cursor.searchNoDups(Cursor.java:2430)
at com.sleepycat.je.Cursor.search(Cursor.java:2397)
- locked <54bb010e> (a com.sleepycat.je.Cursor)
at com.sleepycat.je.Cursor.getSearchKeyRange(Cursor.java:1727)
at org.apache.qpid.server.store.berkeleydb.BDBMessageStore.getContent(BDBMessageStore.java:1926)
at org.apache.qpid.server.store.berkeleydb.BDBMessageStore$StoredBDBMessage.getContent(BDBMessageStore.java:2410)
at org.apache.qpid.server.message.AMQMessage.getContent(AMQMessage.java:240)
at org.apache.qpid.server.queue.AMQQueueMBean.viewMessageContent(AMQQueueMBean.java:416)
at sun.reflect.GeneratedMethodAccessor166.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:93)
at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:27)
at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:208)
at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:120)
at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:262)
at javax.management.StandardMBean.invoke(StandardMBean.java:391)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:836)
at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:761)
at sun.reflect.GeneratedMethodAccessor22.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.qpid.server.management.MBeanInvocationHandlerImpl.invoke(MBeanInvocationHandlerImpl.java:203)
at $Proxy8.invoke(Unknown Source)
at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1427)
at javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:72)
at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1265)
at java.security.AccessController.doPrivileged(Native Method)
at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1367)
at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:788)
at sun.reflect.GeneratedMethodAccessor20.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:305)
at sun.rmi.transport.Transport$1.run(Transport.java:159)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.Transport.serviceCall(Transport.java:155)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

Locked ownable synchronizers:
- locked <3fe04982> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)


_"IoReceiver - /169.93.75.223:55970" - Thread t@261965_
java.lang.Thread.State: WAITING
at sun.misc.Unsafe.park(Native Method)
- waiting to lock <5e37baac> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync) owned by "RMI TCP Connection(5358)-10.119.152.33" t@41433
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:941)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1261)
at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:594)
at com.sleepycat.je.latch.SharedLatch.acquireShared(SharedLatch.java:149)
at com.sleepycat.je.tree.IN.latchShared(IN.java:484)
at com.sleepycat.je.tree.Tree.searchSplitsAllowed(Tree.java:1271)
at com.sleepycat.je.tree.Tree.findBinForInsert(Tree.java:2205)
at com.sleepycat.je.dbi.CursorImpl.putInternal(CursorImpl.java:834)
at com.sleepycat.je.dbi.CursorImpl.put(CursorImpl.java:779)
at com.sleepycat.je.Cursor.putAllowPhantoms(Cursor.java:2243)
at com.sleepycat.je.Cursor.putNoNotify(Cursor.java:2200)
at com.sleepycat.je.Cursor.putNotify(Cursor.java:2117)
at com.sleepycat.je.Cursor.putNoDups(Cursor.java:2052)
at com.sleepycat.je.Cursor.putInternal(Cursor.java:2020)
- locked <38e6eb39> (a com.sleepycat.je.Transaction)
at com.sleepycat.je.Database.putInternal(Database.java:1324)
at com.sleepycat.je.Database.put(Database.java:1194)
at org.apache.qpid.server.store.berkeleydb.BDBMessageStore.addContent(BDBMessageStore.java:1793)
.....
at java.lang.Thread.run(Thread.java:662)

There are lots of threads similar to "IoReceiver - /169.93.75.223:55970" - Thread t@261965 that are waiting to lock *5e37baac*. There is no other report in the thread dump of what has got lock *469f6330*.

- Has anyone seen this before?
- Is this fixed in 5.0.58?
- I am wondering if this problem is the reason why 5.0.48 is no longer available for download. Can anyone confirm this?
- I can't find a way of reporting bugs directly, other than on this forum. Please let me know if there's a better way for me to report this issue.


Thanks,
Phil

Edited by: philipharvey44 on 25-Jul-2012 04:21