Berkeley DB Family

Hello,

After making recent changes to our database in response to a previous issue (bug found and patched in BDB; local code fixed by removing both DB_TXN_SNAPSHOT and DB_MULTIVERSION) we've come across a possible deadlock.

BACKGROUND:

This database has been running for over a year in production and I've never seen a hard deadlock. We use the default deadlock detection engine internal to BDB, transactions, and our code supports the processing of deadlocks and subsequent retries and final abandonment when necessary. The transactions in question involve cursors and sequences; we're using cursors to flip through entries in an existing database, and should no match be found for an update, we insert a new record. Before doing so, we grab the "next ID" (primary key) from a sequence we have (attached, as all sequences are, to its own different DB, done on advice from online docs: "For this reason, it is often preferable for sequence objects to be stored in their own database.") and finally insert the new record.

This is a 64-bit Linux machine. There were 4 operational threads at the time; all were waiting on pthread conditions. As I understand it, deadlocks should have been internally detected and returning DB_LOCK_DEADLOCK to us somewhere? Since we're using the default lock detection engine and firing it constantly, we should not require an Nth thread to monitor the lock tables and manually reject deadlocked transactions, etc?

I wasn't sure what to do when the deadlock occured, but I found some forum posts referencing a db_stat -Co and ran it, along with grabbing a core dump which I still have available. I've reverted the DB binary to older DB_MULTIVERSION code as I work on figuring this out, but if there's something else crucial I should have done, I can run the new code again and wait for another deadlock to happen to run additional diagnostics.

Any ideas or assistance is appreciated. Thank you.

DEADLOCK INFORMATION:

THREAD 2: Attempting to 'get' an asset.
(gdb) bt
#0 0x00002aec07e27496 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00002aec078ddeed in __db_pthread_mutex_lock () from /usr/local/BerkeleyDB.4.7/lib/libdb_cxx-4.7.so
#2 0x00002aec078dda8b in __db_tas_mutex_lock () from /usr/local/BerkeleyDB.4.7/lib/libdb_cxx-4.7.so
#3 0x00002aec0795b8f1 in __lock_get_internal () from /usr/local/BerkeleyDB.4.7/lib/libdb_cxx-4.7.so
#4 0x00002aec0795bc42 in __lock_get () from /usr/local/BerkeleyDB.4.7/lib/libdb_cxx-4.7.so
#5 0x00002aec07987d94 in __db_lget () from /usr/local/BerkeleyDB.4.7/lib/libdb_cxx-4.7.so
#6 0x00002aec07914625 in __ham_get_meta () from /usr/local/BerkeleyDB.4.7/lib/libdb_cxx-4.7.so
#7 0x00002aec07908d4b in __hamc_get () from /usr/local/BerkeleyDB.4.7/lib/libdb_cxx-4.7.so
#8 0x00002aec07979e8a in __dbc_get () from /usr/local/BerkeleyDB.4.7/lib/libdb_cxx-4.7.so
#9 0x00002aec0797aa0d in __dbc_pget () from /usr/local/BerkeleyDB.4.7/lib/libdb_cxx-4.7.so
#10 0x00002aec0798654b in __dbc_pget_pp () from /usr/local/BerkeleyDB.4.7/lib/libdb_cxx-4.7.so
#11 0x00002aec078d4dd7 in Dbc::get () from /usr/local/BerkeleyDB.4.7/lib/libdb_cxx-4.7.so

THREAD 5: Attempting to 'set' an asset.
(gdb) bt
#0 0x00002aec07e27496 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00002aec078ddeed in __db_pthread_mutex_lock () from /usr/local/BerkeleyDB.4.7/lib/libdb_cxx-4.7.so
#2 0x00002aec078dda8b in __db_tas_mutex_lock () from /usr/local/BerkeleyDB.4.7/lib/libdb_cxx-4.7.so
#3 0x00002aec079d0c45 in __seq_get () from /usr/local/BerkeleyDB.4.7/lib/libdb_cxx-4.7.so
#4 0x00002aec078dd02e in DbSequence::get () from /usr/local/BerkeleyDB.4.7/lib/libdb_cxx-4.7.so

THREAD 6: Attempting to 'set' an asset.
(gdb) bt
#0 0x00002aec07e27496 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00002aec078ddeed in __db_pthread_mutex_lock () from /usr/local/BerkeleyDB.4.7/lib/libdb_cxx-4.7.so
#2 0x00002aec078dda8b in __db_tas_mutex_lock () from /usr/local/BerkeleyDB.4.7/lib/libdb_cxx-4.7.so
#3 0x00002aec0795b8f1 in __lock_get_internal () from /usr/local/BerkeleyDB.4.7/lib/libdb_cxx-4.7.so
#4 0x00002aec0795bc42 in __lock_get () from /usr/local/BerkeleyDB.4.7/lib/libdb_cxx-4.7.so
#5 0x00002aec07987d94 in __db_lget () from /usr/local/BerkeleyDB.4.7/lib/libdb_cxx-4.7.so
#6 0x00002aec079893ee in __db_new () from /usr/local/BerkeleyDB.4.7/lib/libdb_cxx-4.7.so
#7 0x00002aec0798bb7e in __db_poff () from /usr/local/BerkeleyDB.4.7/lib/libdb_cxx-4.7.so
#8 0x00002aec07918eb3 in __ham_add_el () from /usr/local/BerkeleyDB.4.7/lib/libdb_cxx-4.7.so
#9 0x00002aec07907fe1 in __hamc_put () from /usr/local/BerkeleyDB.4.7/lib/libdb_cxx-4.7.so
#10 0x00002aec0797b6e5 in __dbc_put () from /usr/local/BerkeleyDB.4.7/lib/libdb_cxx-4.7.so
#11 0x00002aec0796ecde in __db_put () from /usr/local/BerkeleyDB.4.7/lib/libdb_cxx-4.7.so
#12 0x00002aec07985e6c in __db_put_pp () from /usr/local/BerkeleyDB.4.7/lib/libdb_cxx-4.7.so
#13 0x00002aec078d395c in Db::put () from /usr/local/BerkeleyDB.4.7/lib/libdb_cxx-4.7.so

THREAD 7: Attempting to 'set' an asset.
(gdb) bt
#0 0x00002aec07e27496 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00002aec078ddeed in __db_pthread_mutex_lock () from /usr/local/BerkeleyDB.4.7/lib/libdb_cxx-4.7.so
#2 0x00002aec078dda8b in __db_tas_mutex_lock () from /usr/local/BerkeleyDB.4.7/lib/libdb_cxx-4.7.so
#3 0x00002aec0795b8f1 in __lock_get_internal () from /usr/local/BerkeleyDB.4.7/lib/libdb_cxx-4.7.so
#4 0x00002aec0795bc42 in __lock_get () from /usr/local/BerkeleyDB.4.7/lib/libdb_cxx-4.7.so
#5 0x00002aec07987d94 in __db_lget () from /usr/local/BerkeleyDB.4.7/lib/libdb_cxx-4.7.so
#6 0x00002aec07915b6a in __ham_lock_bucket () from /usr/local/BerkeleyDB.4.7/lib/libdb_cxx-4.7.so
#7 0x00002aec07915dc7 in __ham_get_cpage () from /usr/local/BerkeleyDB.4.7/lib/libdb_cxx-4.7.so
#8 0x00002aec079075f9 in __ham_lookup () from /usr/local/BerkeleyDB.4.7/lib/libdb_cxx-4.7.so
#9 0x00002aec07908f9f in __hamc_get () from /usr/local/BerkeleyDB.4.7/lib/libdb_cxx-4.7.so
#10 0x00002aec07979e8a in __dbc_get () from /usr/local/BerkeleyDB.4.7/lib/libdb_cxx-4.7.so
#11 0x00002aec07984045 in __db_get () from /usr/local/BerkeleyDB.4.7/lib/libdb_cxx-4.7.so
#12 0x00002aec079d0374 in __seq_update () from /usr/local/BerkeleyDB.4.7/lib/libdb_cxx-4.7.so
#13 0x00002aec079d0c29 in __seq_get () from /usr/local/BerkeleyDB.4.7/lib/libdb_cxx-4.7.so
#14 0x00002aec078dd02e in DbSequence::get () from /usr/local/BerkeleyDB.4.7/lib/libdb_cxx-4.7.so

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Lock REGINFO information:
Lock Region type
5 Region ID
env/__db.005 Region name
0x2b8dbff65000 Original region address
0x2b8dbff65000 Region address
0x2b8dbff65138 Region primary address
0 Region maximum allocation
0 Region allocated
Region allocations: 225009 allocations, 0 failures, 0 frees, 1 longest
REGION_JOIN_OK Region flags
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Locks grouped by object:
Locker Mode Count Status ----------------- Object ---------------
808cfad0 READ 1 HELD db1 page 0
808cfad3 READ 2 HELD db1 page 0
808cfad5 READ 1 HELD db1 page 0

33 READ 1 HELD db1 handle 0

22 READ 1 HELD db2 handle 0

18 READ 1 HELD db3 handle 0

808cfad5 READ 1 HELD db1 page 3906

808cfad3 READ 1 HELD db1 page 6300
808cfad3 WRITE 1 HELD db1 page 6300

808cfad0 READ 1 HELD db1 page 8272

808cfabe WRITE 2 HELD db4 page 82387

808cfad3 WRITE 1 HELD db4 page 42879

808cfad3 WRITE 1 HELD db4 page 42878

808cfad3 WRITE 1 HELD db4 page 42877

808cfad3 WRITE 1 HELD db4 page 42874

808cfad3 WRITE 1 HELD db4 page 42873

808cfad3 WRITE 1 HELD db4 page 42872

808cfad3 WRITE 1 HELD db4 page 42901

808cfad3 WRITE 1 HELD db4 page 42897

808cfad3 WRITE 1 HELD db4 page 42882

808cfad3 WRITE 1 HELD db4 page 42881

808cfad3 WRITE 1 HELD db4 page 42880

808cfad3 WRITE 1 HELD db4 page 42894

808cfad3 WRITE 1 HELD db4 page 43797

1a READ 1 HELD db3sequence handle 0

1c READ 1 HELD db5 handle 0

20 READ 1 HELD db6 handle 0

24 READ 1 HELD db7 handle 0

808cfad3 READ 13 HELD db4 page 0
808cfabe READ 2 HELD db4 page 0
808cfabe WRITE 1 WAIT db4 page 0
808cfad5 READ 1 WAIT db4 page 0

26 READ 1 HELD db4 handle 0

808cfabe READ 2 HELD db4sequence page 0
808cfad0 READ 1 HELD db4sequence page 0

28 READ 1 HELD db4sequence handle 0

808cfabe READ 3 HELD db8 page 0

2a READ 1 HELD db8 handle 0

808cfabe READ 4 HELD db9 page 0
808cfad0 READ 1 HELD db9 page 0
808cfad3 READ 14 HELD db9 page 0
808cfad5 READ 1 HELD db9 page 0

808cfabe READ 1 HELD db4sequence page 2
808cfabe WRITE 1 HELD db4sequence page 2
808cfad0 READ 1 WAIT db4sequence page 2

2e READ 1 HELD db9 handle 0

808cfabe WRITE 2 HELD db9 page 1

808cfad3 READ 2 HELD db1sequence page 0

35 READ 1 HELD db1sequence handle 0

808cfad3 READ 1 HELD db1sequence page 2
808cfad3 WRITE 1 HELD db1sequence page 2

808cfad5 READ 1 HELD db9 page 2833

808cfad0 WRITE 1 HELD db9 page 7946

808cfad3 WRITE 14 HELD db9 page 8250

808cfabe WRITE 3 HELD db8 page 13301