Berkeley DB Family

Hello,

We currently access bdb 4.5 from c++ in a threaded environment (Linux, CentOS5). We're experimenting with a python replacement, accessing a btree table with 13M records. We're using python's bsddb.db module, a thin layer on top of the c api. If we open a db read-only, setting DB_THREAD, things work fine, and access time is very fast - less than 1 millescond/lookup. However, if we access the DB from multiple threads, access time increases, and quickly. For example, if I use 32 threads, the time per lookup is over 10 ms. This might not seem like much, but in the context of our application, it is significant, and it is a 50X slower than single-threaded.

We're trying various workarounds. We're new to Python. We think it might be the Python GIL, in combination with whatever locking happens as a result of setting DB_THREAD. We've seen some evidence of this issue apart from bdb in some artificial tests we came up with... but it seems like python and bdb together have a sort of combined effect that ends up really punishing the use of multiple threads. And multiple threads is something we'd rather not give up, for other reasons. I've also confirmed with behavior with the most recent version of berkeley (4.7.25 right now).

Currently, I'm setting DB_THREAD | DB_INIT_MPOOL | DB_CREATE for the environment, and DB_THREAD | DB_RDONLY for the table (we never write). Also, I call envp->set_mp_mmapsize with a value larger than the size of the table.

Has anyone else had a similar experience? How did you resolve? Any ideas what else we could try?