Using DB_DBT_APPMALLOC without DB_DBT_MALLOC crashes the process
The docs describe DB_DBT_APPMALLOC as a standalone flag:
http://www.oracle.com/technology/documentation/berkeley-db/db/api_c/db_associate.html
However, this code results in a crash in debug mode:
buffer = (u_char*) malloc(tsize);
result->set_data(buffer);
result->set_size(tsize);
result->set_flags(DB_DBT_APPMALLOC);
The crash is in __db_c_put (db_cam.c), when the block is being freed:
for (skeyp = all_skeys; skeyp - all_skeys < s_count; skeyp++)
FREE_IF_NEEDED(dbenv, skeyp);
When looking at memory, I noticed that Berkeley DB allocates 8 additional bytes in front of the block pointed to by all_skeys[1] (data DBT) if DIAGNOSTIC is defined. The first four bytes are the size of the block, including the overhead and the second one is not changed and still contains the 0xDB fill (CLEAR_BYTE).
The problem is, that FREE_IF_NEEDED calls __os_ufree, which ignores the diagnostic block and tries to deallocate the block as if it were allocated with malloc (i.e. 8 bytes into the actual allocated block).
I spent hours tracking down this problem and found out that if DB_DBT_APPMALLOC is combined with DB_DBT_MALLOC, the problem goes away:
buffer = (u_char*) malloc(tsize);
result->set_data(buffer);
result->set_size(tsize);
result->set_flags(DB_DBT_MALLOC | DB_DBT_APPMALLOC);
Is this the intended behavior and can I release the code with this workaround or there is some other way to handle callback-allocated memory?