We have been using BDB java edition for 5 yrs now and its a great product, it serves our purpose very well.
My question is around cache fragmentation. We have a bdb database with 19 million records. We configure 10gb of heap space. 9gb is consumed. If we make a copy of the database using two separate environments and read from one and write to another and replace the old files with the new and restart our application the amount of memory consumed by the cache drops to 4gb. Is there an explanation for this? Is there a way to compact the data without making a copy (we have dozens of servers with a bdb)? And whats interesting, if I load the cache and dump stats every 100k entries and chart it, not all entries are fragmented (see attached image)
Thanks in advance
orig stats
Cache: Current size, allocations, and eviction activity.
adminBytes=143,773
avgBatchCACHEMODE=0
avgBatchCRITICAL=0
avgBatchDAEMON=0
avgBatchEVICTORTHREAD=0
avgBatchMANUAL=0
cacheTotalBytes=9,438,968,712
dataBytes=9,435,678,791
lockBytes=420
nBINsEvictedCACHEMODE=0
nBINsEvictedCRITICAL=0
nBINsEvictedDAEMON=0
nBINsEvictedEVICTORTHREAD=0
nBINsEvictedMANUAL=0
nBINsFetch=4,017,356
nBINsFetchMiss=2,008,972
nBINsStripped=0
nBatchesCACHEMODE=0
nBatchesCRITICAL=0
nBatchesDAEMON=0
nBatchesEVICTORTHREAD=0
nBatchesMANUAL=0
nCachedBINs=2,008,972
nCachedUpperINs=81,409
nEvictPasses=0
nINCompactKey=884,235
nINNoTarget=1,164
nINSparseTarget=1,210,092
nLNsFetch=19,103,013
nLNsFetchMiss=19,102,106
nNodesEvicted=0
nNodesScanned=0
nNodesSelected=0
nRootNodesEvicted=0
nThreadUnavailable=0
nUpperINsEvictedCACHEMODE=0
nUpperINsEvictedCRITICAL=0
nUpperINsEvictedDAEMON=0
nUpperINsEvictedEVICTORTHREAD=0
nUpperINsEvictedMANUAL=0
nUpperINsFetch=6,350,281
nUpperINsFetchMiss=81,404
requiredEvictBytes=0
sharedCacheTotalBytes=0
stats from copied database:
Cache: Current size, allocations, and eviction activity.
adminBytes=61,341
avgBatchCACHEMODE=0
avgBatchCRITICAL=0
avgBatchDAEMON=0
avgBatchEVICTORTHREAD=0
avgBatchMANUAL=0
cacheTotalBytes=4,313,036,592
dataBytes=4,309,829,103
lockBytes=420
nBINsEvictedCACHEMODE=0
nBINsEvictedCRITICAL=0
nBINsEvictedDAEMON=0
nBINsEvictedEVICTORTHREAD=0
nBINsEvictedMANUAL=0
nBINsFetch=301,191
nBINsFetchMiss=150,409
nBINsStripped=0
nBatchesCACHEMODE=0
nBatchesCRITICAL=0
nBatchesDAEMON=0
nBatchesEVICTORTHREAD=0
nBatchesMANUAL=0
nCachedBINs=150,409
nCachedUpperINs=1,200
nEvictPasses=0
nINCompactKey=151,607
nINNoTarget=11
nINSparseTarget=6
nLNsFetch=19,100,774
nLNsFetchMiss=19,100,389
nNodesEvicted=0
nNodesScanned=0
nNodesSelected=0
nRootNodesEvicted=0
nThreadUnavailable=0
nUpperINsEvictedCACHEMODE=0
nUpperINsEvictedCRITICAL=0
nUpperINsEvictedDAEMON=0
nUpperINsEvictedEVICTORTHREAD=0
nUpperINsEvictedMANUAL=0
nUpperINsFetch=304,360
nUpperINsFetchMiss=1,195
requiredEvictBytes=0
sharedCacheTotalBytes=0