Berkeley DB Family

Hi,

I try to iterate through the entire BDB using EntityCursor and found out that even by just iterating through the db, it consumes as much 1.3G of memory already. ( I have around 3M records in db, each record is of size 1.5K).

My question is: do we have better way (in terms of consuming less memory) to do iteration?


Thanks,





---------------------------code---------------------------------------------------------------------
private void loadDB() throws DatabaseException {
int currCount = 0;
EntityCursor<UserInfo> children = accessor.userInfoById.entities();
try {
log.info("Starting using cursor .... ");
for (UserInfo child : children) {
currCount++;
if (currCount % 100000 == 0) {
int maxMemMb = (int) (((double) (Runtime.getRuntime().maxMemory())) / (1000.0 * 1000.0));
int usedMemMb = (int) (((double) (Runtime.getRuntime().totalMemory() - Runtime.getRuntime().freeMemory())) / (1000.0 * 1000.0));

log.info("[" + currCount + "] max(Mb):" + maxMemMb + " used(Mb):" + usedMemMb);
}
}

log.info("Done iterating! " + " Total number of record in startup: " + currCount);
} catch (Throwable e) {

} finally {
children.close();
}
}


----------------------------output------------------------------------------------------

2008-04-22 22:10:05,287 [main] INFO - opening database...
2008-04-22 22:10:08,420 [main] INFO - opening db done!
2008-04-22 22:10:08,426 [main] INFO - Starting using cursor ....
2008-04-22 22:10:33,659 [main] INFO - [100000] max(Mb):13362 used(Mb):148
2008-04-22 22:10:56,564 [main] INFO - [200000] max(Mb):13362 used(Mb):254
2008-04-22 22:11:18,912 [main] INFO - [300000] max(Mb):13362 used(Mb):352
2008-04-22 22:11:41,739 [main] INFO - [400000] max(Mb):13362 used(Mb):402
2008-04-22 22:12:02,571 [main] INFO - [500000] max(Mb):13362 used(Mb):403
2008-04-22 22:12:21,874 [main] INFO - [600000] max(Mb):13362 used(Mb):448
2008-04-22 22:12:41,091 [main] INFO - [700000] max(Mb):13362 used(Mb):560
2008-04-22 22:12:59,610 [main] INFO - [800000] max(Mb):13362 used(Mb):543
2008-04-22 22:13:17,016 [main] INFO - [900000] max(Mb):13362 used(Mb):672
2008-04-22 22:13:35,002 [main] INFO - [1000000] max(Mb):13362 used(Mb):591
2008-04-22 22:13:53,197 [main] INFO - [1100000] max(Mb):13362 used(Mb):634
2008-04-22 22:14:10,402 [main] INFO - [1200000] max(Mb):13362 used(Mb):584
2008-04-22 22:14:27,341 [main] INFO - [1300000] max(Mb):13362 used(Mb):770
2008-04-22 22:14:43,699 [main] INFO - [1400000] max(Mb):13362 used(Mb):825
2008-04-22 22:14:59,792 [main] INFO - [1500000] max(Mb):13362 used(Mb):669
2008-04-22 22:15:15,328 [main] INFO - [1600000] max(Mb):13362 used(Mb):829
2008-04-22 22:15:30,065 [main] INFO - [1700000] max(Mb):13362 used(Mb):917
2008-04-22 22:15:43,661 [main] INFO - [1800000] max(Mb):13362 used(Mb):888
2008-04-22 22:15:57,055 [main] INFO - [1900000] max(Mb):13362 used(Mb):740
2008-04-22 22:16:09,996 [main] INFO - [2000000] max(Mb):13362 used(Mb):1203
2008-04-22 22:16:22,863 [main] INFO - [2100000] max(Mb):13362 used(Mb):865
2008-04-22 22:16:35,563 [main] INFO - [2200000] max(Mb):13362 used(Mb):1179
2008-04-22 22:16:49,530 [main] INFO - [2300000] max(Mb):13362 used(Mb):1519
2008-04-22 22:17:01,695 [main] INFO - [2400000] max(Mb):13362 used(Mb):856
2008-04-22 22:17:13,294 [main] INFO - [2500000] max(Mb):13362 used(Mb):1033
2008-04-22 22:17:25,183 [main] INFO - [2600000] max(Mb):13362 used(Mb):1199
2008-04-22 22:17:36,991 [main] INFO - [2700000] max(Mb):13362 used(Mb):1285
2008-04-22 22:17:48,845 [main] INFO - [2800000] max(Mb):13362 used(Mb):1366
2008-04-22 22:18:00,661 [main] INFO - [2900000] max(Mb):13362 used(Mb):1343
2008-04-22 22:18:12,187 [main] INFO - [3000000] max(Mb):13362 used(Mb):1327
2008-04-22 22:18:18,311 [main] INFO - Done iterating! Total number of record in startup: 3053660

Cursor iteration done: max(Mb):13362 used(Mb):697