Berkeley DB Family

Please, can you help me with the following situation?

I have to program a procedure to perform a "self join" over a primary database with several secondary databases associated to it. Due to the structure of the database, this procedure works over the secondary databases (with support of duplicates). According to the Berkeley DB documentation, retrieval of duplicates requires the use of cursor operations.

I tested my program with small databases, to verify that it produces correct results, and it worked fine. However, when I ran the program over a database with 100,000 records, it produces the following error:

malloc: 4096: Cannot allocate memory
Dbc::get: Cannot allocate memory
malloc: 4096: Cannot allocate memory
Dbc::get: Cannot allocate memory

I am using Berkeley 5.0, and cursors are opened using cursor(NULL, &cursorp, 0) (no transactions, no CDS).

The program generates some output, but it is "truncated". This is a problem, because I need to run this program over databases with 19,000,000 records.

Does cursor size cause memory problems? How can it be fixed?

Thanks in advance for your help!

PS.- The skeleton of the program is the following:

    Dbc *cursorp, *cursorp1, *cursorp2;
    Dbt key, data, data1, data2;

    try {
        index1.getDb().cursor(NULL, &cursorp, 0);

        if (!cursorp->get(&key, &data, DB_FIRST)) {
            do {
                index1.getDb().cursor(NULL, &cursorp1, 0);

                if (!cursorp1->get(&key, &data1, DB_SET)) {
                    do {
                        Record item1(data1.get_data());
                        index2.getDb().cursor(NULL, &cursorp2, 0);

                        if (!cursorp2->get(&key, &data2, DB_SET)) {
                            do {
                                Record item2(data2.get_data());
                                if (!item2.equal(item1)) {
                                    item1.show(); item2.show(); cout << endl;
                                }
                            }
                            while (!cursorp2->get(&key, &data2, DB_NEXT_DUP));
                        }
                    }
                    while (!cursorp1->get(&key, &data1, DB_NEXT_DUP));
                }
            }
            while (!cursorp->get(&key, &data, DB_NEXT_NODUP));
        }
    }
    catch(DbException &e) {
        cerr << "Error in buildJoin" << endl;
        cerr << e.what() << endl;
        return(e.get_errno());
    }
    catch(std::exception &e) {
        cerr << "Error in buildJoin" << endl;
        cerr << e.what() << endl;
        return(-1);
    }
    if (cursorp != NULL) { cursorp->close(); }
    if (cursorp1 != NULL) { cursorp1->close(); }
    if (cursorp2 != NULL) { cursorp2->close(); }
    return (0);

where index1 and index2 are secondary databases

Edited by: user12501055 on May 15, 2010 5:22 PM