Direct I/O and Linux - any chance for it?
It seems like Berkeley's support for O_DIRECT in Lunux is far from perfect, at least in 2.6.28-11 x86_64. When I build library with --enable-o_direct and use direct_db/direct_log - it gives all sorts of cryptic errors on startup. Looking at internets I'm not the only one with this problem.
Thing is Linux FS cache sucks for servers. It tends to grow into all free memory and then system goes downhill - disks get trashed beyound any reason. At the moment I see load average 9-10 with insane I/O waits after pumping just 3 gigs of data into BDB database from Java app. And it is 8-core box with 12 GB or RAM.
On my desktop Windows PC (which has O_DIRECT working, 1 CPU, 4 GB RAM) same thing does not create any performance issues and even completes faster.
I'm trying to use Berkeley in rather large-scale project, where BDB is to be used as main data store/search index (about 200 GB database, updated with 2 GB of data each hour and should be accessed online).
Maybe someone from Oracle can help with getting O_DIRECT working? Or do I do something horribly wrong here? :)
Technical:
-----
* 2.6.28-11-server #42-Ubuntu SMP Fri Apr 17 02:45:36 UTC 2009 x86_64 GNU/Linux
* 4 x Intel(R) Xeon(R) CPU E5345 @ 2.33GHz (8 cores total)
* 12GB RAM, swapiness=0
* DB is on RAID5, 15 MB/sec average writes.
* db-4.8.24
* configure enable-java enable-o_direct
DB_CONFIG
-----
set_flags DB_TXN_NOSYNC
set_flags DB_TXN_WRITE_NOSYNC
set_flags DB_DIRECT_DB
set_flags DB_DIRECT_LOG
set_flags DB_LOG_AUTOREMOVE
set_verbose DB_VERB_DEADLOCK
set_verbose DB_VERB_RECOVERY
set_lock_timeout 500000
set_txn_timeout 500000
set_lg_max 31457280
set_lg_bsize 104857600
set_cachesize 9 0 10
set_lk_detect DB_LOCK_OLDEST
set_lk_max_lockers 300000
set_lk_max_locks 300000
set_lk_max_objects 300000
-----
Exception:
write: 0x7f5a5009d190, 8192: Invalid argument
write: 0x7f57e5c4e378, 98: Invalid argument
Exception in thread "main" java.lang.IllegalArgumentException: Invalid argument: write: 0x7f5a5009d190, 8192: Invalid argument
write: 0x7f57e5c4e378, 98: Invalid argument
at com.sleepycat.db.internal.db_javaJNI.Db_open(Native Method)
at com.sleepycat.db.internal.Db.open(Db.java:449)
at com.sleepycat.db.DatabaseConfig.openDatabase(DatabaseConfig.java:2106)
at com.sleepycat.db.Environment.openDatabase(Environment.java:314)
at com.sleepycat.compat.DbCompat.openDatabase(DbCompat.java:310)
at com.sleepycat.persist.impl.PersistCatalog.<init>(PersistCatalog.java:183)
at com.sleepycat.persist.impl.Store.<init>(Store.java:178)
at com.sleepycat.persist.EntityStore.<init>(EntityStore.java:109)
-----
Please help.