Hey Henk , hope you can help .
- I've ran vdbench (format=yes) from host (CentOS 6.4) with my system
- At the end of the format phase, after reaching the 99% and printing the "Miscellaneous statistics:" vdbench failed with "open failed"
- localhost-0.stdout.html indicates “error: 24”
- “errno 24” means “Too many open files”.
- “ulimits -n” on the linux host indicates 1024.
- We assume (not sure) that the Vdbench has a bug where he leaves too many open files. (at this instance I was running vdbench with 25000 files)
-I've increased the file open limit from 1024 to 10000 and repeated the scenario - didn't reproduced.
-----------------------------------------------------------------------------------------------------
conf file:
data_errors=1
fsd=fsd1,anchor=/mnt/machine005_ib_if,depth=3,width=10,files=25,sizes=(500k,50,50m,50),openflags=directio
fwd=fwd1,fsd=fsd*,operation=write,rdpct=50,xfersizes=(128k,50,4k,50),fileio=random,fileselect=random,threads=3000
rd=rd1,fwd=fwd*,fwdrate=max,elapsed=7200,interval=5,format=yes
-----------------------------------------------------------------------------------------------------
logfile.html:
12:16:25.560
12:16:25.560 **********************************************************
12:16:25.560 Slave localhost-0 aborting: open failed for /mnt/machine005_ib_if/vdb1_4.dir/vdb2_1.dir/vdb3_9.dir/vdb_f0012.file
12:16:25.560 **********************************************************
12:16:25.560
12:16:25.560 SlaveOnMaster terminating: localhost-0
12:16:27.133
12:16:27.133 Slave localhost-0 prematurely terminated.
12:16:27.133
12:16:27.133 Slave aborted. Abort message received:
12:16:27.133 open failed for /mnt/machine005_ib_if/vdb1_4.dir/vdb2_1.dir/vdb3_9.dir/vdb_f0012.file
12:16:27.133
12:16:27.133 Look at file localhost-0.stdout.html for more information.
12:16:27.133 HTML link: <A HREF="localhost-0.stdout.html">localhost-0.stdout.html</A>
12:16:27.133
12:16:27.133 Slave localhost-0 prematurely terminated.
12:16:27.133
java.lang.RuntimeException: Slave localhost-0 prematurely terminated.
at Vdb.common.failure(common.java:307)
at Vdb.SlaveStarter.startSlave(SlaveStarter.java:191)
at Vdb.SlaveStarter.run(SlaveStarter.java:68)
-----------------------------------------------------------------------------------------------------
localhost-0.stdout.html:
12:16:24.236 12:16:24.236 error: 24
12:16:24.237 12:16:24.236 file_open(), open /mnt/machine005_ib_if/vdb1_5.dir/vdb2_8.dir/vdb3_2.dir/vdb_f0022.file failed
12:16:24.237 12:16:24.237 file_open(), open /mnt/machine005_ib_if/vdb1_8.dir/vdb2_10.dir/vdb3_2.dir/vdb_f0003.file failed
12:16:24.237 12:16:24.236 Memory total Java heap: 167.176 MB; Free: 5.402 MB; Used: 161.773 MB;
12:16:24.237 12:16:24.237 file_open(), open /mnt/machine005_ib_if/vdb1_10.dir/vdb2_8.dir/vdb3_1.dir/vdb_f0003.file failed
12:16:24.238 12:16:24.237 file_open(), open /mnt/machine005_ib_if/vdb1_3.dir/vdb2_2.dir/vdb3_2.dir/vdb_f0013.file failed
12:16:24.238 12:16:24.238 file_open(), open /mnt/machine005_ib_if/vdb1_6.dir/vdb2_3.dir/vdb3_1.dir/vdb_f0003.file failed
12:16:24.238 12:16:24.238 file_open(), open /mnt/machine005_ib_if/vdb1_9.dir/vdb2_9.dir/vdb3_6.dir/vdb_f0020.file failed
12:16:24.239 12:16:24.232 Maximum native memory allocation: 1574961152; Current allocation: 1574961152
12:16:24.239 12:16:24.239
12:16:24.239 12:16:24.239 open failed for /mnt/machine005_ib_if/vdb1_4.dir/vdb2_1.dir/vdb3_9.dir/vdb_f0012.file
12:16:24.240 12:16:24.239
12:16:24.242 java.lang.RuntimeException: open failed for /mnt/machine005_ib_if/vdb1_4.dir/vdb2_1.dir/vdb3_9.dir/vdb_f0012.file
12:16:24.242 at Vdb.common.failure(common.java:307)
12:16:24.242 at Vdb.ActiveFile.openFile(ActiveFile.java:168)
12:16:24.242 at Vdb.FwgThread.openForWrite(FwgThread.java:392)
12:16:24.242 at Vdb.FwgThread.openFile(FwgThread.java:360)
12:16:24.242 at Vdb.OpRead.doOperation(OpRead.java:56)
12:16:24.242 at Vdb.OpReadWrite.doOperation(OpReadWrite.java:88)
12:16:24.242 at Vdb.FwgThread.run(FwgThread.java:177)
-----------------------------------------------------------------------------------------------------