Hi Henk,
I try to run 60 slaves in a parmfile but as long as there has more than 32 slaves. The transaction will be error out. I'm not sure it's time out issue by slaves slow response.
Here are the logs fyi, any suggestion ?
Johnny
Vdbench distribution: vdbench50402
For documentation, see 'vdbench.pdf'.
20:09:32.465 input argument scanned: '-f./parmfile'
20:09:32.476
20:09:32.476 Specifying 'host=(xxx.yyy.com,host_x) has been deprecated.
20:09:32.476 Please specify 'hd=host_x,system=xxx.yyy.com' instead.
20:09:32.476 Vdbench will honor this parameter until the next release.
20:09:32.476
20:09:32.531
20:09:32.534 Note: fwd=fwd1,threads=600,...
20:09:32.534 Note: rd=rd1,fwd=fwd1,...
20:09:32.535 Note: Mismatch between threads requested (600) and used (595)
20:09:32.535
20:09:32.535 Note: Requesting threads across multiple fsds, operations, slaves or hosts
20:09:32.535 may result in fractions of threads.
20:09:32.535 There will be a minimum of one thread for each fsd, operation,
20:09:32.535 slave or host, or integer truncation of the resulting thread count.
20:09:32.535 Contact the author of Vdbench if there are unexplained differences.
20:09:32.535
20:09:32.542 anchor=/home/ubuntu/vdbench/TEST: there will be 8 directories and a maximum of 32 files under this anchor.
20:09:32.542 Estimated maximum size for this anchor: 256k
20:09:32.542
20:09:32.595 Starting slave: ssh 192.168.2.74 -l ubuntu /home/ubuntu/vdbench/vdbench SlaveJvm -m 192.168.2.5 -n 192.168.2.74-10-150318-20.09.32.431 -l fio-drive-1-0 -p 5570
20:09:32.610 Starting slave: ssh 192.168.2.83 -l ubuntu /home/ubuntu/vdbench/vdbench SlaveJvm -m 192.168.2.5 -n 192.168.2.83-18-150318-20.09.32.431 -l fio-drive-10-0 -p 5570
20:09:32.630 Starting slave: ssh 192.168.2.84 -l ubuntu /home/ubuntu/vdbench/vdbench SlaveJvm -m 192.168.2.5 -n 192.168.2.84-19-150318-20.09.32.431 -l fio-drive-11-0 -p 5570
20:09:32.655 Starting slave: ssh 192.168.2.85 -l ubuntu /home/ubuntu/vdbench/vdbench SlaveJvm -m 192.168.2.5 -n 192.168.2.85-20-150318-20.09.32.431 -l fio-drive-12-0 -p 5570
20:09:32.670 Starting slave: ssh 192.168.2.86 -l ubuntu /home/ubuntu/vdbench/vdbench SlaveJvm -m 192.168.2.5 -n 192.168.2.86-21-150318-20.09.32.431 -l fio-drive-13-0 -p 5570
20:09:32.691 Starting slave: ssh 192.168.2.87 -l ubuntu /home/ubuntu/vdbench/vdbench SlaveJvm -m 192.168.2.5 -n 192.168.2.87-22-150318-20.09.32.431 -l fio-drive-14-0 -p 5570
20:09:32.711 Starting slave: ssh 192.168.2.88 -l ubuntu /home/ubuntu/vdbench/vdbench SlaveJvm -m 192.168.2.5 -n 192.168.2.88-23-150318-20.09.32.431 -l fio-drive-15-0 -p 5570
20:09:32.732 Starting slave: ssh 192.168.2.89 -l ubuntu /home/ubuntu/vdbench/vdbench SlaveJvm -m 192.168.2.5 -n 192.168.2.89-24-150318-20.09.32.431 -l fio-drive-16-0 -p 5570
20:09:32.752 Starting slave: ssh 192.168.2.90 -l ubuntu /home/ubuntu/vdbench/vdbench SlaveJvm -m 192.168.2.5 -n 192.168.2.90-25-150318-20.09.32.431 -l fio-drive-17-0 -p 5570
20:09:32.773 Starting slave: ssh 192.168.2.91 -l ubuntu /home/ubuntu/vdbench/vdbench SlaveJvm -m 192.168.2.5 -n 192.168.2.91-26-150318-20.09.32.431 -l fio-drive-18-0 -p 5570
20:09:32.793 Starting slave: ssh 192.168.2.92 -l ubuntu /home/ubuntu/vdbench/vdbench SlaveJvm -m 192.168.2.5 -n 192.168.2.92-27-150318-20.09.32.431 -l fio-drive-19-0 -p 5570
20:09:32.814 Starting slave: ssh 192.168.2.75 -l ubuntu /home/ubuntu/vdbench/vdbench SlaveJvm -m 192.168.2.5 -n 192.168.2.75-11-150318-20.09.32.431 -l fio-drive-2-0 -p 5570
20:09:32.834 Starting slave: ssh 192.168.2.93 -l ubuntu /home/ubuntu/vdbench/vdbench SlaveJvm -m 192.168.2.5 -n 192.168.2.93-28-150318-20.09.32.431 -l fio-drive-20-0 -p 5570
20:09:32.855 Starting slave: ssh 192.168.2.94 -l ubuntu /home/ubuntu/vdbench/vdbench SlaveJvm -m 192.168.2.5 -n 192.168.2.94-29-150318-20.09.32.431 -l fio-drive-21-0 -p 5570
20:09:32.875 Starting slave: ssh 192.168.2.95 -l ubuntu /home/ubuntu/vdbench/vdbench SlaveJvm -m 192.168.2.5 -n 192.168.2.95-30-150318-20.09.32.431 -l fio-drive-22-0 -p 5570
20:09:32.896 Starting slave: ssh 192.168.2.96 -l ubuntu /home/ubuntu/vdbench/vdbench SlaveJvm -m 192.168.2.5 -n 192.168.2.96-31-150318-20.09.32.431 -l fio-drive-23-0 -p 5570
20:09:32.916 Starting slave: ssh 192.168.2.97 -l ubuntu /home/ubuntu/vdbench/vdbench SlaveJvm -m 192.168.2.5 -n 192.168.2.97-32-150318-20.09.32.431 -l fio-drive-24-0 -p 5570
20:09:32.936 Starting slave: ssh 192.168.2.98 -l ubuntu /home/ubuntu/vdbench/vdbench SlaveJvm -m 192.168.2.5 -n 192.168.2.98-33-150318-20.09.32.431 -l fio-drive-25-0 -p 5570
20:09:32.956 Starting slave: ssh 192.168.2.99 -l ubuntu /home/ubuntu/vdbench/vdbench SlaveJvm -m 192.168.2.5 -n 192.168.2.99-34-150318-20.09.32.431 -l fio-drive-26-0 -p 5570
20:09:32.977 Starting slave: ssh 192.168.2.100 -l ubuntu /home/ubuntu/vdbench/vdbench SlaveJvm -m 192.168.2.5 -n 192.168.2.100-35-150318-20.09.32.431 -l fio-drive-27-0 -p 5570
20:09:32.998 Starting slave: ssh 192.168.2.101 -l ubuntu /home/ubuntu/vdbench/vdbench SlaveJvm -m 192.168.2.5 -n 192.168.2.101-36-150318-20.09.32.431 -l fio-drive-28-0 -p 5570
20:09:33.018 Starting slave: ssh 192.168.2.102 -l ubuntu /home/ubuntu/vdbench/vdbench SlaveJvm -m 192.168.2.5 -n 192.168.2.102-37-150318-20.09.32.431 -l fio-drive-29-0 -p 5570
20:09:33.038 Starting slave: ssh 192.168.2.76 -l ubuntu /home/ubuntu/vdbench/vdbench SlaveJvm -m 192.168.2.5 -n 192.168.2.76-12-150318-20.09.32.431 -l fio-drive-3-0 -p 5570
20:09:33.059 Starting slave: ssh 192.168.2.103 -l ubuntu /home/ubuntu/vdbench/vdbench SlaveJvm -m 192.168.2.5 -n 192.168.2.103-38-150318-20.09.32.431 -l fio-drive-30-0 -p 5570
20:09:33.079 Starting slave: ssh 192.168.2.104 -l ubuntu /home/ubuntu/vdbench/vdbench SlaveJvm -m 192.168.2.5 -n 192.168.2.104-39-150318-20.09.32.431 -l fio-drive-31-0 -p 5570
20:09:33.099 Starting slave: ssh 192.168.2.105 -l ubuntu /home/ubuntu/vdbench/vdbench SlaveJvm -m 192.168.2.5 -n 192.168.2.105-40-150318-20.09.32.431 -l fio-drive-32-0 -p 5570
20:09:33.120 Starting slave: ssh 192.168.2.155 -l ubuntu /home/ubuntu/vdbench/vdbench SlaveJvm -m 192.168.2.5 -n 192.168.2.155-41-150318-20.09.32.431 -l fio-drive-33-0 -p 5570
20:09:33.140 Starting slave: ssh 192.168.2.156 -l ubuntu /home/ubuntu/vdbench/vdbench SlaveJvm -m 192.168.2.5 -n 192.168.2.156-42-150318-20.09.32.431 -l fio-drive-34-0 -p 5570
20:09:33.160 Starting slave: ssh 192.168.2.78 -l ubuntu /home/ubuntu/vdbench/vdbench SlaveJvm -m 192.168.2.5 -n 192.168.2.78-13-150318-20.09.32.431 -l fio-drive-5-0 -p 5570
20:09:33.181 Starting slave: ssh 192.168.2.181 -l ubuntu /home/ubuntu/vdbench/vdbench SlaveJvm -m 192.168.2.5 -n 192.168.2.181-43-150318-20.09.32.431 -l fio-drive-59-0 -p 5570
20:09:33.201 Starting slave: ssh 192.168.2.79 -l ubuntu /home/ubuntu/vdbench/vdbench SlaveJvm -m 192.168.2.5 -n 192.168.2.79-14-150318-20.09.32.431 -l fio-drive-6-0 -p 5570
20:09:33.222 Starting slave: ssh 192.168.2.182 -l ubuntu /home/ubuntu/vdbench/vdbench SlaveJvm -m 192.168.2.5 -n 192.168.2.182-44-150318-20.09.32.431 -l fio-drive-60-0 -p 5570
20:09:33.242 Starting slave: ssh 192.168.2.80 -l ubuntu /home/ubuntu/vdbench/vdbench SlaveJvm -m 192.168.2.5 -n 192.168.2.80-15-150318-20.09.32.431 -l fio-drive-7-0 -p 5570
20:09:33.263 Starting slave: ssh 192.168.2.81 -l ubuntu /home/ubuntu/vdbench/vdbench SlaveJvm -m 192.168.2.5 -n 192.168.2.81-16-150318-20.09.32.431 -l fio-drive-8-0 -p 5570
20:09:33.283 Starting slave: ssh 192.168.2.82 -l ubuntu /home/ubuntu/vdbench/vdbench SlaveJvm -m 192.168.2.5 -n 192.168.2.82-17-150318-20.09.32.431 -l fio-drive-9-0 -p 5570
20:09:33.619 Slow getMessage: signon 0 106223 SEND_SIGNON_INFO_TO_MASTER
20:09:33.655 Slow getMessage: signon 0 99154 SEND_SIGNON_INFO_TO_MASTER
20:09:33.695 Slow getMessage: fio-drive-34-0 1 106223 SEND_SIGNON_SUCCESSFUL
20:09:33.731 Slow getMessage: fio-drive-33-0 1 99148 SEND_SIGNON_SUCCESSFUL
20:09:33.929 All slaves are now connected
20:09:33.953 Slow getMessage: fio-drive-34-0 2 106543 HEARTBEAT_MESSAGE
20:09:33.953 Slow getMessage: fio-drive-33-0 2 99468 HEARTBEAT_MESSAGE
20:09:34.019 Slow getMessage: fio-drive-33-0 3 99475 GET_LUN_INFO_FROM_SLAVE
20:09:34.029 Slow getMessage: fio-drive-34-0 3 106548 GET_LUN_INFO_FROM_SLAVE
20:09:34.327 Slow getMessage: fio-drive-33-0 4 99468 STARTING_FILE_STRUCTURE
20:09:34.345 Slow getMessage: fio-drive-34-0 4 106543 STARTING_FILE_STRUCTURE
20:09:34.402 Slow getMessage: fio-drive-33-0 5 99468 ENDING_FILE_STRUCTURE
20:09:34.447 Slow getMessage: fio-drive-33-0 6 99468 SLAVE_READY_TO_GO
20:09:34.451 Slow getMessage: fio-drive-34-0 5 106543 SLAVE_ABORTING
20:09:34.452
20:09:34.452 **********************************************************
20:09:34.452 Slave fio-drive-34-0 aborting: No files available for this slave. Did you ask for more slaves than files?
20:09:34.452 **********************************************************
20:09:34.452
20:09:34.452 Slave fio-drive-1-0 killed by master
20:09:34.452 Slave fio-drive-2-0 killed by master
20:09:34.452 Slave fio-drive-3-0 killed by master
20:09:34.453 Slave fio-drive-5-0 killed by master
20:09:34.453 Slave fio-drive-6-0 killed by master
20:09:34.453 Slave fio-drive-7-0 killed by master
20:09:34.453 Slave fio-drive-8-0 killed by master
20:09:34.453 Slave fio-drive-9-0 killed by master
20:09:34.453 Slave fio-drive-10-0 killed by master
20:09:34.454 Slave fio-drive-11-0 killed by master
20:09:34.454 Slave fio-drive-12-0 killed by master
20:09:34.454 Slave fio-drive-13-0 killed by master
20:09:34.455 Slave fio-drive-14-0 killed by master
20:09:34.457 Slave fio-drive-15-0 killed by master
20:09:34.457 Slave fio-drive-16-0 killed by master
20:09:34.457 Slave fio-drive-17-0 killed by master
20:09:34.457 Slave fio-drive-18-0 killed by master
20:09:34.458 Slave fio-drive-19-0 killed by master
20:09:34.458 Slave fio-drive-20-0 killed by master
20:09:34.458 Slave fio-drive-21-0 killed by master
20:09:34.458 Slave fio-drive-22-0 killed by master
20:09:34.459 Slave fio-drive-23-0 killed by master
20:09:34.460 Slave fio-drive-24-0 killed by master
20:09:34.460 Slave fio-drive-25-0 killed by master
20:09:34.461 Slave fio-drive-26-0 killed by master
20:09:34.461 Slave fio-drive-27-0 killed by master
20:09:34.461 Slave fio-drive-28-0 killed by master
20:09:34.462 Slave fio-drive-29-0 killed by master
20:09:34.462 Slave fio-drive-30-0 killed by master
20:09:34.462 Slave fio-drive-31-0 killed by master
20:09:34.464 Slave fio-drive-32-0 killed by master
20:09:34.464 Slave fio-drive-33-0 killed by master
20:09:34.464 Slave fio-drive-59-0 killed by master
20:09:34.464 Slave fio-drive-60-0 killed by master
20:09:34.464 Slave fio-drive-60-0 killed by master
20:09:36.271
20:09:36.271 Slave fio-drive-34-0 prematurely terminated.
20:09:36.272
20:09:36.272 Slave aborted. Abort message received:
20:09:36.272 No files available for this slave. Did you ask for more slaves than files?
20:09:36.272
20:09:36.272 Look at file fio-drive-34-0.stdout.html for more information.
20:09:36.272
20:09:36.272 Slave fio-drive-34-0 prematurely terminated.
20:09:36.272
java.lang.RuntimeException: Slave fio-drive-34-0 prematurely terminated.
at Vdb.common.failure(common.java:306)
at Vdb.SlaveStarter.startSlave(SlaveStarter.java:179)
at Vdb.SlaveStarter.run(SlaveStarter.java:50)
ubuntu@fio-controller-shyam:~/vdbench$ ./vdbench -f ./parmfile > testresult
java.lang.RuntimeException: Slave fio-drive-34-0 prematurely terminated.
at Vdb.common.failure(common.java:306)
at Vdb.SlaveStarter.startSlave(SlaveStarter.java:179)
at Vdb.SlaveStarter.run(SlaveStarter.java:50)
here is fio-drive-34-0.stdout.html , in case you need more detail.
20:09:51.175 20:10:42.894 SlaveJvm execution parameter: '-m 192.168.2.5'
20:09:51.175 20:10:42.895 SlaveJvm execution parameter: '-n 192.168.2.156-42-150318-20.09.50.012'
20:09:51.175 20:10:42.895 SlaveJvm execution parameter: '-l fio-drive-34-0'
20:09:51.175 20:10:42.895 SlaveJvm execution parameter: '-p 5570'
20:09:51.175 20:10:42.895 SlaveJvm positional parameter: 'SlaveJvm'
20:09:51.193 20:10:42.912 successfully connected to master 192.168.2.5
20:09:51.193 20:10:42.913 Connection to 192.168.2.5 using port 5570 successful
20:09:51.516 20:10:43.236 execute(): /home/ubuntu/vdbench/linux/linux_clock.suse
20:09:51.573 20:10:43.290 ticks_per_second: 100
20:09:51.586 20:10:43.304 execute(): ls -l /home/ubuntu/vdbench/TEST
20:09:51.599 20:10:43.319 Configuration interpretation took 0.08 seconds
Link to Run Definitions: <A HREF="#_131554309">format_for_rd1 For loops: None</A>
<A HREF="#_1272598600">rd1 For loops: None</A>
20:09:51.867 20:10:43.587 File '/var/log/messages' does not exist. Monitoring bypassed
20:09:51.867 20:10:43.587 File '/var/log/messages.1' does not exist. Monitoring bypassed
20:09:51.867 20:10:43.587 Beginning of run setup
20:09:51.867 20:10:43.587 **********************
20:09:51.868
20:09:51.868
20:09:51.910 20:10:43.630 Deleting old file structure.
20:09:51.913 20:10:43.633 Completed the creation of the directory list for /home/ubuntu/vdbench/TEST: 8 directories.
20:09:51.914 20:10:43.634 Memory total Java heap: 123.500 MB; Free: 96.574 MB; Used: 26.926 MB;
20:09:51.985 20:10:43.705 checkMemory()
20:09:51.985 20:10:43.705 Memory total Java heap: 123.500 MB; Free: 108.565 MB; Used: 14.935 MB;
20:09:52.001 20:10:43.721 Generated 0 file names; total anchor size: 0.0
20:09:52.001 20:10:43.721
20:09:52.001 20:10:43.721 No files available for this slave. Did you ask for more slaves than files?
20:09:52.001 20:10:43.721
20:09:52.001 java.lang.RuntimeException: No files available for this slave. Did you ask for more slaves than files?
20:09:52.001 at Vdb.common.failure(common.java:306)
20:09:52.001 at Vdb.FileAnchor.createFileList(FileAnchor.java:389)
20:09:52.001 at Vdb.FileAnchor.initializeFileAnchor(FileAnchor.java:250)
20:09:52.001 at Vdb.FwgRun.startFwg(FwgRun.java:104)
20:09:52.001 at Vdb.SlaveWorker.doFileSystemWorkload(SlaveWorker.java:266)
20:09:52.001 at Vdb.SlaveWorker.run(SlaveWorker.java:133)