Hello All,
Following vdbench workload failed on linux ext4 mounted parttion with errno: INCORRECT_SIZE: 'Vdbench determined that not enough bytes were read or written'
param file:
create_anchors=yes
data_errors=1
validate=yes
fsd=fsd1,anchor=/mnt/local/local-with-delete,depth=4,width=4,files=10,totalsize=25g,distribution=all,size=10m
fwd=fwd1,fsd=fsd1,operation=write,xfersizes=(4k,20,16k,30,256k,30,1088k,20),fileio=random,fileselect=sequential,skew=49,threads=10
fwd=fwd2,fsd=fsd1,operation=read,xfersizes=(4k,20,16k,30,256k,30,1088k,20),fileio=random,fileselect=sequential,skew=49,threads=10
fwd=fwd3,fsd=fsd1,operation=delete,skew=1
fwd=fwd4,fsd=fsd1,operation=create,skew=1
rd=rd1,fwd=(fwd1-fwd4),fwdrate=max,format=yes,elapsed=1800,interval=5
It fails as follows, Can any one please clarify what is wrong here. Am I doing something wrong here in workload file?
Thanks,
Jay
snippet from logfile.html
07:05:44.132 Miscellaneous statistics:
07:05:44.132 (These statistics do not include activity between the last reported interval and shutdown.)
07:05:44.132 FILE_CREATES Files created: 2,560 18/sec
07:05:44.133 DIRECTORY_CREATES Directories created: 340 2/sec
07:05:44.133 WRITE_OPENS Files opened for write activity: 2,560 18/sec
07:05:44.133 DIR_BUSY_MKDIR Directory busy (mkdir): 3 0/sec
07:05:44.133 DIR_EXISTS Directory may not exist (yet): 5 0/sec
07:05:44.133 FILE_CLOSES Close requests: 2,560 18/sec
07:05:44.133
07:05:44.133 Flushing all reports
07:05:44.134 Java Heap in MB. max: 455.500; current: 77.500; used: 30.334; free: 47.166
07:05:44.201 slv=localhost-0 fwd=fwd1 fsd=fsd1 anchor=/mnt/local/local-with-delete threads=10 skew=49.00 operation=write
07:05:44.202 slv=localhost-0 fwd=fwd2 fsd=fsd1 anchor=/mnt/local/local-with-delete threads=10 skew=49.00 operation=read
07:05:44.202 slv=localhost-0 fwd=fwd3 fsd=fsd1 anchor=/mnt/local/local-with-delete threads= 1 skew= 1.00 operation=delete
07:05:44.202 slv=localhost-0 fwd=fwd4 fsd=fsd1 anchor=/mnt/local/local-with-delete threads= 1 skew= 1.00 operation=create
07:05:44.206 Waiting for synchronization of all slaves
07:05:44.374 localhost-0: Created totalsize= 25.0g using 2,560 of 3,400 files for anchor=/mnt/local/local-with-delete
07:05:44.506 Synchronization of all slaves complete
07:05:45.000 Starting RD=rd1; elapsed=1800; fwdrate=max. For loops: None
<a name="_517938326"></a><i><b>07:05:45.001 Starting RD=rd1; elapsed=1800; fwdrate=max. For loops: None</b></i>
Dec 19, 2017 .Interval. .ReqstdOps... ...cpu%... read ....read..... ....write.... ..mb/sec... mb/sec .xfer.. ...mkdir.... ...rmdir.... ...create... ....open.... ...close.... ...delete...
rate resp total sys pct rate resp rate resp read write total size rate resp rate resp rate resp rate resp rate resp rate resp
07:05:50.010 1 10445 1.392 22.1 2.10 67.8 7033.6 1.514 3335.2 1.131 2056 974.6 3031.6 306585 0.0 0.000 0.0 0.000 27.6 0.153 226.8 0.055 223.2 0.008 49.4 2.331
07:05:53.590 killCommand: /opt/vdbench/vdbench SlaveJvm -m localhost -n localhost-10-171219-07.03.21.254 -l localhost-0 -p 5570
07:05:53.590
07:05:53.590 'data_errors=1' requested. Abort rd=rd1 after last error.
07:05:53.590
07:05:53.591 removeKillAtEnd(): Unable to find /opt/vdbench/vdbench SlaveJvm -m localhost -n localhost-10-171219-07.03.21.254 -l localhost-0 -p 5570
java.lang.RuntimeException: 'data_errors=1' requested. Abort rd=rd1 after last error.
at Vdb.common.failure(common.java:335)
at Vdb.common.failure(common.java:284)
at Vdb.ErrorLog.countErrorsOnMaster(ErrorLog.java:213)
at Vdb.SlaveOnMaster.processSlave(SlaveOnMaster.java:204)
at Vdb.SlaveOnMaster.run(SlaveOnMaster.java:42)
errorlog.html
07:05:53.089 localhost-0: 07:05:53.050 07:05:52.553 op: read lun: /mnt/local/local-with-delete/vdb.1_3.dir/vdb.2_2.dir/vdb.3_2.dir/vdb.4_4.dir/vdb_f0005.file lba:
3342336 0x00330000 xfer: 1114112 errno: INCORRECT_SIZE: 'Vdbench determined that not enough bytes were read or written'
localhost-0.stdout.html
07:05:44.385 07:05:44.385 Starting initializeFileAnchor for fsd1
07:05:44.385 07:05:44.385 Reusing existing FileAnchor structure.
07:05:44.385 07:05:44.385 Completed initializeFileAnchor for fsd1: 0.00
07:05:44.414 07:05:44.413 Started 10 threads for fwd=fwd1,fsd=fsd1,operation=write
07:05:44.417 07:05:44.417 Started 10 threads for fwd=fwd2,fsd=fsd1,operation=read
07:05:44.418 07:05:44.418 Started 1 threads for fwd=fwd3,fsd=fsd1,operation=delete
07:05:44.421 07:05:44.421 Started 1 threads for fwd=fwd4,fsd=fsd1,operation=create
07:05:44.421 07:05:44.421 Started 22 FwgThreads
07:05:44.421 07:05:44.421 Waiting for task synchronization
07:05:44.422 07:05:44.421 Requesting full garbage collection
07:05:44.422 07:05:44.422 GcTracker: cum: 4 intv: 0 ms: 0 mss: 0.00% Heap_MB max: 911 curr: 123 used: 50 free: 73
07:05:44.473 07:05:44.472 GcTracker: cum: 6 intv: 2 ms: 48 mss: 0.00% Heap_MB max: 911 curr: 123 used: 19 free: 104
07:05:44.473 07:05:44.473 task_wait_start_complete() end
<a name="_517938326"></a><i><b>07:05:45.001 Starting RD=rd1; elapsed=1800; fwdrate=max. For loops: None</b></i>
07:05:45.002 07:05:45.001 task_run_all(): 23 tasks
07:05:52.552 07:05:52.552 Invalid byte count. Expecting 1114112
07:05:52.553 07:05:52.552 but read only 770048 bytes.
07:05:52.735 07:05:52.734 Invalid byte count. Expecting 1114112
07:05:52.735 07:05:52.734 but read only 737280 bytes.
07:05:52.790 07:05:52.789 Invalid byte count. Expecting 1114112
07:05:52.790 07:05:52.790 but read only 851968 bytes.
07:05:52.887 07:05:52.887 Invalid byte count. Expecting 262144
07:05:52.887 07:05:52.887 but read only 77824 bytes.
07:05:53.051 07:05:53.050 07:05:53.050 07:05:52.553 op: read lun: /mnt/local/local-with-delete/vdb.1_3.dir/vdb.2_2.dir/vdb.3_2.dir/vdb.4_4.dir/vdb_f0005.file lba: 3342336 0x00330000 xfer: 1114112 errno: INCORRECT_SIZE: 'Vdbench determined that not enough bytes were read or written'
07:05:53.051 07:05:53.051 readAndValidate error using file /mnt/local/local-with-delete/vdb.1_3.dir/vdb.2_2.dir/vdb.3_2.dir/vdb.4_4.dir/vdb_f0005.file
07:05:53.051 Error: INCORRECT_SIZE: 'Vdbench determined that not enough bytes were read or written'
07:05:53.051 lba: 3342336
07:05:53.051 xfersize: 1114112
07:05:53.051 blocks_done: 1
07:05:53.051 bytes_done: 16384
07:05:53.051 open_for_read: false
07:05:53.051 fhandle: 30
07:05:53.052 07:05:53.051 07:05:53.051 07:05:53.051 op: read lun: /mnt/local/local-with-delete/vdb.1_1.dir/vdb.2_4.dir/vdb.3_2.dir/vdb_f0004.file lba: 10223616 0x009C0000 xfer: 262144 errno: INCORRECT_SIZE: 'Vdbench determined that not enough bytes were read or written'
07:05:53.052 07:05:53.052 readAndValidate error using file /mnt/local/local-with-delete/vdb.1_1.dir/vdb.2_4.dir/vdb.3_2.dir/vdb_f0004.file
07:05:53.052 Error: INCORRECT_SIZE: 'Vdbench determined that not enough bytes were read or written'
07:05:53.052 lba: 10223616
07:05:53.052 xfersize: 262144
07:05:53.052 blocks_done: 42
07:05:53.052 bytes_done: 9658368
07:05:53.052 open_for_read: false
07:05:53.052 fhandle: 45
07:05:53.052 07:05:53.052 07:05:53.052 07:05:53.052 op: read lun: /mnt/local/local-with-delete/vdb.1_2.dir/vdb.2_3.dir/vdb.3_1.dir/vdb_f0006.file lba: 6684672 0x00660000 xfer: 1114112 errno: INCORRECT_SIZE: 'Vdbench determined that not enough bytes were read or written'
07:05:53.053 07:05:53.053 readAndValidate error using file /mnt/local/local-with-delete/vdb.1_2.dir/vdb.2_3.dir/vdb.3_1.dir/vdb_f0006.file
07:05:53.053 Error: INCORRECT_SIZE: 'Vdbench determined that not enough bytes were read or written'
07:05:53.053 lba: 6684672
07:05:53.053 xfersize: 1114112
07:05:53.053 blocks_done: 7