Hi Henk,
I want check with you about the debugging flow of vdbench journal verify failure. Please correct me if something is wrong.
1. Vdbench journal verify failed with error below.
13:09:03.785 localhost-0: At least one Data Validation error detected.
13:09:03.793 localhost-0: All data in the disk block (assumed to be 512 bytes long) will
13:09:03.802 localhost-0: be printed in the errorlog; lines that do not compare will be
13:09:03.808 localhost-0: marked with '*'.
13:09:03.816 localhost-0:
13:09:03.824 localhost-0: Byte 0x00 - 0x07: Byte offset of this disk block
13:09:03.829 localhost-0: Byte 0x08 - 0x0f: Timestamp: number of microseconds since 1/1/1970
13:09:03.835 localhost-0: Byte 0x10 : Data Validation key from 1 - 126
13:09:03.844 localhost-0: Byte 0x11 : Checksum of timestamp
13:09:03.851 localhost-0: Byte 0x12 - 0x13: Reserved
13:09:03.861 localhost-0: Byte 0x14 - 0x1b: SD or FSD name in ASCII hexadecimal
13:09:03.869 localhost-0: Byte 0x1c - 0x1f: Reserved
13:09:03.876 localhost-0: Byte 0x20 - 0x1ff: 480 bytes of compression data pattern
13:09:03.883 localhost-0:
13:09:03.891 localhost-0: On the left: the data that was expected ('.' marks unknown value).
13:09:03.896 localhost-0: On the right: the data that was found.
13:09:03.902 localhost-0: Time of first failure: Wednesday, March 23, 2016 13:09:03.725 PDT
13:09:03.909 localhost-0:
13:09:03.916 localhost-0: dvpost: /root/cohesity_nfs_1458717476/vdbench_data/vdb.1_1.dir/vdb_f0013.file fsd1 fsd1 0x138879000 0x08360000 1024 0x0 0x52eb331ca73f9 0x1 0x3 0x4d 0x0 1937408 36028797018963971
13:09:03.923 localhost-0:
13:09:03.933 localhost-0: Data Validation error for fsd=fsd1; FSD lba: 0x140bd9000; DV xfersize: 1024; relative sector in block: 0x00 ( 0)
13:09:03.939 localhost-0: File name: /root/cohesity_nfs_1458717476/vdbench_data/vdb.1_1.dir/vdb_f0013.file; file lba: 0x08360000
13:09:03.945 localhost-0: ===> Data Validation Key miscompare.
13:09:03.950 localhost-0: The sector below was written Wednesday, March 23, 2016 01:34:12.928 PDT
13:09:03.956 localhost-0: 0x000 00000001 40bd9000 ........ ........ 00000001 40bd9000 00052eb3 31ca73f9
13:09:03.963 localhost-0: 0x010 03..0000 66736431 20202020 00000000 024d0000 31647366 20202020 00000000
13:09:03.968 localhost-0: There are no mismatches in bytes 32-511
13:09:03.977 localhost-0: dvpost: /root/cohesity_nfs_1458717476/vdbench_data/vdb.1_1.dir/vdb_f0013.file fsd1 fsd1 0x138879000 0x08360000 1024 0x1 0x52eb331ca73f9 0x1 0x3 0x4d 0x0 1937408 36028797018963971
13:09:03.984 localhost-0:
13:09:03.989 localhost-0: Data Validation error for fsd=fsd1; FSD lba: 0x140bd9200; DV xfersize: 1024; relative sector in block: 0x01 ( 1)
13:09:03.995 localhost-0: File name: /root/cohesity_nfs_1458717476/vdbench_data/vdb.1_1.dir/vdb_f0013.file; file lba: 0x08360000
13:09:04.001 localhost-0: ===> Data Validation Key miscompare.
13:09:04.008 localhost-0: The sector below was written Wednesday, March 23, 2016 01:34:12.928 PDT
13:09:04.015 localhost-0: 0x000 00000001 40bd9200 ........ ........ 00000001 40bd9200 00052eb3 31ca73f9
13:09:04.021 localhost-0: 0x010 03..0000 66736431 20202020 00000000 024d0000 31647366 20202020 00000000
13:09:04.026 localhost-0: There are no mismatches in bytes 32-511
13:09:04.032 localhost-0: 13:09:03.732 op: read lun: /root/cohesity_nfs_1458717476/vdbench_data/vdb.1_1.dir/vdb_f0013.file lba: 137756672 0x08360000 xfer: 1024 errno: 60003: '60003 A Data Validation error was discovered'
13:09:04.040
13:09:04.044 'data_error=1' requested. Abort after last error.
2. According to vdbench, corrupted file and lba is /root/cohesity_nfs_1458717476/vdbench_data/vdb.1_1.dir/vdb_f0013.file; file lba: 0x08360000
lba in decimal is 0x08360000 = (137756672)base10
3. Now, using dd command read 137756672 offset to inspect the data.
dd if=/root/cohesity_nfs_1458717476/vdbench_data/vdb.1_1.dir/vdb_f0013.file of=/tmp/121 ibs=1 obs=1 skip=137756671 count=100
Here I am expecting dd is skipping first 137756671 bytes and retuning next 100 bytes in the file.
4. When I hexdump the data read in step3, there is a mismatch with vdbench output in step 1 and hexdump output in step 3.
[root@stress-vm-template ~]# hexdump /tmp/121
0000000 0100 0000 0000 bd90 b340 052e f900 ca73
0000010 0031 4d00 6602 6473 2031 2020 0020 0000
0000020 0000 0000 0000 0000 0000 0000 0000 0000
0000030 0000 0000 0000 0000 b800 ed9d 003e 0000
Is my understanding of vdbench output correct? please advise. Thanks in advance.