Database Software

Hi,
I am consistently getting the following error since yesterday while mounting the ASM disk.

1. I created the ASM disks successfully and created the appropriate ASM DiskGroups using asmca utility.
2. I was able to see the diskgroups as mounted from asmcmd tool.
3. All of a sudden, all the diskgroups were dismounted yesterday and I see the following error in the alert logs for all the
diskgroups.
4. This is using ASM11g

As I exhausted all my avenues to fix, I have turned to you for further assistance. Please do the needful.

------------

SQL> /* ASMCMD */ALTER DISKGROUP DataVSPSI MOUNT
NOTE: cache registered group DATAVSPSI number=3 incarn=0x7934bf5d
NOTE: cache began mount (first) of group DATAVSPSI number=3 incarn=0x7934bf5d
NOTE: Assigning number (3,0) to disk (ORCL:D01_VSP_SI)
Thu Dec 27 17:38:29 2012
NOTE: start heartbeating (grp 3)
kfdp_query(DATAVSPSI): 11
kfdp_queryBg(): 11
NOTE: cache opening disk 0 of grp 3: D01_VSP_SI label:D01_VSP_SI
NOTE: F1X0 found on disk 0 au 2 fcn 0.0
NOTE: cache mounting (first) external redundancy group 3/0x7934BF5D (DATAVSPSI)
NOTE: starting recovery of thread=1 ckpt=2.1 group=3 (DATAVSPSI)
WARNING: IO Failed. group:3 disk(number.incarnation):0.0xeae44fae disk_path:ORCL:D01_VSP_SI
AU:4 disk_offset(bytes):4333568 io_size:122880 operation:Read type:asynchronous
result:I/O error process_id:32162
WARNING: IO Failed. group:3 disk(number.incarnation):0.0xeae44fae disk_path:ORCL:D01_VSP_SI
AU:4 disk_offset(bytes):4202496 io_size:131072 operation:Read type:asynchronous
result:I/O error process_id:32162
ORA-15080: synchronous I/O operation to a disk failed
ERROR: ASM recovery failed to read ACD
NOTE: cache initiating offline of disk 0 group DATAVSPSI
NOTE: process 32162 initiating offline of disk 0.3940831150 (D01_VSP_SI) with mask 0x7e in group 3
WARNING: Disk D01_VSP_SI in mode 0x7f is now being taken offline
NOTE: initiating PST update: grp = 3, dsk = 0/0xeae44fae, mode = 0x15
kfdp_updateDsk(): 12
kfdp_updateDskBg(): 12
ERROR: too many offline disks in PST (grp 3)
WARNING: Disk D01_VSP_SI in mode 0x7f offline aborted
Thu Dec 27 17:38:29 2012
NOTE: halting all I/Os to diskgroup DATAVSPSI
NOTE: crash recovery signalled OER-15130
ERROR: ORA-15130 signalled during mount of diskgroup DATAVSPSI
NOTE: cache dismounting (clean) group 3/0x7934BF5D (DATAVSPSI)
NOTE: lgwr not being msg'd to dismount
NOTE: cache dismounted group 3/0x7934BF5D (DATAVSPSI)
...........
-----------------------------------------------

Also, I think the lower level disk is fine as I am able to write to the disk as follows :

[oracle@rmanqa01 trace]$ dd of=/dev/sdp1
Test Test
0+1 records in
0+1 records out
10 bytes (10 B) copied, 4.97435 seconds, 0.0 kB/s
[oracle@rmanqa01 trace]$ id
uid=500(oracle) gid=500(oinstall) groups=6(disk),500(oinstall),501(dba),502(oper),503(asmadmin),504(asmdba),505(asmoper),506(horcm)
[oracle@rmanqa01 trace]$ ls -l /dev/sdp1
brw-rw---- 1 root disk 8, 241 Dec 28 11:59 /dev/sdp1
[oracle@rmanqa01 trace]$

--------------------------------------------------

The oracleasm also lists the disk D01_VSP_SI as follows :

[root@rmanqa01 log]# /etc/init.d/oracleasm listdisks
A01_VSP_SI
ADSK01
ARCH_AMS_SI
D01_VSP_SI
DATA_AMS_SI
DDSK01
DEMO_ARCH
DEMO_DATA
L01_VSP_SI
RDSK01
REDO_AMS_SI
You have new mail in /var/spool/mail/root

-------------------------

[root@rmanqa01 log]# /etc/init.d/oracleasm querydisk D01_VSP_SI
Disk "D01_VSP_SI" is a valid ASM disk
[root@rmanqa01 log]#

-------------------------
kfed tool was giving proper data till yesterday.
Today, I am getting the following :

./kfed read /dev/oracleasm/disks/D01_VSP_SI

kfbh.endian: 84 ; 0x000: 0x54
kfbh.hard: 101 ; 0x001: 0x65
kfbh.type: 115 ; 0x002: *** Unknown Enum ***
kfbh.datfmt: 116 ; 0x003: 0x74
kfbh.block.blk: 1936020512 ; 0x004: T=0 NUMB=0x73655420
kfbh.block.obj: 2147486324 ; 0x008: TYPE=0x8 NUMB=0xa74
kfbh.check: 2886846267 ; 0x00c: 0xac11c73b
kfbh.fcn.base: 0 ; 0x010: 0x00000000
kfbh.fcn.wrap: 0 ; 0x014: 0x00000000
kfbh.spare1: 0 ; 0x018: 0x00000000
kfbh.spare2: 0 ; 0x01c: 0x00000000
ERROR!!!, failed to get the oracore error message
[oracle@rmanqa01 bin]$
--------------------------------------------------------

Please help.

Thanks

V V

Edited by: user13479556 on Dec 28, 2012 12:00 PM