I have a setup with 4 internal drives and 12 external drives(external storage, multipath). The diskgroup was orignally created on the 4 internal drives and then the disk group altered to add the external storage. Everything was fine until the external storage array got powered down. Then we rebooted both boxes and ASM did not come up. I also see the HEADER_STATUS as UNKNOWN for all the external storage drives.
Tue Oct 14 11:49:34 2008
SQL> ALTER DISKGROUP ALL MOUNT
Tue Oct 14 11:49:34 2008
NOTE: cache registered group DATA number=1 incarn=0xf4a85ef9
Tue Oct 14 11:49:34 2008
Loaded ASM Library - Generic Linux, version 2.0.3 (KABI_V2) library for asmlib interface
Tue Oct 14 11:49:34 2008
ORA-15186: ASMLIB error function = [asm_open], error = [1], mesg = [Operation not permitted]
Tue Oct 14 11:49:34 2008
ORA-15186: ASMLIB error function = [asm_open], error = [1], mesg = [Operation not permitted]
Tue Oct 14 11:49:34 2008
ORA-15186: ASMLIB error function = [asm_open], error = [1], mesg = [Operation not permitted]
Tue Oct 14 11:49:34 2008
ORA-15186: ASMLIB error function = [asm_open], error = [1], mesg = [Operation not permitted]
Tue Oct 14 11:49:34 2008
ORA-15186: ASMLIB error function = [asm_open], error = [1], mesg = [Operation not permitted]
Tue Oct 14 11:49:34 2008
ORA-15186: ASMLIB error function = [asm_open], error = [1], mesg = [Operation not permitted]
Tue Oct 14 11:49:34 2008
ORA-15186: ASMLIB error function = [asm_open], error = [1], mesg = [Operation not permitted]
Tue Oct 14 11:49:34 2008
ORA-15186: ASMLIB error function = [asm_open], error = [1], mesg = [Operation not permitted]
Tue Oct 14 11:49:34 2008
ORA-15186: ASMLIB error function = [asm_open], error = [1], mesg = [Operation not permitted]
Tue Oct 14 11:49:34 2008
ORA-15186: ASMLIB error function = [asm_open], error = [1], mesg = [Operation not permitted]
Tue Oct 14 11:49:34 2008
ORA-15186: ASMLIB error function = [asm_open], error = [1], mesg = [Operation not permitted]
Tue Oct 14 11:49:34 2008
ORA-15186: ASMLIB error function = [asm_open], error = [1], mesg = [Operation not permitted]
Tue Oct 14 11:49:34 2008
NOTE: Hbeat: instance first (grp 1)
Tue Oct 14 11:49:39 2008
NOTE: start heartbeating (grp 1)
Tue Oct 14 11:49:39 2008
NOTE: PST update: grp = 1, dsk = 4, mode = 0x4
NOTE: PST update: grp = 1, dsk = 5, mode = 0x4
NOTE: PST update: grp = 1, dsk = 6, mode = 0x4
NOTE: PST update: grp = 1, dsk = 7, mode = 0x4
NOTE: PST update: grp = 1, dsk = 8, mode = 0x4
NOTE: PST update: grp = 1, dsk = 9, mode = 0x4
NOTE: PST update: grp = 1, dsk = 10, mode = 0x4
NOTE: PST update: grp = 1, dsk = 11, mode = 0x4
NOTE: PST update: grp = 1, dsk = 12, mode = 0x4
NOTE: PST update: grp = 1, dsk = 13, mode = 0x4
NOTE: PST update: grp = 1, dsk = 14, mode = 0x4
NOTE: PST update: grp = 1, dsk = 15, mode = 0x4
Tue Oct 14 11:49:39 2008
ERROR: too many offline disks in PST (grp 1)
Tue Oct 14 11:49:39 2008
NOTE: cache dismounting group 1/0xF4A85EF9 (DATA)
NOTE: dbwr not being msg'd to dismount
ERROR: diskgroup DATA was not mounted
Tue Oct 14 11:49:39 2008
WARNING: PST-initiated MANDATORY DISMOUNT of group DATA not performed - group not mounted
Tue Oct 14 11:49:39 2008
Errors in file /home/oracle/admin/+ASM/bdump/+asm_b000_6358.trc:
ORA-15001: diskgroup "DATA" does not exist or is not mounted
Tue Oct 14 12:31:17 2008
Shutting down instance: further logons disabled
PATH NAME HEADER_STATUS MOUNT_STATUS MODE_STATUS STATE GROUP_NUMBER DISK_NUMBER FREE_MB READ_ERRS WRITE_ERRS
----------- ------ ------------- ------------ ----------- ------- ------------ ----------- -------- ---------- ----------
ORCL:DISK1 MEMBER CLOSED ONLINE NORMAL 0 0 0
ORCL:DISK9 UNKNOWN CLOSED ONLINE NORMAL 0 15 0
ORCL:DISK11 UNKNOWN CLOSED ONLINE NORMAL 0 2 0
ORCL:DISK12 UNKNOWN CLOSED ONLINE NORMAL 0 3 0
ORCL:DISK13 UNKNOWN CLOSED ONLINE NORMAL 0 4 0
ORCL:DISK14 UNKNOWN CLOSED ONLINE NORMAL 0 5 0
ORCL:DISK15 UNKNOWN CLOSED ONLINE NORMAL 0 6 0
ORCL:DISK16 UNKNOWN CLOSED ONLINE NORMAL 0 7 0
ORCL:DISK2 MEMBER CLOSED ONLINE NORMAL 0 8 0
ORCL:DISK3 MEMBER CLOSED ONLINE NORMAL 0 9 0
ORCL:DISK4 MEMBER CLOSED ONLINE NORMAL 0 10 0
ORCL:DISK5 UNKNOWN CLOSED ONLINE NORMAL 0 11 0
ORCL:DISK6 UNKNOWN CLOSED ONLINE NORMAL 0 12 0
ORCL:DISK7 UNKNOWN CLOSED ONLINE NORMAL 0 13 0
ORCL:DISK8 UNKNOWN CLOSED ONLINE NORMAL 0 14 0
ORCL:DISK10 UNKNOWN CLOSED ONLINE NORMAL 0 1 0
16 rows selected.