I started getting the below alerts by saying Grid DG is Dismounted.
NAME TOTAL_MB/1024 USED FREE_MB/1024 GROUP_NUMBER STATE
------------------------------ ------------- ---------- ------------ ------------ -----------
GRID_DG 0 0 0 0 DISMOUNTED
It is mounted on Another node.
06:27:50 # crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
When i checked ASM alert log file, i found the below:
Wed Apr 20 00:15:06 2016
WARNING: Waited 15 secs for write IO to PST disk 0 in group 1.
WARNING: Waited 15 secs for write IO to PST disk 0 in group 1.
WARNING: Waited 15 secs for write IO to PST disk 0 in group 2.
WARNING: Waited 15 secs for write IO to PST disk 1 in group 2.
WARNING: Waited 15 secs for write IO to PST disk 2 in group 2.
WARNING: Waited 15 secs for write IO to PST disk 0 in group 2.
WARNING: Waited 15 secs for write IO to PST disk 1 in group 2.
WARNING: Waited 15 secs for write IO to PST disk 2 in group 2.
Wed Apr 20 00:15:27 2016
NOTE: process _b000_+asm1 (113712) initiating offline of disk 0.3915943420 (GRID_ASMDISK001) with mask 0x7e in group 2
NOTE: process _b000_+asm1 (113712) initiating offline of disk 1.3915943421 (GRID_ASMDISK002) with mask 0x7e in group 2
NOTE: process _b000_+asm1 (113712) initiating offline of disk 2.3915943422 (GRID_ASMDISK003) with mask 0x7e in group 2
NOTE: checking PST: grp = 2
Wed Apr 20 00:15:27 2016
GMON checking disk modes for group 2 at 13 for pid 34, osid 113712
ERROR: no read quorum in group: required 2, found 0 disks
NOTE: checking PST for grp 2 done.
NOTE: initiating PST update: grp = 2, dsk = 0/0xe9688dfc, mask = 0x6a, op = clear
NOTE: initiating PST update: grp = 2, dsk = 1/0xe9688dfd, mask = 0x6a, op = clear
NOTE: initiating PST update: grp = 2, dsk = 2/0xe9688dfe, mask = 0x6a, op = clear
GMON updating disk modes for group 2 at 14 for pid 34, osid 113712
ERROR: no read quorum in group: required 2, found 0 disks
Wed Apr 20 00:15:27 2016
NOTE: cache dismounting (not clean) group 2/0x13187D19 (GRID_DG)
NOTE: messaging CKPT to quiesce pins Unix process pid: 113739, image: oracle@cernucndb3.cernerasp.com (B001)
Wed Apr 20 00:15:27 2016
NOTE: halting all I/Os to diskgroup 2 (GRID_DG)
Wed Apr 20 00:15:27 2016
NOTE: LGWR doing non-clean dismount of group 2 (GRID_DG)
NOTE: LGWR sync ABA=14.72 last written ABA 14.72
Wed Apr 20 00:15:27 2016
kjbdomdet send to inst 2
detach from dom 2, sending detach message to inst 2
Wed Apr 20 00:15:27 2016
WARNING: Read Failed. group:2 disk:0 AU:0 offset:0 size:4096
WARNING: Read Failed. group:2 disk:1 AU:0 offset:0 size:4096
WARNING: Read Failed. group:2 disk:2 AU:0 offset:0 size:4096
WARNING: Offline for disk GRID_ASMDISK001 in mode 0x7f failed.
WARNING: Offline for disk GRID_ASMDISK002 in mode 0x7f failed.
WARNING: Offline for disk GRID_ASMDISK003 in mode 0x7f failed.
Wed Apr 20 00:15:27 2016
List of instances:
1 2
Dirty detach reconfiguration started (new ddet inc 1, cluster inc 48)
Global Resource Directory partially frozen for dirty detach
* dirty detach - domain 2 invalid = TRUE
130 GCS resources traversed, 0 cancelled
Dirty Detach Reconfiguration complete
Wed Apr 20 00:15:27 2016
ASM Health Checker found 1 new failures
Wed Apr 20 00:15:27 2016
WARNING: dirty detached from domain 2
NOTE: cache dismounted group 2/0x13187D19 (GRID_DG)
SQL> alter diskgroup GRID_DG dismount force /* ASM SERVER:320371993 */
Wed Apr 20 00:15:28 2016
NOTE: cache deleting context for group GRID_DG 2/0x13187d19
GMON dismounting group 2 at 15 for pid 37, osid 113739
NOTE: Disk GRID_ASMDISK001 in mode 0x7f marked for de-assignment
NOTE: Disk GRID_ASMDISK002 in mode 0x7f marked for de-assignment
NOTE: Disk GRID_ASMDISK003 in mode 0x7f marked for de-assignment
NOTE:Waiting for all pending writes to complete before de-registering: grpnum 2
Wed Apr 20 00:15:58 2016
SUCCESS: diskgroup GRID_DG was dismounted
SUCCESS: alter diskgroup GRID_DG dismount force /* ASM SERVER:320371993 */
Wed Apr 20 00:15:58 2016
NOTE: diskgroup resource ora.GRID_DG.dg is offline
SUCCESS: ASM-initiated MANDATORY DISMOUNT of group GRID_DG
Wed Apr 20 00:15:58 2016
Errors in file /u01/oracle/product/diag/asm/+asm/+ASM1/trace/+ASM1_ora_37900.trc:
ORA-15078: ASM diskgroup was forcibly dismounted
Errors in file /u01/oracle/product/diag/asm/+asm/+ASM1/trace/+AS
Please suggest