GI version: 11.2.0.4
DB Version: 11.2.0.4
OS : Oracle Linux 6.5
We have a 2-node RAC DB and today early morning, in Node1, there was some issue accessing OCR_VOTE diskgroup and the CRS went down.
But the DB seems to be running fine. We came to know this only from OEM alerts saying
"[crsd(27504)]CRS-2765:Resource 'ora.OCR_VOTE_DG.dg' has failed on server 'wmsthprd198'."
-- From Node2
"[ohasd(25660)]CRS-2769:Unable to failover resource 'ora.crsd'. See /grid_infra/product/11.2.0/log/wmsthprd199/alertwmsthprd199.log for details. "
In the morning, when we reached office, We found the following in the grid alert log.
[crsd(27504)]CRS-2765:Resource 'ora.OCR_VOTE_DG.dg' has failed on server 'wmsthprd198'.
2015-11-11 05:36:50.643:
[crsd(27504)]CRS-2878:Failed to restart resource 'ora.OCR_VOTE_DG.dg'
2015-11-11 05:36:50.651:
So, We had to manually mount the OCR_VOTE_DG diskgroup in Node1. But, the CRS seems to be down. A restart of the Node1 will most likely fix the issue. But, I have 2 questions
Question1 (a theoretical one)
CRS is not strictly mandatory to be up in a Node for the DB to functioning. Right ? Of course, I cannot do anythithing with srvctl in this node.But DB instance seems to fine.
Question2.
A restart of the CRS in this node will most likely fix this issue. But the next available maintenance window is only after 36 hours. Is it OK to leave the cluster in this state in this Node ?
[root@wmsthprd198 ~]# /grid_infra/product/11.2.0/bin/crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
[root@wmsthprd198 ~]# /grid_infra/product/11.2.0/bin/crsctl status resource -t
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4000: Command Status failed, or completed with errors.
[root@wmsthprd198 ~]# su - oracle
[oracle@wmsthprd198 ~]$
[oracle@wmsthprd198 ~]$ . oraenv
ORACLE_SID = [oracle] ? BRCSPRD1
The Oracle base has been set to /dboracle/product
[oracle@wmsthprd198 ~]$
[oracle@wmsthprd198 ~]$ srvctl status database -d BRCSPRD1
PRCD-1027 : Failed to retrieve database BRCSPRD1
PRCR-1115 : Failed to find entities of type resource that match filters ((NAME == ora.brcsprd1.db) && (TYPE == ora.database.type)) and contain attributes VERSION,ORACLE_HOME,DATABASE_TYPE
Cannot communicate with crsd
[oracle@wmsthprd198 ~]$
[oracle@wmsthprd198 ~]$ sqlplus / as sysdba
SQL*Plus: Release 11.2.0.4.0 Production on Wed Nov 11 14:21:00 2015
Copyright (c) 1982, 2013, Oracle. All rights reserved.
Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Data Mining and Real Application Testing options
SQL> select INSTANCE_NAME, status from gv$instance;
INSTANCE_NAME STATUS
---------------- ------------
BRCSPRD1 OPEN
BRCSPRD2 OPEN
In Node2, everything is fine ie. CRS stack is up
# /grid_infra/product/11.2.0/bin/crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
#