I have a 2 node oracle RAC system on virtual machines. I had to reboot the servers for some reason. Now when I start the cluster I see both the nodes start cluster resources individually and not able to obtain status of other node's resource. Please see outputs of the following commands:
[root@Node1 bin]# ./crsctl check cluster -all
**************************************************************
Node1:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
Node2:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
On node 1
[root@Node1 bin]# ./crsctl status resource -t
--------------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DGFRA.dg
ONLINE ONLINE Node1
ora.DGQDB1.dg
ONLINE ONLINE Node1
ora.DGOCR.dg
ONLINE ONLINE Node1
ora.DGREDO1.dg
ONLINE ONLINE Node1
ora.DGREDO2.dg
ONLINE ONLINE Node1
ora.LISTENER.lsnr
ONLINE ONLINE Node1
ora.asm
ONLINE ONLINE Node1 Started
ora.gsd
OFFLINE OFFLINE Node1
ora.net1.network
ONLINE ONLINE Node1
ora.ons
ONLINE ONLINE Node1
ora.registry.acfs
ONLINE ONLINE Node1
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE Node1
ora.LISTENER_SCAN2.lsnr
1 ONLINE ONLINE Node1
ora.LISTENER_SCAN3.lsnr
1 ONLINE ONLINE Node1
ora.cvu
1 ONLINE ONLINE Node1
ora.Node1.vip
1 ONLINE ONLINE Node1
ora.Node2.vip
1 ONLINE INTERMEDIATE Node1 FAILED OVER
ora.QDB1.db
1 ONLINE OFFLINE Instance Shutdown
2 ONLINE ONLINE Node1 Open
ora.oc4j
1 ONLINE ONLINE Node1
ora.scan1.vip
1 ONLINE ONLINE Node1
ora.scan2.vip
1 ONLINE ONLINE Node1
ora.scan3.vip
1 ONLINE ONLINE Node1
On node 2
[root@Node2 bin]# ./crsctl status resource -t
--------------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DGFRA.dg
ONLINE ONLINE Node2
ora.DGQDB1.dg
ONLINE ONLINE Node2
ora.DGOCR.dg
ONLINE ONLINE Node2
ora.DGREDO1.dg
ONLINE ONLINE Node2
ora.DGREDO2.dg
ONLINE ONLINE Node2
ora.LISTENER.lsnr
ONLINE ONLINE Node2
ora.asm
ONLINE ONLINE Node2 Started
ora.gsd
OFFLINE OFFLINE Node2
ora.net1.network
ONLINE ONLINE Node2
ora.ons
ONLINE ONLINE Node2
ora.registry.acfs
ONLINE ONLINE Node2
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE Node2
ora.LISTENER_SCAN2.lsnr
1 ONLINE ONLINE Node2
ora.LISTENER_SCAN3.lsnr
1 ONLINE ONLINE Node2
ora.cvu
1 ONLINE ONLINE Node2
ora.Node1.vip
1 ONLINE OFFLINE
ora.Node2.vip
1 ONLINE ONLINE Node2
ora.QDB1.db
1 ONLINE ONLINE Node2 Open
2 ONLINE OFFLINE
ora.oc4j
1 ONLINE ONLINE Node2
ora.scan1.vip
1 ONLINE ONLINE Node2
ora.scan2.vip
1 ONLINE ONLINE Node2
ora.scan3.vip
1 ONLINE ONLINE Node2
[oracle@Node2 ~]$ cluvfy comp nodereach -n Node1 -verbose
Verifying node reachability
Checking node reachability...
Check: Node reachability from node "Node2"
Destination Node Reachable?
------------------------------------ ------------------------
Node1 yes
Result: Node reachability check passed from node "Node2"
Verification of node reachability was successful.
[oracle@Node1 ~]$ cluvfy comp nodereach -n Node2 -verbose
Verifying node reachability
Checking node reachability.....
Check: Node reachability from node "Node1"
Destination Node Reachable?
------------------------------------ ------------------------
Node2 yes
Result: Node reachability check passed from node "Node1"
Verification of node reachability was successful.
When I try to start cluster resource on other node:
[root@Node1 bin]# ./crsctl start resource ora.QDB1.db -n Node2
CRS-2546: Server 'Node2' is not online
CRS-4000: Command Start failed, or completed with errors.
[root@Node2 bin]# ./crsctl start resource ora.QDB1.db -n Node1
CRS-2546: Server 'Node1' is not online
CRS-4000: Command Start failed, or completed with errors.
Any ideas how to go about troubleshooting this ?