Hi Support ,
We are having two node RAC, due to some hardware issue both servers goes down. Both servers came up now but RAC Services/Cluster has bee started in only one node where as in other node we are facing issue.
OS: Solaris.
version Oracle: 10.2.0
Refer the logs:
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
oracle$ crsctl check crs
Failure 1 contacting CSS daemon
Cannot communicate with CRS
Cannot communicate with EVM
oracle$ crsctl start crs
Insufficient user privileges. {Using Oracle user}
root$ crsctl start crs
crsctl: not found {Using root user}
root$/etc/init.d/init.crs start
Startup will be queued to init within 30 seconds.
CRS logs:
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
2014-04-04 18:20:02.741: [ CRSRTI][1] CSS is not ready. Received status 3 from CSS. Waiting for good status ..
2014-04-04 18:20:03.748: [ COMMCRS][2058]clsc_connect: (100a513d0) no listener at (ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_ch1d5bl0_localhost))
2014-04-04 18:20:03.748: [ CSSCLNT][1]clsssInitNative: connect failed, rc 9
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
CSS logs: Are not getting generated after march 2013.
I have verified the permissions and ownership of crs/bin, it seems ok. /etc/inittab all enteries are availble. OCR and voting disk are in sync/same with correct permissions.
Permissions:
-------------------------------------------------------------------------------------
drwxr-xr-x 2 root oinstall 3072 Mar 12 2013 bin
$ id
uid=102(oracle) gid=200(oinstall)
root$ id
uid=0(root) gid=0(root)
------------------------------------------------------------------------------------
thanks....