We have 3 node RAC of 11.2.0.1 on Redhat 5.2. Recently one node is down and cannot be restarted. The message in alert_log file at the time of shutdown is
Wed Jan 05 22:43:47 2011
Clearing standby activation ID 860457525 (0x33498a35)
The primary database controlfile was created using the
'MAXLOGFILES 192' clause.
There is space for up to 186 standby redo logfiles
Use the following SQL commands on the standby database to create
standby redo logfiles that match the primary database:
ALTER DATABASE ADD STANDBY LOGFILE 'srl1.f' SIZE 52428800;
.......
ALTER DATABASE ADD STANDBY LOGFILE 'srl7.f' SIZE 52428800;
WARNING: OMF is enabled on this database. Creating a physical
standby controlfile, when OMF is enabled on the primary
database, requires manual RMAN intervention to resolve OMF
datafile pathnames.
NOTE: Please refer to the RMAN documentation for procedures
describing how to manually resolve OMF datafile pathnames.
Wed Jan 05 23:53:15 2011
ERROR: slave communication error with ASM; terminating process 15991
Errors in file /u01/app/oracle/diag/rdbms/cchprd/cchprd1/trace/cchprd1_lgwr_15991.trc:
Wed Jan 05 23:53:17 2011
PMON (ospid: 15950): terminating the instance due to error 470
Wed Jan 05 23:53:17 2011
System state dump is made for local instance
System State dumped to trace file /u01/app/oracle/diag/rdbms/cchprd/cchprd1/trace/cchprd1_diag_15958.trc
Wed Jan 05 23:53:18 2011
ORA-1092 : opitsk aborting process
Instance terminated by PMON, pid = 15950
Tried to start it with srvctl, but failed
[oracle@cchORdb1 ~]$ srvctl status database -d cchprd
Instance cchprd1 is not running on node cchordb1
Instance cchprd2 is running on node cchordb2
Instance cchprd3 is running on node cchordb3
[oracle@cchORdb1 ~]$ srvctl start instance -d cchprd -i cchprd1
PRCR-1013 : Failed to start resource ora.cchprd.db
PRCR-1064 : Failed to start resource ora.cchprd.db on node cchordb1
ORA-01031: insufficient privileges
ORA-01034: ORACLE not available
ORA-27101: shared memory realm does not exist
Linux-x86_64 Error: 2: No such file or directory
Process ID: 0
Session ID: 0 Serial number: 0
ORA-01031: insufficient privileges
ORA-01031: insufficient privileges
CRS-2674: Start of 'ora.cchprd.db' on 'cchordb1' failed
ORA-01034: ORACLE not available
ORA-27101: shared memory realm does not exist
Linux-x86_64 Error: 2: No such file or directory
Process ID: 0
Session ID: 0 Serial number: 0
ORA-01031: insufficient privileges
sqlplus did not work either
[oracle@cchORdb1 ~]$ sqlplus / as sysdba
SQL*Plus: Release 11.2.0.1.0 Production on Mon Jan 10 10:33:00 2011
Copyright (c) 1982, 2009, Oracle. All rights reserved.
ERROR:
ORA-01031: insufficient privileges
The ASM is running
rebooted the server, but did not solve the problem.
Why SYSDBA has insufficient privileges problem? How to solve it?
Thanks