Hello All,
Env: 2 node 10.2.0.4 RAC on Solaris 10 SPARC
All of a sudden the DB instance on Node 2 crashed. The alert.log says "Instance terminated by MMAN".
There were no changes recently.
I tried to start it up using
srvctl and it failed with
[SHCL1N02]$ srvctl start instance -d SHCL1_PRMY -i SHCL1N02
PRKP-1001 : Error starting instance SHCL1N02 on node myjpsuolicdbd02
CRS-1028: Dependency analysis failed because of:
CRS-0223: Resource 'ora.SHCL1_prmy.SHCL1N02.inst' has placement error.
[SHCL1N02]$
The nodeapps and crs processes are all running happily, the instance alone has a problem.
I tried
(1)Stop nodeapps, Stop CRS, Start CRS => Still instance on node2 didn't come up.
(2)sqlplus / as sysdba, startup => Instance came up, but the ora.SHCL1_prmy.SHCL1N02.inst is still offline.
Below is the error in the crsd.log file.
2009-11-04 10:18:30.627: [ CRSRES][2970821] CRS-1028: Dependency analysis failed because of:
'Resource in UNKNOWN state: ora.SHCL1_prmy.SHCL1N02.inst'
2009-11-04 10:21:23.269: [ CRSRES][2970843] StopResource: setting CLI values
2009-11-04 10:21:23.340: [ CRSRES][2970843] Attempting to stop `ora.SHCL1_prmy.SHCL1N02.inst` on member `myjpsuolicdbd02`
2009-11-04 10:21:30.478: [ CRSAPP][2970843] StopResource error for ora.SHCL1_prmy.SHCL1N02.inst error code = 1
2009-11-04 10:21:30.502: [ CRSRES][2970843] Stop of `ora.SHCL1_prmy.SHCL1N02.inst` on member `myjpsuolicdbd02` succeeded.
2009-11-04 10:21:49.867: [ CRSRES][2970861] startRunnable: setting CLI values
2009-11-04 10:21:49.895: [ CRSRES][2970861] Attempting to start `ora.SHCL1_prmy.SHCL1N02.inst` on member `myjpsuolicdbd02`
2009-11-04 10:21:55.019: [ CRSAPP][2970861] StartResource error for ora.SHCL1_prmy.SHCL1N02.inst error code = 1
2009-11-04 10:22:00.583: [ CRSAPP][2970861] StopResource error for ora.SHCL1_prmy.SHCL1N02.inst error code = 1
2009-11-04 10:22:00.592: [ CRSRES][2970861] X_OP_StopResourceFailed : Stop Resource failed
(File: rti.cpp, line: 1803
2009-11-04 10:22:00.593: [ CRSRES][2970861][ALERT] `ora.SHCL1_prmy.SHCL1N02.inst` on member `myjpsuolicdbd02` has experienced an unrecoverable failure.
2009-11-04 10:22:00.593: [ CRSRES][2970861] Human intervention required to resume its availability.
Any clue would be much helpful.
TIA,
JJ