Fresh cluster inst., servers reboot, cannot restart clusters or ASM
cayenneAug 10 2010 — edited Aug 23 2010Hello all,
I just installed 11Gr2 cluster over 5 nodes. I used ASM in the installer, to hold the voting disk, etc for it.
I installed the RDBMS binairies successfully across all nodes. NO INSTANCES YET.
A few days went by....
I was getting ready to do post installing patches...and found things looking strange. I found the (working on node1), the clustering systems was not running.
I looked, and the servers (all 5 of them) for some reason had rebooted since install.
I tried starting the cluster:
crsctl start cluster -all.
Took while to return, and then errored with a timeout msg.
I checked to see if it was up:
./crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4530: Communications failure contacting Cluster Synchronization Services daemon
CRS-4533: Event Manager is online
It then dawned on me...maybe ASM wasn't up either?
Nope...not running.
I tried to start it locally on node 1
I set the SID, and tried using sqlplus
I got:
ORA-01078: failure in processing system parameters
LRM-00109: could not open parameter file '/u01/app/oracle/product/11.2.0/dbhome_1/dbs/init+ASM1.ora'
I looked...nothing in that directory at all but a simple init.ora file.
I tried shutting down the cluster with
crsctl stop cluster -all
I got a ton of messages for each node like:
CRS-4548: Unable to connect to CRSD
CRS-2678: 'ora.crsd' on 'node1' has experienced an unrecoverable failure
CRS-0267: Human intervention required to resume its availability.
I'm trying to get through to Oracle support, but they're running slow.
Any ideas here?
I used OUI to create ASM for the cluster...why would it not put an init file there to point to the spfile in ASM?
I'm guessing this is the reason the nodes couldn't talk or sync. Trouble is...how do I start ASM without an init file? I seem to recall there might be a way to create a file to point to the ASM for the spfile, but I'm new to this too...and not sure where to point or the syntax to use.
Will have starting cluster up with no ASM have done any damage...if so, how to fix it?
As you can tell, learning about clusters/RAC and ASM....and I'm not finding good reference materials on troubleshooting. Heck, the install docs are bad enough....
Thank you in advance for any advice or links...
cayenne
ps. this is on RHEL5
Edited by: cayenne on Aug 10, 2010 12:33 PM