Database instance fails to start when the other nodes is down
774189Aug 18 2010 — edited Aug 20 2010Hi
I have a frustrating issue.
If we power off one of our two nodes in our RAC, if you then reboot the other node, when it comes up the database instance fails to come online and hence register with the listener.
If both nodes are up and you reboot either node there are no problems and the instance come online and register.
I've done quite a bit of digging and it seems to be that the ASM instance is not coming up properly and as the spfiledatabaseX.ora for the database live on the ASM the server cannot access them, so cannot start the DB.
Initially the ASM instance +ASM1 was up, it then went down, it then came up again....but what is very strange is that it only fails to work when the second node is not available.
If any one has any ideas or pointers I'd be grateful....
Matt
I've included output from various commands I've used below.
Its a 2 node Oracle 10.2.0.4.2 W2K3 system with an iSCSI SAN backend running ASM.
C:\oracle\product\10.2.0\crs\BIN>srvctl start database -d databaseX
PRKP-1001 : Error starting instance databaseX1 on node ser-ref-s1
CRS-1005: Failed to get required resources
CRS-0223: Resource 'ora.databaseX.databaseX1.inst' has placement error.
2010-08-18 07:48:20.578: [ RACG][2052] [6352][2052][ora.databaseX.databaseX1.inst]: clsrcremaction: clsrcrpcreq message returned: 1 ' SQL*Plus: Release 10.2.0.4.0 - Production on Wed Aug 18 07:48:19 2010 Copyright (c) 1982, 2007, Oracle. All Rights Reserved. Enter user-name: Connected to an idle instance. SQL> ORA-01565: error in identifying file '+DATA/databaseX/spfiledatabaseX.ora' ORA-17'
2010-08-18 07:48:20.609: [ RACG][2052] [6352][2052][ora.databaseX.databaseX1.inst]: end for resource = ora.databaseX.databaseX1.inst, action = start, status = 1, time = 1.748s
crs_stat -t (when node 2 is powered down)
Name Type Target State Host
------------------------------------------------------------
ora....e1.inst application ONLINE OFFLINE
ora....e2.inst application ONLINE OFFLINE
ora.databaseX.db application ONLINE OFFLINE
ora....SM1.asm application ONLINE ONLINE ser-ref-s1
ora....S1.lsnr application ONLINE ONLINE ser-ref-s1
ora....-s1.gsd application ONLINE ONLINE ser-ref-s1
ora....-s1.ons application ONLINE ONLINE ser-ref-s1
ora....-s1.vip application ONLINE ONLINE ser-ref-s1
ora....SM2.asm application ONLINE OFFLINE
ora....S2.lsnr application ONLINE OFFLINE
ora....-s2.gsd application ONLINE OFFLINE
ora....-s2.ons application ONLINE OFFLINE
ora....-s2.vip application ONLINE ONLINE ser-ref-s1
I've checked and sqlnet.authentication_services=(NTS)