Cannot start RAC DB using srvctl start database
565009Nov 15 2009 — edited Nov 20 2009Hello Fellows,
We used to have a 10.1.0.4 RAC DB on top of CRS 10.1.0.4 on top of Sun cluster 3.1 on both Sun Solaris SPARC 64-bit 5.9 nodes.
We :
- Unregisted the current RAC DB from the 10.1.0.4 CRS.
- Removed the CRS 10.1.0.4 permanently and cleanly
- Installed a new Clusterware 10.2.0.1 and patched to 10.2.0.4 + patch 8705958.
- CRS servcies are healthy and up :
$ crs_status.sh
ora.tibisdb1.LISTENER_TIBISDB1.lsnr ONLINE ....etc for all other services
....
Now we are trying to upgrade the RAC DB from 10.1.0.4 to 10.2.
When i started the DBUA it i chose the cluster mode and then DBUA tried to startup both DB instances but it failed. To reproduce this, i registed the databse TIBIS with the 2 instances to the CRS and then tried to startup the DB but it failed:
$ crs_status.sh
ora.TIBIS.TIBIS1.inst OFFLINE OFFLINE
ora.TIBIS.TIBIS2.inst OFFLINE OFFLINE
ora.TIBIS.db OFFLINE OFFLINE ...etc
.....
$ srvctl start database -d TIBIS
PRKP-1001 : Error starting instance TIBIS1 on node tibisdb1
tibisdb1:ora.TIBIS.TIBIS1.inst:
tibisdb1:ora.TIBIS.TIBIS1.inst:
SQL*Plus: Release 10.1.0.4.0 ...
Connected to an idle instance.
ORA-00603: ORACLE server session terminated by fatal error
ORA-27504: IPC error creating OSD context
ORA-27300: OS system dependent operation:IPC init failed with status: 65
ORA-27301: OS failure message: Package not installed
ORA-27302: failure occurred at: skgxpcini
ORA-27303: additional information:libskgxpd.so called
tibisdb1:ora.TIBIS.TIBIS1.inst:libskgxp10.so should reference real implementation.
tibisdb1:ora.TIBIS.TIBIS1.inst:SQL> Disconnected
CRS-0215: Could not start resource 'ora.TIBIS.TIBIS1.inst'.
PRKP-1001 : Error starting instance TIBIS2 on node tibisdb2
CRS-0215: Could not start resource 'ora.TIBIS.TIBIS2.inst
BTW, i can open the instances fine through SQLPLUS.
I did a full SRVCTL trace for the command and got the same errors above only.
I also followed also Note :Note : 814896.1
############################################################
1. cd $ORACLE_HOME/rdbms/lib
2. rename the original library (if exists)
mv libskgxp10.so libskgxp10.so.old
3. Relink to configure UDP for IPC
make -f ins_rdbms.mk rac_on ipc_udp ioracle
4. Check whether the library exists
ls -l $ORACLE_HOME/lib/libskgxp10.so
############################################################
Same result when try to SRVCTL START DATABASE -D TIBIS
RECENT CHANGES:
- Unregisted the current 10.1.0.4 RAC DB from the 10.1.0.4 CRS.
- Removed the CRS 10.1.0.4 permanently and cleanly
- Installed a new Clusterware 10.2.0.1 and patched to 10.2.0.4 + patch 8705958.
- Try to upgrade/register the 101 RAC DB to 10.2 .
O/S DETAILS:
$ uname -a
SunOS tibisdb1 5.9 Generic_122300-29 sun4u sparc SUNW,Sun-Fire-V240
CLUSTERWARE:
Sun Cluster 3.1
Please advise,
Thank you.
Feras