Hi All,
While setting up RAC 10g on VMWare with Cent OS 4, i ran into a few issues which were solved by a reboot (last resort)
The problem was after installing the database, while checking the cluster resources using crs_stat -t, the status was showing offline for 2 resources.
[oracle@rac1 bin]$ ./crs_stat -t
Name Type Target State Host
------------------------------------------------------------
ora....SM1.asm application ONLINE OFFLINE rac1
ora....C1.lsnr application ONLINE OFFLINE rac1
ora.rac1.gsd application ONLINE ONLINE rac1
ora.rac1.ons application ONLINE ONLINE rac1
ora.rac1.vip application ONLINE ONLINE rac1
ora....SM2.asm application ONLINE ONLINE rac2
ora....C2.lsnr application ONLINE ONLINE rac2
ora.rac2.gsd application ONLINE ONLINE rac2
ora.rac2.ons application ONLINE ONLINE rac2
ora.rac2.vip application ONLINE ONLINE rac2
The +ASM1 instance and listener on RAC1 was not started when I checked. So i started them manually.
But while starting the instance, i got an error
WARNING: cache failed to read fn=3 indblk=0 from disk(s): 2
ORA-15196: invalid ASM block header [kfc.c:7910] [endian_kfbh] [3] [2147483648] [0 != 1]
NOTE: a corrupted block was dumped to the trace file
System State dumped to trace file /u01/app/oracle/admin/+ASM/bdump/+asm1_lgwr_12518.trc
NOTE: cache initiating offline of disk 2 group 1
WARNING: offlining disk 2.3912953495 (DATA_0002) with mask 0x3
NOTE: PST update: grp = 1, dsk = 2, mode = 0x6
Wed Sep 12 22:43:44 2012
ERROR: too many offline disks in PST (grp 1)
Wed Sep 12 22:43:44 2012
NOTE: PST not enabling heartbeating (grp 1): group dismounted
I was unable to dismount and remount the disk. It was giving an error saying
ORA-15130: diskgroup "DISK" is being dismounted
First of all can anyone tell me what "ora....SM1.asm" and "ora....C1.lsnr" are? Why was the TARGET column showing as online?
I also tried shutting down the resources and then restarting the rawdevice services. But after doing that, while trying to start the resources (crsctl start resources) the command was just hung on both the instances.
I then didn't know what to do and just rebooted both the machines and then saw that everything was up. I managed to create the database and everything is running fine now.
I didn't know where else to look for errors. Can anyone help me figure out the issue please?
Also now after setting everything up, I see the below error in the ASM alert log for both nodes.
Errors in file /u01/app/oracle/admin/+ASM/bdump/+asm1_rbal_9071.trc:
ORA-15183: ASMLIB initialization error [/opt/oracle/extapi/32/asm/orcl/1/libasm.so]
ORA-15183: ASMLIB initialization error [driver/agent not installed]
I checked oracle support doc ID 1054033.1 and tried checking the permissions. But I don't see the file asmadmin present in $ORACLE_HOME/bin
From the support doc
Solution
1. Manually change the oracle binary ownership and permission:
as root user:
# cd $ORACLE_HOME/bin
# chgrp asmadmin oracle
# chmod 6751 oracle
# ls -l oracle
Thanks,
Abhishek.