Okay ,
let me explain whole scenario....
--Installing 11.2.0.3.0 RAC on RHEL 6.5
--Silent Install ( GI 4-node RAC)
--Every prerequisites successful , except resolve.conf ( as we don't use DNS to resolve scan ) in runcluvfy.sh stage -pre .....
--Installation started , remote oprations successful, root.sh script prompted and DAMN ... given below error
=======o/p of root.sh scirpt on-screen=======
User ignored Prerequisites during installation
/u01/app/11.2.0/grid/bin/sqlplus -V ... failed rc=127 with message:
/u01/app/11.2.0/grid/bin/sqlplus: error while loading shared libraries: libsqlplus.so: cannot open shared object file: No such file or directory
Failed to add (property/value):('VERSION/'') for checkpoint:ROOTCRS_STACK.Error code is 256
/u01/app/11.2.0/grid/bin/ocrconfig.bin: symbol lookup error: /u01/app/11.2.0/grid/lib/libclntsh.so.11.1: undefined symbol: nnfyboot
Failed to create or upgrade OLR
/u01/app/11.2.0/grid/perl/bin/perl -I/u01/app/11.2.0/grid/perl/lib -I/u01/app/11.2.0/grid/crs/install /u01/app/11.2.0/grid/crs/install/rootcrs.pl execution failed
======root.sh logs ends======
Now , this issue is been there for almost 20 hours with Oracle Support and i've tried almost possible work-arounds suggested by Oracle Support ( reboot , reinstall , and ..... )
well i don't doubt in Oracle Support by asking in here about issue , but to get chance to have someone had already faced this same issue.
Now , ultimately i wish i've find a culprit which are below o/p.
=========================================
[root@node1 lib]# ls -ltr libsqlplus.so
-rw-r--r-- 1 grid oinstall 1477446 Aug 5 2011 libsqlplus.so
[root@node1 lib]# ls -ltr libclntsh.so.11.1
-rw-r--r-- 1 root oinstall 52697691 Dec 10 00:46 libclntsh.so.11.1
[root@node1 lib]# ls -ltr libnnz11.so
-rw-r--r-- 1 root oinstall 7955322 Aug 17 2011 libnnz11.so
[root@node1 bin]# ls -ltr sqlplus
-rwxr-xr-x 1 grid oinstall 9352 Aug 5 2011 sqlplus
======
[grid@node1 grid]$ ldd /u01/app/11.2.0/grid/bin/sqlplus
linux-vdso.so.1 => (0x00007fff7e72a000)
libsqlplus.so => not found
libclntsh.so.11.1 => not found
libnnz11.so => not found
libdl.so.2 => /lib64/libdl.so.2 (0x00000031f7400000)
libm.so.6 => /lib64/libm.so.6 (0x00000031f8400000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00000031f7c00000)
libnsl.so.1 => /lib64/libnsl.so.1 (0x00000031f9c00000)
libc.so.6 => /lib64/libc.so.6 (0x00000031f7800000)
/lib64/ld-linux-x86-64.so.2 (0x00000031f7000000)
========================================
your questions and comments are welcomed....