HI oracle 11g ASM RAC under OEL 5.6
before 1 week ago i face issue in my disk and then i create new diskgroup and disk and then i do full restore and it get successful,
today when i found the below error in alert log file
Errors in file /home/app/oracle/diag/rdbms/medi/MEDI1/trace/MEDI1_ora_26970.trc:
ORA-15180: could not open dynamic library ASM Library - Generic Linux, version 2.0.4 (KABI_V2), error [open]
ERROR: error ORA-15180 caught in ASM I/O path
Thu May 16 09:36:28 2013
Errors in file /home/app/oracle/diag/rdbms/medi/MEDI1/trace/MEDI1_ora_26970.trc:
ORA-15180: could not open dynamic library ASM Library - Generic Linux, version 2.0.4 (KABI_V2), error [open]
ERROR: error ORA-15180 caught in ASM I/O path
Thu May 16 09:36:45 2013
Errors in file /home/app/oracle/diag/rdbms/medi/MEDI1/trace/MEDI1_ora_26970.trc:
ORA-15180: could not open dynamic library ASM Library - Generic Linux, version 2.0.4 (KABI_V2), error [open]
ERROR: error ORA-15180 caught in ASM I/O path
Thu May 16 09:39:04 2013
Thread 1 cannot allocate new log, sequence 4224
Checkpoint not complete
Current log# 1 seq# 4223 mem# 0: +DATA/medi/onlinelog/group_1.310.815125761
Current log# 1 seq# 4223 mem# 1: /backup01/oracle/archives/archs/MEDI/onlinelog/o1_mf_1_8rvmgjsn_.log
Thread 1 advanced to log sequence 4224 (LGWR switch)
Current log# 2 seq# 4224 mem# 0: +DATA/medi/onlinelog/group_2.311.815125763
Current log# 2 seq# 4224 mem# 1: /backup01/oracle/archives/archs/MEDI/onlinelog/o1_mf_2_8rvmgldn_.log
Thu May 16 09:39:09 2013
Archived Log entry 54503 added for thread 1 sequence 4223 ID 0xd83eea63 dest 1:
Thu May 16 09:39:29 2013
Errors in file /home/app/oracle/diag/rdbms/medi/MEDI1/trace/MEDI1_ora_26970.trc:
ORA-15180: could not open dynamic library ASM Library - Generic Linux, version 2.0.4 (KABI_V2), error [open]
ERROR: error ORA-15180 caught in ASM I/O path
Thu May 16 09:39:45 2013
Errors in file /home/app/oracle/diag/rdbms/medi/MEDI1/trace/MEDI1_ora_26970.trc:
ORA-15180: could not open dynamic library ASM Library - Generic Linux, version 2.0.4 (KABI_V2), error [open]
ERROR: error ORA-15180 caught in ASM I/O path
Errors in file /home/app/oracle/diag/rdbms/medi/MEDI1/trace/MEDI1_ora_26970.trc:
ORA-00202: control file: '+DATA/medi/controlfile/current.257.815111193'
ORA-15081: failed to submit an I/O operation to a disk
Thu May 16 09:40:22 2013
Errors in file /home/app/oracle/diag/rdbms/medi/MEDI1/trace/MEDI1_ora_26970.trc:
ORA-15180: could not open dynamic library ASM Library - Generic Linux, version 2.0.4 (KABI_V2), error [open]
ERROR: error ORA-15180 caught in ASM I/O path
Errors in file /home/app/oracle/diag/rdbms/medi/MEDI1/trace/MEDI1_ora_26970.trc:
ORA-00202: control file: '+DATA/medi/controlfile/current.257.815111193'
ORA-15081: failed to submit an I/O operation to a disk
and i check the below
[oracle@medi01 bin]$ ./crsctl check cluster -all
**************************************************************
medi01:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
medi02:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
[oracle@medi01 bin]$ ./srvctl status database -d MEDI
Instance MEDI1 is running on node medi01
Instance MEDI2 is running on node medi02
[oracle@medi01 bin]$ ./srvctl status asm -a
ASM is running on medi01,medi02
ASM is enabled.
[oracle@medi01 bin]$
SQL> select status from v$instance;
STATUS
------------
OPEN
SQL> select open_mode from v$database;
OPEN_MODE
--------------------
READ WRITE
SQL> select * from v$recover_file;
no rows selected
SQL>
please advise