Database Software

We have used Oracle 10G R2 RAC + ASM on Redhat 4 (EMC cx700 Storage)

I found below errors on alert log and can't inserted, updated and deleted datas in database.

Sun May 27 01:12:34 2007
SUCCESS: diskgroup ARCH was mounted
SUCCESS: diskgroup ARCH was dismounted
SUCCESS: diskgroup ARCH was mounted
SUCCESS: diskgroup ARCH was dismounted
SUCCESS: diskgroup ARCH was mounted
SUCCESS: diskgroup ARCH was dismounted
Sun May 27 01:19:11 2007
Errors in file /oracle/product/admin/DB/udump/db3_ora_15854.trc:
ORA-00376: file 5 cannot be read at this time
ORA-01110: data file 5: '+DATA/db/datafile/undotbs3.257.617849281'
ORA-00376: file 5 cannot be read at this time
ORA-01110: data file 5: '+DATA/db/datafile/undotbs3.257.617849281'
ORA-00376: file 5 cannot be read at this time
ORA-01110: data file 5: '+DATA/db/datafile/undotbs3.257.617849281'
ORA-00372: file 5 cannot be modified at this time
ORA-01110: data file 5: '+DATA/db/datafile/undotbs3.257.617849281'

So:
I checked and recovered data file
SQL> select name,status,file# from v$datafile where status ='RECOVER';
NAME
--------------------------------------------------------------------------------
STATUS FILE#
------- ----------
+DATA/db/datafile/undotbs3.257.617849281
RECOVER 5


RMAN> run {
allocate channel t1 type 'SBT_TAPE';
allocate channel t2 type DISK;
recover datafile 5;
}
recover completed.

SQL> alter database datafile 5 online;

Butttttttttttttttttt:


What is going on?

I checked EMC storage, not found any disk error.
I checked alert log of ASM, no found anything.

I don't know What the problem ?
Because I had the same problem 2 days ago.

This day error on undo datafile node 3.
2 days aGo; error on undo node 4.

Please please anybody
What bug or anything wrong ?

Please introduce me

trace file:
/oracle/product/admin/DB/udump/db3_ora_15854.trc
Oracle Database 10g Enterprise Edition Release 10.2.0.3.0 - 64bit Production
With the Partitioning, Real Application Clusters, OLAP and Data Mining options
ORACLE_HOME = /oracle/product/10.2.0/db
System name: Linux
Node name: db03.domain
Release: 2.6.9-42.ELsmp
Version: #1 SMP Wed Jul 12 23:32:02 EDT 2006
Machine: x86_64
Instance name: DB3
Redo thread mounted by this instance: 3
Oracle process number: 65
Unix process pid: 15854, image: oracleDB3@db03.domain

*** SERVICE NAME:(BE) 2007-05-27 01:19:11.574
*** SESSION ID:(591.62658) 2007-05-27 01:19:11.574
*** 2007-05-27 01:19:11.574
ksedmp: internal or fatal error
ORA-00376: file 5 cannot be read at this time
ORA-01110: data file 5: '+DATA/db/datafile/undotbs3.257.617849281'
ORA-00376: file 5 cannot be read at this time
ORA-01110: data file 5: '+DATA/db/datafile/undotbs3.257.617849281'
ORA-00376: file 5 cannot be read at this time
ORA-01110: data file 5: '+DATA/db/datafile/undotbs3.257.617849281'
ORA-00372: file 5 cannot be modified at this time
ORA-01110: data file 5: '+DATA/db/datafile/undotbs3.257.617849281'
Current SQL statement for this session:
INSERT INTO USAGE VALUES (:B1 ,:B2 ,:B3 ,:B4 ,:B5 ,:B6 ,:B7 ,:B8 ,:B9 ,:B10 ,:B11 ,:B12 ,:B13 ,:B14 ,:B15 ,:B16 ,:B17 ,:B18 ,:B19 ,:B20 ,:B21 ,:B22 ,:B23 ,:B24 ,:B25 ,:B26 ,:B27 ,:B28 ,:B29 ,:B30 ,:B31 ,:B32 ,:B33 ,:B34 ,:B35 ,:B36 ,:B37 ,:B38 ,:B39 ,:B40 ,:B41 ,:B42 ,:B43 ,:B44 ,:B45 ,:B46 ,:B47 ,:B48 ,:B49 ,:B50 )
----- PL/SQL Call Stack -----
object line object
handle number name
0x21dc2b4b8 780 package body MGR.AC
0x21e4815b0 3 anonymous block
----- Call Stack Trace -----
calling call entry argument values in hex
location type point (? means dubious value)
-------------------- -------- -------------------- ----------------------------
ksedst()+31 call ksedst1() 000000000 ? 000000001 ?
7FBFFF7290 ? 7FBFFF72F0 ?
7FBFFF7230 ? 000000000 ?
ksedmp()+610 call ksedst() 000000000 ? 000000001 ?
7FBFFF7290 ? 7FBFFF72F0 ?
7FBFFF7230 ? 000000000 ?
ksupop()+3581 call ksedmp() 000000003 ? 000000001 ?
7FBFFF7290 ? 7FBFFF72F0 ?
7FBFFF7230 ? 000000000 ?
opiodr()+3899 call ksupop() 000000002 ? 000000001 ?
7FBFFF7290 ? 7FBFFF72F0 ?
7FBFFF7230 ? 000000000 ?
rpidrus()+198 call opiodr() 000000066 ? 000000006 ?
.....................................................................................................................
.......etc.............................

Message was edited by:
HunterX (Surachart Opun)