Hello all,
We are running an Oracle 11g RAC with 2 nodes. One of the nodes backup is failing with the following RMAN
error, but I'm sure it is an Oracle database issue:
Database Backup status: Failed
Start Time : 2021-02-24 15:22:09
End Time : 2021-02-24 15:28:50
Total Runtime : 6 mins
See log file /oracle/app/oracle/admin/rman_backup/logs/full_backup_devrsadb1.log.1 for more details.
See last lines of log file below:
input archived log thread=2 sequence=9481 RECID=20334 STAMP=1065182545 input archived log thread=2 sequence=9482 RECID=20336 STAMP=1065182575 input archived log thread=1
sequence=11075 RECID=20339 STAMP=1065182640 channel ORA_DISK_4: starting piece 1 at 24-FEB-2021 15:26:55
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-00601: fatal error in recovery manager
RMAN-03004: fatal error during execution of command
ORA-01092: ORACLE instance terminated. Disconnection forced ORACLE error from target database:
ORA-03135: connection lost contact
Process ID: 23459
Here are steps I tried to recover:
RMAN> restore database;
Starting restore at 24-FEB-21
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=18 instance=devrsadb1 device type=DISK
allocated channel: ORA_DISK_2
channel ORA_DISK_2: SID=404 instance=devrsadb1 device type=DISK
allocated channel: ORA_DISK_3
channel ORA_DISK_3: SID=591 instance=devrsadb1 device type=DISK
allocated channel: ORA_DISK_4
channel ORA_DISK_4: SID=786 instance=devrsadb1 device type=DISK
channel ORA_DISK_1: starting datafile backup set restore
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of restore command at 02/24/2021 15:37:59
ORA-19573: cannot obtain sub-exclusive enqueue for datafile 5
RMAN> shutdown immediate;
database closed
database dismounted
Oracle instance shut down
RMAN> startup mount
connected to target database (not started)
Oracle instance started
database mounted
Total System Global Area 1603411968 bytes
Fixed Size 2253664 bytes
Variable Size 822086816 bytes
Database Buffers 771751936 bytes
Redo Buffers 7319552 bytes
RMAN> restore database;
Starting restore at 24-FEB-21
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=577 instance=devrsadb1 device type=DISK
allocated channel: ORA_DISK_2
channel ORA_DISK_2: SID=1152 instance=devrsadb1 device type=DISK
allocated channel: ORA_DISK_3
channel ORA_DISK_3: SID=1338 instance=devrsadb1 device type=DISK
allocated channel: ORA_DISK_4
channel ORA_DISK_4: SID=5 instance=devrsadb1 device type=DISK
channel ORA_DISK_1: starting datafile backup set restore
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of restore command at 02/24/2021 15:41:16
ORA-19573: cannot obtain sub-exclusive enqueue for datafile 5
RMAN> alter database open;
database opened
It appears that datafile 5 is the culprit. Has anyone experienced this issue, and if so, how did you go about trouble-shooting & resolving?
Appreciate any guidance.
Thanks