Hi All,
I'm just trying to restore 2 node RAC database to a single instance database server but getting below error while trying to catalog backup pieces in RMAN repository while I was able to restore the controlfile from that backups that are moved from RAC node.
Source Info:
--------------
Env : 2 Node RAC
DB Version : 11.2.0
OS - OEL 5
Filesystem : ASM
Target Info:
--------------
Env : Single Instance
DB Version : 11.2.0
OS : OEL 5
Filesystem : Non-ASM
1 - I taken RMAN backup of RAC database like below.
RMAN> run{
2> backup as compressed backupset format '/u01/app/rac_backup/%U.bak' database plus archivelog;
3> backup format '/u01/app/rac_backup/ctrl.ctl' current controlfile;
4> }
Starting backup at 07-JUN-16
current log archived
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=49 instance=ractst1 device type=DISK.
.....
.....
channel ORA_DISK_1: starting compressed archived log backup set
channel ORA_DISK_1: specifying archived log(s) in backup set
input archived log thread=1 sequence=30 RECID=49 STAMP=913896843
input archived log thread=2 sequence=25 RECID=48 STAMP=913896843
channel ORA_DISK_1: starting piece 1 at 07-JUN-16
channel ORA_DISK_1: finished piece 1 at 07-JUN-16
piece handle=/u01/app/rac_backup/2br7htck_1_1.bak tag=TAG20160607T121412 comment=NONE
channel ORA_DISK_1: backup set complete, elapsed time: 00:00:03
Finished backup at 07-JUN-16
Starting backup at 07-JUN-16
using channel ORA_DISK_1
channel ORA_DISK_1: starting compressed full datafile backup set
channel ORA_DISK_1: specifying datafile(s) in backup set
input datafile file number=00001 name=+DATA/ractst/datafile/system.256.913464111
input datafile file number=00002 name=+DATA/ractst/datafile/sysaux.257.913464113
input datafile file number=00003 name=+DATA/ractst/datafile/undotbs1.258.913464113
input datafile file number=00005 name=+DATA/ractst/datafile/undotbs2.264.913464273
input datafile file number=00004 name=+DATA/ractst/datafile/users.259.913464113
channel ORA_DISK_1: starting piece 1 at 07-JUN-16
channel ORA_DISK_1: finished piece 1 at 07-JUN-16
piece handle=/u01/app/rac_backup/2cr7htco_1_1.bak tag=TAG20160607T121416 comment=NONE
channel ORA_DISK_1: backup set complete, elapsed time: 00:01:05
Finished backup at 07-JUN-16
Starting backup at 07-JUN-16
current log archived
using channel ORA_DISK_1
channel ORA_DISK_1: starting compressed archived log backup set
channel ORA_DISK_1: specifying archived log(s) in backup set
input archived log thread=2 sequence=26 RECID=51 STAMP=913896922
input archived log thread=1 sequence=31 RECID=50 STAMP=913896922
channel ORA_DISK_1: starting piece 1 at 07-JUN-16
channel ORA_DISK_1: finished piece 1 at 07-JUN-16
piece handle=/u01/app/rac_backup/2dr7htet_1_1.bak tag=TAG20160607T121525 comment=NONE
channel ORA_DISK_1: backup set complete, elapsed time: 00:00:01
Finished backup at 07-JUN-16
Starting backup at 07-JUN-16
using channel ORA_DISK_1
channel ORA_DISK_1: starting full datafile backup set
channel ORA_DISK_1: specifying datafile(s) in backup set
including current control file in backup set
channel ORA_DISK_1: starting piece 1 at 07-JUN-16
channel ORA_DISK_1: finished piece 1 at 07-JUN-16
piece handle=/u01/app/rac_backup/ctrl.ctl tag=TAG20160607T121527 comment=NONE
channel ORA_DISK_1: backup set complete, elapsed time: 00:00:03
Finished backup at 07-JUN-16
Starting Control File and SPFILE Autobackup at 07-JUN-16
piece handle=+FLASH/ractst/autobackup/2016_06_07/s_913896933.292.913896935 comment=NONE
Finished Control File and SPFILE Autobackup at 07-JUN-16
2 - I moved backup files to target server.
3 - Stared the target instance in NOMOUNT State using pfile moved from RAC database.
4 - Restored controlfile from backup piece at target server.
RMAN> restore controlfile from '/u01/app/rac_backup/ctrl.ctl';
Starting restore at 07-JUN-16
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=20 device type=DISK
channel ORA_DISK_1: restoring control file
channel ORA_DISK_1: restore complete, elapsed time: 00:00:01
output file name=/u01/app/oracle/oradata/control01.ctl
Finished restore at 07-JUN-16
5 - Mounted the target database.
RMAN> sql 'alter database mount';
sql statement: alter database mount
released channel: ORA_DISK_1
6 - Problem is here when trying to catalog my backup.
RMAN> catalog start with '/u01/app/rac_backup';
Starting implicit crosscheck backup at 07-JUN-16
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=20 device type=DISK
ORA-03113: end-of-file on communication channel
ORA-01403: no data found
ORA-01403: no data found
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of catalog command at 06/07/2016 12:19:03
ORA-03113: end-of-file on communication channel
ORA-01403: no data found
ORA-01403: no data found
Tried to catalog again and this time got Internal error:
========
RMAN> catalog start with '/u01/app/rac_backup';
Starting implicit crosscheck backup at 07-JUN-16
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-00601: fatal error in recovery manager
RMAN-03012: fatal error during compilation of command
RMAN-03028: fatal error code for command catalog : 600
RMAN-00600: internal error, arguments [10019] [krmxcr] [] [] []
From Alert Log file:
============
Tue Jun 07 12:18:47 2016
alter database mount
Tue Jun 07 12:18:51 2016
Successful mount of redo thread 1, with mount id 522610215
Database mounted in Exclusive Mode
Lost write protection disabled
Completed: alter database mount
Tue Jun 07 12:19:03 2016
ERROR: slave communication error with ASM; terminating process 5326
Errors in file /u01/app/oracle/diag/rdbms/ractst/proddb/trace/proddb_ora_5326.trc:
Tue Jun 07 12:20:45 2016
ERROR: slave communication error with ASM; terminating process 5371
Errors in file /u01/app/oracle/diag/rdbms/ractst/proddb/trace/proddb_ora_5371.trc:
Tue Jun 07 12:32:10 2016
db_recovery_file_dest_size of 3072 MB is 11.17% used. This is a
user-specified limit on the amount of space that will be used by this
database for recovery-related files, and does not reflect the amount of
space available in the underlying filesystem or ASM diskgroup.
From Trace File:
=============
System name: Linux
Node name: clonedb
Release: 2.6.32-300.10.1.el5uek
Version: #1 SMP Wed Feb 22 17:22:40 EST 2012
Machine: i686
Instance name: proddb
Redo thread mounted by this instance: 1
Oracle process number: 20
Unix process pid: 5326, image: oracle@clonedb (TNS V1-V3)
*** 2016-06-07 12:19:03.175
*** SESSION ID:(20.13) 2016-06-07 12:19:03.175
*** CLIENT ID:() 2016-06-07 12:19:03.175
*** SERVICE NAME:() 2016-06-07 12:19:03.175
*** MODULE NAME:(rman@clonedb (TNS V1-V3)) 2016-06-07 12:19:03.175
*** ACTION NAME:(0000006 STARTED62) 2016-06-07 12:19:03.175
ERROR: slave communication error with ASM; terminating process 5326
*** 2016-06-07 12:19:03.176
dbkedDefDump(): Starting a non-incident diagnostic dump (flags=0x0, level=2, mask=0x0)
----- Error Stack Dump -----
----- SQL Statement (None) -----
Current SQL information unavailable - no cursor.
----- PL/SQL Stack -----
----- PL/SQL Call Stack -----
object line object
handle number name
0x5feaa0a8 3670 package body SYS.X$DBMS_BACKUP_RESTORE
0x5feaa0a8 3650 package body SYS.X$DBMS_BACKUP_RESTORE
----- Call Stack Trace -----
calling call entry argument values in hex
location type point (? means dubious value)
-------------------- -------- -------------------- ----------------------------
skdstdst()+41 call kgdsdst() BFC474E8 ? 2 ?
ksedst1()+77 call skdstdst() BFC474E8 ? 0 ? 1 ? ABAD6A0 ?
855B77A ? ABAD6A0 ?
ksedst()+33 call ksedst1() 0 ? 1 ?
dbkedDefDump()+2699 call ksedst() 0 ? 2E0 ? ED ? D ? 3F ?
B6C3139C ?
ksedmp()+47 call dbkedDefDump() 2 ? 0 ?
kfTerminateMe()+75 call ksedmp() 2 ? 3 ? F667C4C ? F699DD4 ?
4 ? BFC47958 ?
--More--
kfncLogical()+433 call kfncSlaveSubmit() BFC47C90 ? 0 ? F6989F8 ?
kfioLogical()+126 call kfncLogical() BFC4948C ? 4C ?
ksfdglbsz()+357 call kfioLogical() BFC4948C ? 4C ? 0 ?
krbprbsz()+97 call ksfdglbsz() BFC4948C ? 4C ? 9 ? 0 ?
krbmdbp()+1788 call krbprbsz() BFC4948C ? 1 ?
krbidbp()+843 call krbmdbp() 56 ? 3F ? 3678DACD ?
BFC4948C ? 0 ? 3678DACC ?
42 ? 1 ? 0 ? 1 ? BFC49754 ?
pevm_icd_call_commo call 00000000 B6C5DAB0 ? C ? B6C34B74 ?
n()+308
pfrinstr_ICAL()+132 call pevm_icd_call_commo B6C5DAB0 ? 0 ? 1 ? 36 ? C ?
n() 0 ? B6C34B74 ?
pfrrun_no_tool()+45 call 00000000 B6C5DAB0 ? 102959FE ?
B6C5DAF0 ?
pfrrun()+846 call pfrrun_no_tool() B6C5DAB0 ? 10291D18 ?
B6C5DAF0 ?
plsql_run()+652 call pfrrun() B6C5DAB0 ? 0 ? 0 ? 0 ? 0 ?
0 ?
pricar()+867 call plsql_run() B6C5DAB0 ? 9A ? B6C2FFC0 ?
0 ? 0 ? B6C5BBE8 ?
pricbr()+472 call pricar() BFC4B018 ? BFC4A5E8 ?
prient2()+921 call pricbr() BFC4B018 ? BFC4A5E8 ? 0 ?
prient()+2527 call prient2() BFC4B018 ? BFC4A5E8 ? A ?
BFC4BD14 ? 0 ?
kkxrpc()+437 call prient() BFC4B018 ? 106DAC60 ?
BFC4BD84 ? 2D ? 0 ? 0 ?
BFC4BD1C ? 19 ? A ?
BFC4BD14 ?
kporpc()+524 call kkxrpc() 4C ? F ? BFC4DA28 ?
opiodr()+958 call 00000000 4C ? F ? BFC4DA28 ?
Please suggest me if I'm missing something in this process that could lead this issue or this is some kind of bug detected.