Oracle 11.2.0.2 SE-One
Oracle Linux 5.6 x86-64
Weekly refresh of a test db from prod, using rman DUPLICATE DATABASE, failed with “RMAN-05556: not all datafiles have backups that can be recovered to SCN”
Background Summary:
Weekly inc 0 backup of production starts on Sunday at 0100, normally completes around 1050. Includes backups of archivelogs
Another backup of just archivelogs runs on Sunday at 1200, normally completes NLT 1201.
On the test server, the refresh job starts on Sunday at 1325. In the past this script used a set until time \"to_date('`date +%Y-%m-%d` 11:55:00','YYYY-MM-DD hh24:mi:ss')\"; -- hard-coded for ‘today at 11:55’.
For a variety of reasons I decided to replace this semi-hard coding of the UNTIL with a value determined by querying the rman catalog, getting the completion time of the inc 0 backup. This tested out just fine in my vbox lab, even when I deliberately drove some updates and log switches during the period the backup was running. But the first time to go live I got the above reported error.
Details:
The key part of the inc 0 backup is this (run from a shell script)
export BACKUP_LOC=/u01/backup/dbprod
$ORACLE_HOME/bin/rman target=/ catalog rman/***@rmcat<<EOF
configure backup optimization on;
configure default device type to disk;
configure retention policy to recovery window of 2 days;
crosscheck backup;
crosscheck archivelog all;
delete noprompt force obsolete;
delete noprompt force expired backup;
delete noprompt force expired archivelog all;
configure controlfile autobackup on;
configure controlfile autobackup format for device type disk to '$BACKUP_LOC/%d_%F_ctl.backup';
CONFIGURE CHANNEL DEVICE TYPE DISK FORMAT '$BACKUP_LOC/%U.rman' MAXPIECESIZE 4096 M;
sql "alter system archive log current";
show all;
backup as compressed backupset archivelog all delete all input format "$BACKUP_LOC/%U.alog";
backup as compressed backupset incremental level 0 database tag tag_dbprod;
sql "alter system archive log current";
backup as compressed backupset archivelog all delete all input format "$BACKUP_LOC/%U.alog";
list recoverable backup;
EOF
The archivelog-only backup (runs at noon) looks like this:
export BACKUP_LOC=/u01/backup/dbprod
$ORACLE_HOME/bin/rman target=/ catalog rman/***@rmcat<<EOF
configure backup optimization on;
configure default device type to disk;
configure retention policy to recovery window of 2 days;
crosscheck backup;
crosscheck archivelog all;
delete noprompt force obsolete;
delete noprompt force expired backup;
delete noprompt force expired archivelog all;
configure controlfile autobackup on;
configure controlfile autobackup format for device type disk to '$BACKUP_LOC/%d_%F_ctl.backup';
CONFIGURE CHANNEL DEVICE TYPE DISK FORMAT '$BACKUP_LOC/%U.rman' MAXPIECESIZE 4096 M;
sql "alter system archive log current";
show all;
backup as compressed backupset archivelog all delete all input format "$BACKUP_LOC/%U.alog";
list recoverable backup;
EOF
And the original refresh looked like this:
>> a step to ftp the backups from the prod server to the test server, and some other housekeeping <<, then
cd /backup/dbtest
echo "connect catalog rman/***@rmcat" > /backup/dbtest/dbtest_refresh.rman
echo "connect target sys/*******@dbprod" >> /backup/dbtest/dbtest_refresh.rman
echo "connect auxiliary /" >> /backup/dbtest/dbtest_refresh.rman
echo "run {" >> /backup/dbtest/dbtest_refresh.rman
echo "set until time \"to_date('`date +%Y-%m-%d` 11:55:00','YYYY-MM-DD hh24:mi:ss')\";" >> /backup/dbtest/dbtest_refresh.rman
echo "duplicate target database to DBTEST;" >> /backup/dbtest/dbtest_refresh.rman
echo "}" >> /backup/dbtest/dbtest_refresh.rman
So, my mod to the refresh was
bkup_point=`sqlplus -s rman/***@rmcat <<EOF1
set echo off verify off feedback off head off pages 0 trimsp on
select to_char(max(completion_time),'yyyy-mm-dd hh24:mi:ss')
from rc_backup_set_details
where db_name='DBPROD'
and backup_type='D'
and incremental_level=0
;
exit
EOF1`
cd /backup/dbtest
echo "connect catalog rman/***@rmcat" > /backup/dbtest/dbtest_refresh.rman
echo "connect target sys/*******@dbprod" >> /backup/dbtest/dbtest_refresh.rman
echo "connect auxiliary /" >> /backup/dbtest/dbtest_refresh.rman
echo "run {" >> /backup/dbtest/dbtest_refresh.rman
echo "set until time \"to_date('${bkup_point}','YYYY-MM-DD hh24:mi:ss')\";" >> /backup/dbtest/dbtest_refresh.rman
echo "duplicate target database to DBTEST;" >> /backup/dbtest/dbtest_refresh.rman
echo "}" >> /backup/dbtest/dbtest_refresh.rman
Now the fun begins.
First, an echo in the refresh script confirmed the ‘bkup_point’:
=======================================================
We will restore to 2013-08-25 10:41:38
=======================================================
Internally, rman reset the ‘until’ as follows:
executing command: SET until clause
Starting Duplicate Db at 25-Aug-2013 15:35:44
allocated channel: ORA_AUX_DISK_1
channel ORA_AUX_DISK_1: SID=162 device type=DISK
contents of Memory Script:
{
set until scn 45633141350;
Examining the result of LIST BACKUP (the last step of all of my rman scripts) the full backup shows this:
BS Key Type LV Size Device Type Elapsed Time Completion Time
------- ---- -- ---------- ----------- ------------ --------------------
5506664 Full 61.89M DISK 00:00:03 25-Aug-2013 02:11:32
BP Key: 5506678 Status: AVAILABLE Compressed: NO Tag: TAG20130825T021129
Piece Name: /u01/backup/dbprod/DBPROD_c-3960114099-20130825-00_ctl.backup
SPFILE Included: Modification time: 24-Aug-2013 22:33:08
SPFILE db_unique_name: DBPROD
Control File Included: Ckp SCN: 45628880455 Ckp time: 25-Aug-2013 02:11:29
BS Key Type LV Size Device Type Elapsed Time Completion Time
------- ---- -- ---------- ----------- ------------ --------------------
5507388 Incr 0 206.03G DISK 08:30:00 25-Aug-2013 10:41:30
List of Datafiles in backup set 5507388
File LV Type Ckp SCN Ckp Time Name
---- -- ---- ---------- -------------------- ----
1 0 Incr 45628880495 25-Aug-2013 02:11:38 +SMALL/dbprod/datafile/system.258.713574775
>>>>>>>>> snip lengthy list <<<<<<<<<
74 0 Incr 45628880495 25-Aug-2013 02:11:38 +SMALL/dbprod/event_i2.dbf
Backup Set Copy #1 of backup set 5507388
Device Type Elapsed Time Completion Time Compressed Tag
----------- ------------ -------------------- ---------- ---
DISK 08:30:00 25-Aug-2013 10:41:36 YES TAG_DBPROD
List of Backup Pieces for backup set 5507388 Copy #1
BP Key Pc# Status Piece Name
------- --- ----------- ----------
5507391 1 AVAILABLE /u01/backup/dbprod/eeoi55iq_1_1.rman
>>>>>>>>>>>>> snip lengthy list <<<<<<<<<<<
5507442 52 AVAILABLE /u01/backup/dbprod/eeoi55iq_52_1.rman
Notice the slight difference in time between what is reported in the LIST BACKUP and what was reported by my query to the catalog.
Continuing with the backup list, the second archivelog backup in the script generated six backupsets. The fifth set showed:
BS Key Size Device Type Elapsed Time Completion Time
------- ---------- ----------- ------------ --------------------
5507687 650.19M DISK 00:02:18 25-Aug-2013 10:54:53
BP Key: 5507694 Status: AVAILABLE Compressed: YES Tag: TAG20130825T104156
Piece Name: /u01/backup/dbprod/ekoi643j_1_1.alog
List of Archived Logs in backup set 5507687
Thrd Seq Low SCN Low Time Next SCN Next Time
---- ------- ---------- -------------------- ---------- ---------
1 1338518 45632944587 25-Aug-2013 05:58:18 45632947563 25-Aug-2013 05:58:20
>>>>>>>>>>>>> snip lengthy list <<<<<<<<<<<
1 1338572 45633135750 25-Aug-2013 10:08:21 45633140240 25-Aug-2013 10:08:24
1 1338573 45633140240 25-Aug-2013 10:08:24 45633141350 25-Aug-2013 10:30:06
1 1338574 45633141350 25-Aug-2013 10:30:06 45633141705 25-Aug-2013 10:41:51
1 1338575 45633141705 25-Aug-2013 10:41:51 45633141725 25-Aug-2013 10:41:55
Notice the availability of the archivelogs including the referenced scn.
Investigation of the ftp portion of the refresh script confirmed that all backup pieces were copied from the prod server.
So what am I overlooking? Having reverted back to the original script to get the refresh completed,