Hi Experts,
While taking a RMAN cold backup for one of my dev database I was getting the above error. The script was working completely fine for almost 6months and this is the first time I have faced an error like this.
=====================================================RMAN_COLD Backup Script ======================================
location=/backups/RMAN_COLD/TRCDEV/db
. /mnt/DEVapps/oracle/trcdevdb/11.2.0/dbhome_1/TRCDEV_gddevdb1.env
export ORA_NLS10=/mnt/DEVapps/oracle/trcdevdb/11.2.0/dbhome_1/nls/data/9idata
PATH=$ORACLE_HOME/bin:.:$PATH; export PATH
export NLS_DATE_FORMAT='DD-MON-YY HH24:MI:SS'
time rman nocatalog log /backups/RMAN_COLD/TRCDEV/log/trcdev_rman_cold_log.log <<++RMAN_END++
connect target /
CONFIGURE CONTROLFILE AUTOBACKUP CLEAR;
CONFIGURE CONTROLFILE AUTOBACKUP FORMAT FOR DEVICE TYPE DISK CLEAR;
CONFIGURE CONTROLFILE AUTOBACKUP ON;
CONFIGURE CONTROLFILE AUTOBACKUP FORMAT FOR DEVICE TYPE DISK TO '${location}/cf%F';
shutdown immediate;
startup mount;
run {
allocate channel c0 type disk format '${location}/DF_%d_%U_S%s_T%t';
allocate channel c1 type disk format '${location}/DF_%d_%U_S%s_T%t';
allocate channel c2 type disk format '${location}/DF_%d_%U_S%s_T%t';
allocate channel c3 type disk format '${location}/DF_%d_%U_S%s_T%t';
allocate channel c4 type disk format '${location}/DF_%d_%U_S%s_T%t';
allocate channel c5 type disk format '${location}/DF_%d_%U_S%s_T%t';
allocate channel c6 type disk format '${location}/DF_%d_%U_S%s_T%t';
allocate channel c7 type disk format '${location}/DF_%d_%U_S%s_T%t';
backup
filesperset 50
database;
release channel c0;
release channel c1;
release channel c2;
release channel c3;
release channel c4;
release channel c5;
release channel c6;
release channel c7;
}
run {
allocate channel c0 type disk format '${location}/cf_%d_%U_S%s_T%t';
backup current controlfile format '${location}/cf_%d_%U_S%s_T%t';
release channel c0;
}
shutdown normal;
startup;
exit;
++RMAN_END++
=======================================Script Ends======================================
Contents of RMAN log file:-
Recovery Manager: Release 11.2.0.3.0 - Production on Fri Sep 13 21:12:01 2013
Copyright (c) 1982, 2011, Oracle and/or its affiliates. All rights reserved.
RMAN>
RMAN>
connected to target database: TRCDEV (DBID=3630982182)
using target database control file instead of recovery catalog
..
..
..
RMAN>
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of shutdown command at 09/13/2013 22:12:08
ORA-01013: user requested cancel of current operation
RMAN>
database is already started
RMAN> 2> 3> 4> 5> 6> 7> 8> 9> 10> 11> 12> 13> 14> 15> 16> 17> 18> 19> 20> 21>
DBGANY: CMD type=allocate cmdid=1 status=NOT STARTED
DBGANY: 1 STEPstepid=1 cmdid=1 status=NOT STARTED chid=c0
DBGANY: 1 TEXTNOD = -- devalloc
DBGANY: 2 TEXTNOD = declare
DBGANY: 3 TEXTNOD = devtype varchar2(255);
DBGANY: 4 TEXTNOD = chid varchar2(255);
DBGANY: 5 TEXTNOD = debug number := null;
DBGANY: 6 TEXTNOD = options number := null;
DBGANY: 7 TEXTNOD = node varchar2(255);
DBGANY: 8 TEXTNOD = maxsize number;
DBGANY: 9 TEXTNOD = kbytes number := null;
DBGANY: 10 TEXTNOD = parallel binary_integer := null;
DBGANY: 11 TEXTNOD = readrate number := null;
DBGANY: 12 TEXTNOD = rate number := null;
DBGANY: 13 TEXTNOD = sendcmd varchar2(256);
DBGANY: 14 TEXTNOD = vendor varchar2(256);
DBGANY: 15 TEXTNOD = israc boolean;
DBGANY: 16 TEXTNOD = instname varchar2(17);
DBGANY: 17 TEXTNOD = begin
DBGANY: 18 TEXTNOD =
DBGANY: 19 PRMVAL = chid := 'c0';
DBGANY: 20 TEXTNOD = if debug is not null then
DBGANY: 21 TEXTNOD = krmicd.execSql(
DBGANY: 22 TEXTNOD = 'alter session set events ''immediate trace name krb_trace level '
DBGANY: 23 TEXTNOD = ||debug||'''');
DBGANY: 24 TEXTNOD = end if;
DBGANY: 25 TEXTNOD = if options is not null then
DBGANY: 26 TEXTNOD = krmicd.execSql(
DBGANY: 27 TEXTNOD = 'alter session set events ''immediate trace name krb_options level '
DBGANY: 28 TEXTNOD = ||options||'''');
DBGANY: 29 TEXTNOD = end if;
DBGANY: 30 TEXTNOD = devtype := sys.dbms_backup_restore.deviceAllocate( ident => chid,
DBGANY: 31 TEXTNOD = node => node,
DBGANY: 32 PRMVAL = type=>null,dupcnt=>1
DBGANY: 33 TEXTNOD = );
DBGANY: 34 TEXTNOD = if kbytes is null then
DBGANY: 35 TEXTNOD = maxsize := sys.dbms_backup_restore.deviceQuery
DBGANY: 36 TEXTNOD = (sys.dbms_backup_restore.DEVICEQUERY_MAXSIZE);
DBGANY: 37 TEXTNOD = else
DBGANY: 38 TEXTNOD = maxsize := kbytes;
DBGANY: 39 TEXTNOD = end if;
| DBGANY: | 40 TEXTNOD = if maxsize > 0 then |
| DBGANY: | 41 TEXTNOD = | sys.dbms_backup_restore.setlimit |
| DBGANY: | 42 TEXTNOD = | (sys.dbms_backup_restore.kbytes, maxsize); |
| DBGANY: | 43 TEXTNOD = end if; |
| DBGANY: | 44 TEXTNOD = krmicd.setChannelInfo |
| DBGANY: | 45 TEXTNOD = | (devtype, |
| DBGANY: | 46 TEXTNOD = | node, |
| DBGANY: | 47 TEXTNOD = | maxsize, |
| DBGANY: | 48 TEXTNOD = | sys.dbms_backup_restore.deviceQuery |
| DBGANY: | 49 TEXTNOD = | (sys.dbms_backup_restore.DEVICEQUERY_PROXY), |
| DBGANY: | 50 TEXTNOD = | sys.dbms_backup_restore.deviceQuery |
| DBGANY: | 51 TEXTNOD = | (sys.dbms_backup_restore.DEVICEQUERY_MAXPROXY)); |
| DBGANY: | 52 TEXTNOD = if parallel is not null then |
| DBGANY: | 53 TEXTNOD = | sys.dbms_backup_restore.setlimit |
| DBGANY: | 54 TEXTNOD = | (sys.dbms_backup_restore.parallel, parallel); |
| DBGANY: | 55 TEXTNOD = end if; |
| DBGANY: | 56 TEXTNOD = if readrate is not null then |
| DBGANY: | 57 TEXTNOD = | sys.dbms_backup_restore.setlimit |
| DBGANY: | 58 TEXTNOD = | (sys.dbms_backup_restore.readrate, readrate); |
| DBGANY: | 59 TEXTNOD = end if; |
| DBGANY: | 60 TEXTNOD = if rate is not null then |
| DBGANY: | 61 TEXTNOD = | sys.dbms_backup_restore.setlimit |
| DBGANY: | 62 TEXTNOD = | (sys.dbms_backup_restore.max_read_kbytes, rate); |
| DBGANY: | 63 TEXTNOD = end if; |
| DBGANY: | 64 TEXTNOD = if sendcmd is not null then |
| DBGANY: | 65 TEXTNOD = | sys.dbms_backup_restore.devicecommand(sendcmd, NULL); |
| DBGANY: | 66 TEXTNOD = end if; |
| DBGANY: | 67 TEXTNOD = krmicd.writeMsg(8030, chid); |
| DBGANY: | 68 TEXTNOD = krmicd.getInstance(instname, israc); |
| DBGANY: | 69 TEXTNOD = if (israc) then |
| DBGANY: | 70 TEXTNOD = | krmicd.writeMsg(8605, chid, to_char(krmicd.getSid), |
| DBGANY: | 71 TEXTNOD = | to_char(instname), devtype); |
| DBGANY: | 72 TEXTNOD = else |
| DBGANY: | 73 TEXTNOD = | krmicd.writeMsg(8500, chid, to_char(krmicd.getSid), devtype); |
| DBGANY: | 74 TEXTNOD = end if; |
| DBGANY: | 75 TEXTNOD = vendor := sys.dbms_backup_restore.deviceQuery |
| DBGANY: | 76 TEXTNOD = | (sys.dbms_backup_restore.DEVICEQUERY_VENDOR); |
| DBGANY: | 77 TEXTNOD = if vendor is not null then |
| DBGANY: | 78 TEXTNOD = | krmicd.writemsg(8526, chid, vendor); |
| DBGANY: | 79 TEXTNOD = end if; |
| DBGANY: | 80 TEXTNOD = end; |
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-00601: fatal error in recovery manager
RMAN-03004: fatal error during execution of command
RMAN-00600: internal error, arguments [6000] [] [] [] []
Contents of alert log file when backup started:-
Shutting down instance (immediate)
Stopping background process SMCO
Shutting down instance: further logons disabled
Stopping background process QMNC
Stopping background process MMNL
Stopping background process MMON
License high water mark = 116
Fri Sep 13 21:17:07 2013
Active call for process 1187 user 'oraDEV' program 'oracle@gddevdb1.tireco.com'
SHUTDOWN: waiting for active calls to complete.
Fri Sep 13 21:20:13 2013
Incremental checkpoint up to RBA [0x51.1d8e.0], current log tail at RBA [0x51.1d9a.0]
Fri Sep 13 21:40:15 2013
Incremental checkpoint up to RBA [0x51.1df0.0], current log tail at RBA [0x51.1df0.0]
Fri Sep 13 22:00:17 2013
Incremental checkpoint up to RBA [0x51.1e18.0], current log tail at RBA [0x51.1e2a.0]
Fri Sep 13 22:12:08 2013
Active call for process 1187 user 'oraDEV' program 'oracle@gddevdb1.tireco.com'
SHUTDOWN: Active sessions prevent database close operation
Shutdown did not complete in 60 minutes
Instance shutdown cancelled
alter database mount
ORA-1100 signalled during: alter database mount...
Fri Sep 13 22:20:20 2013
Please let me know how do I troubleshoot the root cause for this error. I checked metalink and found a note which says to "resync catalog" and I did that but it hasn't helped in my case. Please help a quick response will be highly appreciated.
-Thanks