Rman cause: Shutting down instance: further logons disabled
669965Oct 17 2010 — edited Nov 24 2010Dear all,
Need your advise. PROD database instance hit this error "Shutting down instance: further logons disabled" when perform rman process. PRDO is on HP-UX and cluster as HP Passive-active mode.
$ rman target / catalog rman/xxxx@rcatdb.world
$ Verified alert log: /home/oracle/admin/PROD/bdump/alert_PROD.1016, below are shown:
Sat Oct 16 14:34:47 2010
Undo Segment 14 Onlined
Sat Oct 16 15:26:33 2010
Errors in file /ora/oracle/product/9.2.0/rdbms/log/prod_ora_5063.trc:
ORA-00600: internal error code, arguments: [2103], [1], [0], [1], [900], [], [], []
Sat Oct 16 15:26:34 2010
Errors in file /ora/oracle/product/9.2.0/rdbms/log/prod_ora_5063.trc:
ORA-00600: internal error code, arguments: [2103], [1], [0], [1], [900], [], [], []
Sat Oct 16 15:26:34 2010
Errors in file /ora/oracle/product/9.2.0/rdbms/log/prod_ora_5063.trc: -
ORA-00600: internal error code, arguments: [2103], [1], [0], [1], [900], [], [], []
Sat Oct 16 15:26:50 2010
Shutting down instance: further logons disabled
Sat Oct 16 15:33:46 2010
Errors in file /ora/oracle/product/9.2.0/rdbms/log/prod_ora_11664.trc:
ORA-00600: internal error code, arguments: [2103], [1], [0], [1], [900], [], [], []
Sat Oct 16 15:33:48 2010
Errors in file /ora/oracle/product/9.2.0/rdbms/log/prod_ora_11664.trc: -
ORA-12012: error on auto execute of job 516
ORA-12008: error in materialized view refresh path
ORA-00600: internal error code, arguments: [2103], [1], [0], [1], [900], [], [], []
ORA-06512: at "SYS.DBMS_SNAPSHOT", line 803
ORA-06512: at "SYS.DBMS_SNAPSHOT", line 860
ORA-06512: at "SYS.DBMS_IREFRESH", line 683
ORA-06512: at "SYS.DBMS_REFRESH", line 195
ORA-06512: at line 1
Sat Oct 16 15:42:17 2010
Errors in file /ora/oracle/product/9.2.0/rdbms/log/prod_ora_12354.trc: -
ORA-00600: internal error code, arguments: [2103], [2], [0], [1], [900], [], [], []
Sat Oct 16 15:42:22 2010
Errors in file /ora/oracle/product/9.2.0/rdbms/log/prod_ora_5047.trc: -
ORA-00471: DBWR process terminated with error
Sat Oct 16 15:42:22 2010
PMON: terminating instance due to error 471
Instance terminated by PMON, pid = 5047
Sat Oct 16 15:56:48 2010
Starting ORACLE instance (normal)
From the trc file:
Errors in file /ora/oracle/product/9.2.0/rdbms/log/prod_ora_5063.trc:
*** 2010-10-16 15:26:31.380
*** SESSION ID:(9.1) 2010-10-16 15:26:31.380
TIMEOUT ON CONTROL FILE ENQUEUE
mode=S, type=0, wait=1, eqt=900
===================================================
Errors in file /ora/oracle/product/9.2.0/rdbms/log/prod_ora_11664.trc
*** 2010-10-16 15:33:45.440
*** SESSION ID:(76.2030) 2010-10-16 15:33:45.440
TIMEOUT ON CONTROL FILE ENQUEUE
mode=S, type=0, wait=1, eqt=900
===================================================
Errors in file /ora/oracle/product/9.2.0/rdbms/log/prod_ora_12354.trc
*** 2010-10-16 15:42:16.646
*** SESSION ID:(81.2225) 2010-10-16 15:42:16.646
TIMEOUT ON CONTROL FILE ENQUEUE
mode=S, type=0, wait=1, eqt=900
===================================================
*** 2010-10-16 15:42:22.294
*** SESSION ID:(1.1) 2010-10-16 15:42:22.294
error 471 detected in background process
ORA-00471: DBWR process terminated with error
From first failure log:
---------------------------
From:
Sent: Saturday, October 16, 2010 13:03
To: DIST-IT-DBA
Subject: RMAN LOG: PROD Weekly Full Level 0 backup (CCM: 4091)
Begin RMAN on Sat Oct 16 13:27:05 PDT 2010
Recovery Manager: Release 9.2.0.5.0 - 64bit Production
Copyright (c) 1995, 2002, Oracle Corporation. All rights reserved.
connected to target database: PROD (DBID=4254449308) connected to recovery catalog database
RMAN> DELETE BACKUP;
2> BACKUP INCREMENTAL LEVEL 0 DATABASE FILESPERSET 4;
3> CROSSCHECK BACKUP;
4> EXIT;
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-00601: fatal error in recovery manager
RMAN-03004: fatal error during execution of command
RMAN-10039: error encountered while polling for RPC completion on channel default
RMAN-10006: error running SQL statement: select act from x$ksusex where sid=:1 and serial=:2
RMAN-10002: ORACLE error: ORA-03113: end-of-file on communication channel
RMAN-03002: failure of delete command at 10/16/2010 14:03:02
RMAN-03014: implicit resync of recovery catalog failed
RMAN-03009: failure of full resync command on default channel at 10/16/2010 14:03:02
ORA-03113: end-of-file on communication channel
End RMAN on Sat Oct 16 14:03:02 PDT 2010