DATABASE HANG AT ALTER DATABASE DISMOUNT
Dear One and All,
I am using following in 2 NODE RAC environment:
OS = HP-UX
Database = Oracle 9i
Instance1 = mosa1
Instance2 = mosa2
node1 = hp1
node2 = hp2
After giving below command:
$srvctl stop database -d dbname
Database hangs at ALTER DATABASE DISMOUNT in alert log file.
Diagnosis:
Following are the contents in alert MOSA1 (INSTANCE 1 ) log file:
==========================================
Shutting down instance: further logons disabled
Mon Jun 6 14:18:45 2011
Shutting down instance (immediate)
License high water mark = 495
Mon Jun 6 14:23:47 2011
Active call for process 17568 user 'oracle' program 'oracle@hp_data1 (TNS V1-V3)'
Active call for process 10779 user 'oracle' program 'oracle@hp_data1 (TNS V1-V3)'
Active call for process 28287 user 'oracle' program 'oracle@hp_data1 (TNS V1-V3)'
Active call for process 9456 user 'oracle' program 'oracle@hp_data1 (TNS V1-V3)'
Active call for process 25267 user 'oracle' program 'oracle@hp_data1 (TNS V1-V3)'
Active call for process 7438 user 'oracle' program 'oracle@hp_data1 (TNS V1-V3)'
Active call for process 16812 user 'oracle' program 'oracle@hp_data1 (TNS V1-V3)'
Active call for process 4101 user 'oracle' program 'oracle@hp_data1 (TNS V1-V3)'
SHUTDOWN: waiting for active calls to complete.
Mon Jun 6 14:36:45 2011
IPC Send timeout detected. Sender ospid 21603
Mon Jun 6 14:36:53 2011
Errors in file /u01/app/oracle/admin/mosa/bdump/mosa1_lck0_21603.trc:
ORA-27506: IPC error connecting to a port
Mon Jun 6 14:37:28 2011
Errors in file /u01/app/oracle/admin/mosa/bdump/mosa1_lck0_21603.trc:
ORA-27506: IPC error connecting to a port
Mon Jun 6 14:38:08 2011
Errors in file /u01/app/oracle/admin/mosa/bdump/mosa1_lck0_21603.trc:
ORA-27506: IPC error connecting to a port
Mon Jun 6 14:39:51 2011
ALTER DATABASE CLOSE NORMAL
Mon Jun 6 14:39:53 2011
SMON: disabling tx recovery
SMON: disabling cache recovery
Mon Jun 6 14:39:53 2011
Shutting down archive processes
Archiving is disabled
Mon Jun 6 14:39:53 2011
ARCH shutting down
Mon Jun 6 14:39:53 2011
ARCH shutting down
Mon Jun 6 14:39:53 2011
ARC0: Archival stopped
Mon Jun 6 14:39:53 2011
ARC1: Archival stopped
Mon Jun 6 14:39:53 2011
Thread 1 closed at log sequence 5075
Successful close of redo thread 1
Mon Jun 6 14:39:55 2011
Completed: ALTER DATABASE CLOSE NORMAL
Mon Jun 6 14:39:55 2011
ALTER DATABASE DISMOUNT
Following are the contents in Errors in file /u01/app/oracle/admin/mosa/bdump/mosa1_lck0_21603.trc:
================================================================
/u01/app/oracle/admin/mosa/bdump/mosa1_lck0_21603.trc
Oracle9i Enterprise Edition Release 9.2.0.8.0 - 64bit Production
With the Partitioning, Real Application Clusters, OLAP and Oracle Data Mining options
JServer Release 9.2.0.8.0 - Production
ORACLE_HOME = /u01/app/oracle/product/920
System name: HP-UX
Node name: hp_data1
Release: B.11.11
Version: U
Machine: 9000/800
Instance name: mosa1
Redo thread mounted by this instance: 1
Oracle process number: 17
Unix process pid: 21603, image: oracle@hp_data1 (LCK0)
* last lines in this trace file are *
KSTDUMP: End of in-memory trace dump
*** 2011-06-06 14:36:51.242
ORA-27506: IPC error connecting to a port
*** 2011-06-06 14:37:28.882
ORA-27506: IPC error connecting to a port
*** 2011-06-06 14:38:08.936
ORA-27506: IPC error connecting to a port
Followed following steps to troubleshoot the issue:
=================================
1. Locate any client connections to the database using ps, and grep for any processes belonging to this .
Example: ps -ef | grep V733
2. Look for processes that include a 'Local=No' designation.
Example: osupport 6235 1 0 Nov 24 0:01 oracleV733 (LOCAL=NO)
3. Kill the Unix process(es) with the 'Local=No' designation.
Example: Kill -9 6235
Killed above processess but database still hangs at ALTER DATABASE DISMOUNT.
Following are the contents in alert MOSA2 (INSTANCE 2 ) log file:
==========================================
Mon Jun 6 14:08:16 2011
Active call for process 20896 user 'oracle' program 'oracle@hp_data2 (TNS V1-V3)'
SHUTDOWN: waiting for active calls to complete.
Mon Jun 6 14:17:35 2011
ALTER DATABASE CLOSE NORMAL
Mon Jun 6 14:17:36 2011
SMON: disabling tx recovery
SMON: disabling cache recovery
Mon Jun 6 14:17:36 2011
Shutting down archive processes
Archiving is disabled
Mon Jun 6 14:17:36 2011
ARCH shutting down
Mon Jun 6 14:17:36 2011
ARCH shutting down
Mon Jun 6 14:17:36 2011
ARC0: Archival stopped
Mon Jun 6 14:17:36 2011
ARC1: Archival stopped
Mon Jun 6 14:17:36 2011
Thread 2 closed at log sequence 5353
Successful close of redo thread 2
Mon Jun 6 14:17:38 2011
Completed: ALTER DATABASE CLOSE NORMAL
Mon Jun 6 14:17:38 2011
ALTER DATABASE DISMOUNT
Mon Jun 6 14:21:16 2011
IPC Send timeout detected. Receiver ospid 5676
Mon Jun 6 14:21:16 2011
Errors in file /u01/app/oracle/admin/mosa/bdump/mosa2_lck0_5676.trc:
Kindly help me to resolve this issue on PRODUCTIOn server.
Waiting for the response.
Thank you,