Switchover failed
655491Jan 16 2009 — edited Sep 21 2010We have a data guard configuration, Oracle 10g R2 RAC (2 nodes), Windows 2003 Server R2, ASM, both primary and standby are using RAC and ASM, physical standby. After we failover from site1 to site2, we rebuilt site1 as physical standby and tried to switchover to site1 to be the primary. It failed about 30 seconds after switchover command. It gave the following message on the apply instance of site 1.
*
Welcome to DGMGRL, type "help" for information.
Connected.
Performing switchover NOW, please wait...
Error:
ORA-03113: end-of-file on communication channel
Unable to switchover, primary database is still "site2"
*
The apply instance on site1 was shutdown, but the another instance on site1 was not. Both instances on primary are still alive.
I checked the alert log on this instance, nothing special. But I checked the alert log on the current primary, and found the following message. Before I did switchover, I use DGMGRL to make sure everything shows SUCCESS. So why I still got this error?
Thank you so much in advance,
--------------------------------------------------------------------------------------------------------------------
Fri Jan 16 08:28:44 2009
ALTER SYSTEM SET log_archive_dest_state_2='ENABLE' SCOPE=BOTH;
Fri Jan 16 08:28:48 2009
******************************************************************
LGWR: Setting 'active' archival for destination LOG_ARCHIVE_DEST_2
******************************************************************
LNSb started with pid=65, OS id=6456
Fri Jan 16 08:28:51 2009
LGWR: Standby redo logfile selected to archive thread 2 sequence 54
LGWR: Standby redo logfile selected for thread 2 sequence 54 for destination LOG_ARCHIVE_DEST_2
Thread 2 advanced to log sequence 54
Current log# 9 seq# 54 mem# 0: +DATA/site2/onlinelog/group_9.270.676144611
Current log# 9 seq# 54 mem# 1: +RECOVERY/site2/onlinelog/group_9.1555.676144611
Fri Jan 16 08:29:00 2009
Destination LOG_ARCHIVE_DEST_2 is SYNCHRONIZED
LGWR: Standby redo logfile selected to archive thread 2 sequence 55
LGWR: Standby redo logfile selected for thread 2 sequence 55 for destination LOG_ARCHIVE_DEST_2
Thread 2 advanced to log sequence 55
Current log# 10 seq# 55 mem# 0: +DATA/site2/onlinelog/group_10.269.676144613
Current log# 10 seq# 55 mem# 1: +RECOVERY/site2/onlinelog/group_10.1560.676144613
Fri Jan 16 08:29:15 2009
Thread 2 cannot allocate new log, sequence 56
Checkpoint not complete
Current log# 10 seq# 55 mem# 0: +DATA/site2/onlinelog/group_10.269.676144613
Current log# 10 seq# 55 mem# 1: +RECOVERY/site2/onlinelog/group_10.1560.676144613
LGWR: Standby redo logfile selected to archive thread 2 sequence 56
LGWR: Standby redo logfile selected for thread 2 sequence 56 for destination LOG_ARCHIVE_DEST_2
Thread 2 advanced to log sequence 56
Current log# 8 seq# 56 mem# 0: +DATA/site2/onlinelog/group_8.271.676144609
Current log# 8 seq# 56 mem# 1: +RECOVERY/site2/onlinelog/group_8.1553.676144611
Fri Jan 16 08:29:26 2009
*********************************************************************
WARNING: Detected remote instance standby database network disconnect
WARNING: Current instance standby database disconnect may be required
*********************************************************************
LGWR: Error 1092 disconnecting from destination LOG_ARCHIVE_DEST_2 standby host '(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=tcp)(HOST=SITE1C)(PORT=1521)))(CONNECT_DATA=(SERVICE_NAME=site1_XPT)(INSTANCE_NAME=site12)(SERVER=dedicated)))'
Fri Jan 16 08:29:33 2009
Destination LOG_ARCHIVE_DEST_2 is UNSYNCHRONIZED
LGWR: Failed to archive log 8 thread 2 sequence 56 (1092)
Fri Jan 16 08:29:33 2009
Errors in file c:\oracle\product\10.2.0\admin\site2\bdump\site22_lgwr_7932.trc:
ORA-16169: LGWR network server invalid parameters
LGWR: Error 16169 closing archivelog file '(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=tcp)(HOST=SITE1C)(PORT=1521)))(CONNECT_DATA=(SERVICE_NAME=site1_XPT)(INSTANCE_NAME=site12)(SERVER=dedicated)))'
Thread 2 advanced to log sequence 57
Current log# 3 seq# 57 mem# 0: +DATA/site2/onlinelog/group_3.276.676144605
Current log# 3 seq# 57 mem# 1: +RECOVERY/site2/onlinelog/group_3.1563.676144605