To test our instance, we killed PMON on the Primary and enabled the Standby to be the Primary. Now the new Primary (after bringing it back up) will not switch over to be the Standby (it's very stubborn that way).
Here's the error messages:
19:09:21 nolog> startup mount
ORACLE instance started.
Total System Global Area 676433920 bytes
Fixed Size 2162760 bytes
Variable Size 239079352 bytes
Database Buffers 427819008 bytes
Redo Buffers 7372800 bytes
Database mounted.
19:09:34 nolog> alter database commit to switchover to standby with session shutdown;
alter database commit to switchover to standby with session shutdown
*
ERROR at line 1:
ORA-01109: database not open
Elapsed: 00:00:00.00
19:09:50 nolog> alter database open;
Database altered.
Elapsed: 00:00:07.61
19:10:05 nolog> alter database commit to switchover to standby with session shutdown;
alter database commit to switchover to standby with session shutdown
*
ERROR at line 1:
ORA-16416: Switchover target is not synchronized with the primary
Elapsed: 00:00:11.09
19:10:19 nolog>
Alert Log shows:
Wed Dec 05 19:09:27 2012
ALTER DATABASE MOUNT
NOTE:Loaded library: System
SUCCESS: diskgroup DG_01 was mounted
SUCCESS: diskgroup FRA_DG_01 was mounted
Setting recovery target incarnation to 2
Successful mount of redo thread 1, with mount id 3967177063
Database mounted in Exclusive Mode
Lost write protection disabled
Completed: ALTER DATABASE MOUNT
Wed Dec 05 19:09:48 2012
Using STANDBY_ARCHIVE_DEST parameter default value as USE_DB_RECOVERY_FILE_DEST
RFS[1]: Assigned to RFS process 15293
RFS[1]: Database mount ID mismatch [0xec75ffab:0xec765167] (3967156139:3967177063)
RFS[1]: Not using real application clusters
Errors in file /data/oracle/app/oracle/diag/rdbms/ecpmtsb/ECPMTSB/trace/ECPMTSB_rfs_15293.trc:
ORA-16009: remote archive log destination must be a STANDBY database
Wed Dec 05 19:09:50 2012
alter database commit to switchover to standby with session shutdown
ALTER DATABASE COMMIT TO SWITCHOVER TO PHYSICAL STANDBY (ECPMTSB)
ORA-1109 signalled during: alter database commit to switchover to standby with session shutdown...
alter database open
Wed Dec 05 19:09:58 2012
LGWR: STARTING ARCH PROCESSES
Wed Dec 05 19:09:58 2012
ARC0 started with pid=24, OS id=15622
Wed Dec 05 19:09:58 2012
ARC1 started with pid=25, OS id=15624
Wed Dec 05 19:09:58 2012
ARC2 started with pid=26, OS id=15626
Wed Dec 05 19:09:58 2012
ARC3 started with pid=27, OS id=15628
Wed Dec 05 19:09:58 2012
ARC4 started with pid=28, OS id=15630
Wed Dec 05 19:09:58 2012
ARC5 started with pid=29, OS id=15632
ARC0: Archival started
Wed Dec 05 19:09:58 2012
ARC6 started with pid=30, OS id=15634
ARC1: Archival started
ARC2: Archival started
ARC3: Archival started
ARC4: Archival started
ARC5: Archival started
ARC6: Archival started
LGWR: STARTING ARCH PROCESSES COMPLETE
Thread 1 opened at log sequence 1339
Current log# 2 seq# 1339 mem# 0: +FRA_DG_01/ecpmtsb/onlinelog/group_2.401.801169497
Successful open of redo thread 1
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Wed Dec 05 19:09:58 2012
SMON: enabling cache recovery
ARC2: STARTING ARCH PROCESSES
ARC0: Becoming the 'no FAL' ARCH
ARC0: Becoming the 'no SRL' ARCH
ARC6: Becoming the heartbeat ARCH
ARC7: Archival started
Wed Dec 05 19:09:58 2012
ARC7 started with pid=31, OS id=15638
ARC2: STARTING ARCH PROCESSES COMPLETE
Successfully onlined Undo Tablespace 2.
Verifying file header compatibility for 11g tablespace encryption..
Verifying 11g file header compatibility for tablespace encryption completed
SMON: enabling tx recovery
Database Characterset is WE8MSWIN1252
Opening with internal Resource Manager plan
Starting background process FBDA
Wed Dec 05 19:09:59 2012
FBDA started with pid=32, OS id=15645
replication_dependency_tracking turned off (no async multimaster replication found)
Starting background process QMNC
Wed Dec 05 19:09:59 2012
QMNC started with pid=33, OS id=15651
Wed Dec 05 19:10:02 2012
db_recovery_file_dest_size of 51200 MB is 1.96% used. This is a
user-specified limit on the amount of space that will be used by this
database for recovery-related files, and does not reflect the amount of
space available in the underlying filesystem or ASM diskgroup.
Wed Dec 05 19:10:05 2012
Completed: alter database open
alter database commit to switchover to standby with session shutdown
ALTER DATABASE COMMIT TO SWITCHOVER TO PHYSICAL STANDBY (ECPMTSB)
Wed Dec 05 19:10:08 2012
Thread 1 advanced to log sequence 1340 (LGWR switch)
Wed Dec 05 19:10:08 2012
Shutting down archive processes
Current log# 1 seq# 1340 mem# 0: +FRA_DG_01/ecpmtsb/onlinelog/group_1.402.801169497
Wed Dec 05 19:10:08 2012
ARCH shutting down
ARC7: Archival stopped
Waiting for all non-current ORLs to be archived...
Waiting for the ORL for thread 1 sequence 1339 to be archived...
Archived Log entry 37 added for thread 1 sequence 1339 ID 0xec73e4a2 dest 1:
Wed Dec 05 19:10:18 2012
ORL for thread 1 sequence 1339 has been archived...
All non-current ORLs have been archived.
Waiting for all FAL entries to be archived...
All FAL entries have been archived.
Waiting for potential switchover target to become synchronized...
No viable Physical Standby switchover targets available
ORA-16416 signalled during: alter database commit to switchover to standby with session shutdown...
The Parameter for DEST :
19:10:19 nolog> show parameter dest
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
audit_file_dest string /data/oracle/app/oracle/admin/
ECPMTSB/adump
background_dump_dest string /data/oracle/app/oracle/diag/r
dbms/ecpmtsb/ECPMTSB/trace
core_dump_dest string /data/oracle/app/oracle/admin/
ECPMTSB/cdump
db_create_file_dest string
db_create_online_log_dest_1 string
db_create_online_log_dest_2 string
db_create_online_log_dest_3 string
db_create_online_log_dest_4 string
db_create_online_log_dest_5 string
db_recovery_file_dest string +FRA_DG_01
db_recovery_file_dest_size big integer 50G
diagnostic_dest string /data/oracle/app/oracle
log_archive_dest string
log_archive_dest_1 string LOCATION=USE_DB_RECOVERY_FILE_
DEST VALID_FOR=(ALL_LOGFILES,A
LL_ROLES) DB_UNIQUE_NAME=ECPMT
SB
log_archive_dest_10 string
log_archive_dest_2 string SERVICE=ECPMT VALID_FOR=(ONLIN
E_LOGFILES,PRIMARY_ROLE) DB_UN
IQUE_NAME=ECPMT
log_archive_dest_3 string LOCATION=+FRA_DG_01/ecpmtsb/st
andbylog VALID_FOR=(STANDBY_L
OGFILES,STANDBY_ROLE) DB_UNIQU
E_NAME=ECPMTSB
log_archive_dest_4 string
log_archive_dest_5 string
log_archive_dest_6 string
log_archive_dest_7 string
log_archive_dest_8 string
log_archive_dest_9 string
log_archive_dest_state_1 string enable
log_archive_dest_state_10 string enable
log_archive_dest_state_2 string DEFER
log_archive_dest_state_3 string ENABLE
log_archive_dest_state_4 string enable
log_archive_dest_state_5 string enable
log_archive_dest_state_6 string enable
log_archive_dest_state_7 string enable
log_archive_dest_state_8 string enable
log_archive_dest_state_9 string enable
log_archive_duplex_dest string
log_archive_min_succeed_dest integer 2
standby_archive_dest string ?/dbs/arch
user_dump_dest string /data/oracle/app/oracle/diag/r
dbms/ecpmtsb/ECPMTSB/trace
Anyone that can help would be much appreciated. Thank you!
Edited by: BostonDBA on Dec 5, 2012 2:14 PM