Hello All,
I just setup a physical data guard yesterday (have not configured broker yet), log shipping is fine, log applying only work right after enabled log apply, after that it stopped to apply any new logs that are shipped over from primary. If I stop/start the apply process again, it'll pick up all the ones that were shipped over, then stop again. Anyone run into this wired scenario before?
=========================================================================
Related init parameters on primary side:
PARAMETER_NAME VALUE
------------------------------ -----------------------------------------------------------------------------------------------
db_file_name_convert EL9PRDCN, EL9PRD
db_flashback_retention_target 4320
db_name EL9PRD
db_unique_name EL9PRD
fal_client
fal_server EL9PRDCIN1
log_archive_config DG_CONFIG=(EL9PRD,EL9PRDCN)
log_archive_dest_1 LOCATION=USE_DB_RECOVERY_FILE_DEST VALID_FOR=(ALL_LOGFILES,ALL_ROLES) DB_UNIQUE_NAME=EL9PRD
log_archive_dest_2 SERVICE=EL9PRDCIN1 ASYNC VALID_FOR=(ONLINE_LOGFILES,PRIMARY_ROLE) DB_UNIQUE_NAME=EL9PRDCN
log_archive_dest_state_1 enable
log_archive_dest_state_2 ENABLE
log_archive_format %t_%s_%r.arc
log_archive_max_processes 8
log_file_name_convert EL9PRDCN, EL9PRD
remote_login_passwordfile EXCLUSIVE
standby_file_management AUTO
=================================================================================
Related init parameters on DR side:
PARAMETER_NAME VALUE
------------------------------ ----------------------------------------------------------------------------------------------------
db_file_name_convert EL9PRD, EL9PRDDR
db_flashback_retention_target 4320
db_name EL9PRD
db_unique_name EL9PRDDR
fal_client EL9PRDDR_DG
fal_server EL9PRD_DG
log_archive_config DG_CONFIG=(EL9PRDDR,EL9PRD)
log_archive_dest_1 LOCATION=USE_DB_RECOVERY_FILE_DEST VALID_FOR=(ALL_LOGFILES,ALL_ROLES) DB_UNIQUE_NAME=EL9PRDDR
log_archive_dest_2 service=EL9PRD_DG async valid_for=(online_logfiles, primary_role) db_unique_name=EL9PRD
log_archive_dest_state_1 enable
log_archive_dest_state_2 ENABLE
log_archive_format %t_%s_%r.arc
log_archive_max_processes 8
log_file_name_convert EL9PRD, EL9PRDDR
remote_login_passwordfile EXCLUSIVE
standby_file_management AUTO
=========================================================================
Alert log segments that show the behavior I described:
ALTER DATABASE RECOVER MANAGED STANDBY DATABASE CANCEL
2021-06-10T14:41:30.680613-04:00
PR00 (PID:103615): MRP0: Background Media Recovery cancelled with status 16037
2021-06-10T14:41:30.681147-04:00
Errors in file /u01/app/oracle/diag/rdbms/el9prddr/EL9PRDDR/trace/EL9PRDDR_pr00_103615.trc:
ORA-16037: user requested cancel of managed recovery operation
PR00 (PID:103615): Managed Standby Recovery not using Real Time Apply
Recovery interrupted!
stopping change tracking
2021-06-10T14:41:30.927907-04:00
Errors in file /u01/app/oracle/diag/rdbms/el9prddr/EL9PRDDR/trace/EL9PRDDR_pr00_103615.trc:
ORA-16037: user requested cancel of managed recovery operation
2021-06-10T14:41:31.049238-04:00
Background Media Recovery process shutdown (EL9PRDDR)
2021-06-10T14:41:31.671474-04:00
Managed Standby Recovery Canceled (EL9PRDDR)
Completed: ALTER DATABASE RECOVER MANAGED STANDBY DATABASE CANCEL
2021-06-10T14:41:43.549673-04:00
ALTER DATABASE RECOVER MANAGED STANDBY DATABASE DISCONNECT
2021-06-10T14:41:43.559560-04:00
Attempt to start background Managed Standby Recovery process (EL9PRDDR)
Starting background process MRP0
2021-06-10T14:41:43.601127-04:00
MRP0 started with pid=51, OS id=226862
2021-06-10T14:41:43.602054-04:00
Background Managed Standby Recovery process started (EL9PRDDR)
2021-06-10T14:41:48.650576-04:00
Started logmerger process
2021-06-10T14:41:48.730852-04:00
PR00 (PID:227108): Managed Standby Recovery starting Real Time Apply
2021-06-10T14:41:52.442976-04:00
Parallel Media Recovery started with 96 slaves
2021-06-10T14:41:52.571139-04:00
stopping change tracking
PR00 (PID:227108): Media Recovery Waiting for T-2.S-777
2021-06-10T14:41:53.171909-04:00
rfs (PID:104682): Opened log for T-2.S-777 dbid 3995036533 branch 1066899250
2021-06-10T14:41:53.222371-04:00
rfs (PID:104682): Archived Log entry 115 added for B-1066899250.T-2.S-777 ID 0xf9332923 LAD:2
2021-06-10T14:41:53.610903-04:00
Completed: ALTER DATABASE RECOVER MANAGED STANDBY DATABASE DISCONNECT
2021-06-10T14:41:54.039431-04:00
PR00 (PID:227108): Media Recovery Log +RECOC1/EL9PRDDR/ARCHIVELOG/2021_06_10/thread_2_seq_777.709.1074868913
2021-06-10T14:41:54.157571-04:00
PR00 (PID:227108): Media Recovery Log +RECOC1/EL9PRDDR/ARCHIVELOG/2021_06_10/thread_1_seq_4402.884.1074857301
2021-06-10T14:41:54.805099-04:00
PR00 (PID:227108): Media Recovery Log +RECOC1/EL9PRDDR/ARCHIVELOG/2021_06_10/thread_1_seq_4403.889.1074859099
PR00 (PID:227108): Media Recovery Waiting for T-2.S-778
PR00 (PID:227108): Fetching gap from T-2.S-778 to T-2.S-778
2021-06-10T14:42:54.841490-04:00
rfs (PID:104829): Opened log for T-2.S-778 dbid 3995036533 branch 1066899250
2021-06-10T14:42:55.591211-04:00
rfs (PID:104829): Archived Log entry 116 added for B-1066899250.T-2.S-778 ID 0xf9332923 LAD:2
2021-06-10T14:42:56.394153-04:00
PR00 (PID:227108): Media Recovery Log +RECOC1/EL9PRDDR/ARCHIVELOG/2021_06_10/thread_2_seq_778.914.1074868975
2021-06-10T14:42:56.599778-04:00
PR00 (PID:227108): Media Recovery Log +RECOC1/EL9PRDDR/ARCHIVELOG/2021_06_10/thread_1_seq_4404.894.1074860897
PR00 (PID:227108): Media Recovery Waiting for T-2.S-779
PR00 (PID:227108): Fetching gap from T-2.S-779 to T-2.S-779
2021-06-10T14:43:57.116824-04:00
rfs (PID:105147): Opened log for T-2.S-779 dbid 3995036533 branch 1066899250
2021-06-10T14:43:57.174999-04:00
rfs (PID:105147): Archived Log entry 117 added for B-1066899250.T-2.S-779 ID 0xf9332923 LAD:2
2021-06-10T14:43:57.550602-04:00
PR00 (PID:227108): Media Recovery Log +RECOC1/EL9PRDDR/ARCHIVELOG/2021_06_10/thread_2_seq_779.915.1074869037
2021-06-10T14:43:58.133608-04:00
PR00 (PID:227108): Media Recovery Log +RECOC1/EL9PRDDR/ARCHIVELOG/2021_06_10/thread_1_seq_4405.468.1074862697
2021-06-10T14:43:58.742143-04:00
PR00 (PID:227108): Media Recovery Waiting for T-2.S-780
PR00 (PID:227108): Fetching gap from T-2.S-780 to T-2.S-780
2021-06-10T14:44:58.689420-04:00
rfs (PID:105320): Opened log for T-2.S-780 dbid 3995036533 branch 1066899250
2021-06-10T14:44:58.753112-04:00
rfs (PID:105320): Archived Log entry 118 added for B-1066899250.T-2.S-780 ID 0xf9332923 LAD:2
2021-06-10T14:44:59.100155-04:00
PR00 (PID:227108): Media Recovery Log +RECOC1/EL9PRDDR/ARCHIVELOG/2021_06_10/thread_2_seq_780.916.1074869099
2021-06-10T14:44:59.226748-04:00
PR00 (PID:227108): Media Recovery Log +RECOC1/EL9PRDDR/ARCHIVELOG/2021_06_10/thread_1_seq_4406.463.1074864497
PR00 (PID:227108): Media Recovery Waiting for T-2.S-781
PR00 (PID:227108): Fetching gap from T-2.S-781 to T-2.S-781
2021-06-10T14:45:52.047691-04:00
rfs (PID:105342): Opened log for T-2.S-781 dbid 3995036533 branch 1066899250
2021-06-10T14:45:52.098831-04:00
rfs (PID:105342): Archived Log entry 119 added for B-1066899250.T-2.S-781 ID 0xf9332923 LAD:2
2021-06-10T14:45:52.982210-04:00
PR00 (PID:227108): Media Recovery Log +RECOC1/EL9PRDDR/ARCHIVELOG/2021_06_10/thread_2_seq_781.918.1074869153
2021-06-10T14:45:53.100168-04:00
PR00 (PID:227108): Media Recovery Log +RECOC1/EL9PRDDR/ARCHIVELOG/2021_06_10/thread_1_seq_4407.908.1074866299
2021-06-10T14:45:54.083613-04:00
PR00 (PID:227108): Media Recovery Waiting for T-2.S-782
PR00 (PID:227108): Fetching gap from T-2.S-782 to T-2.S-782
2021-06-10T14:46:04.909472-04:00
rfs (PID:104110): Opened log for T-2.S-782 dbid 3995036533 branch 1066899250
2021-06-10T14:46:04.983307-04:00
rfs (PID:104110): Archived Log entry 120 added for B-1066899250.T-2.S-782 ID 0xf9332923 LAD:2
2021-06-10T14:46:05.853708-04:00
PR00 (PID:227108): Media Recovery Log +RECOC1/EL9PRDDR/ARCHIVELOG/2021_06_10/thread_2_seq_782.919.1074869165
2021-06-10T14:46:06.029823-04:00
PR00 (PID:227108): Media Recovery Log +RECOC1/EL9PRDDR/ARCHIVELOG/2021_06_10/thread_1_seq_4408.909.1074868099
PR00 (PID:227108): Media Recovery Waiting for T-2.S-783
2021-06-10T14:46:18.250796-04:00
rfs (PID:104642): Opened log for T-2.S-783 dbid 3995036533 branch 1066899250
2021-06-10T14:46:18.309543-04:00
rfs (PID:104642): Archived Log entry 121 added for B-1066899250.T-2.S-783 ID 0xf9332923 LAD:2
2021-06-10T14:46:19.175252-04:00
PR00 (PID:227108): Media Recovery Log +RECOC1/EL9PRDDR/ARCHIVELOG/2021_06_10/thread_2_seq_783.920.1074869179
PR00 (PID:227108): Media Recovery Waiting for T-1.S-4409 (in transit)
2021-06-10T14:46:19.320204-04:00
Recovery of Online Redo Log: Thread 1 Group 9 Seq 4409 Reading mem 0
Mem# 0: +RECOC1/EL9PRDDR/standbylog/standby_group_9.log
PR00 (PID:227108): Media Recovery Waiting for T-2.S-784
2021-06-10T14:58:18.136284-04:00
rfs (PID:159573): Selected LNO:10 for T-1.S-4410 dbid 3995036533 branch 1066899250
2021-06-10T14:58:18.300482-04:00
ARC0 (PID:86425): Archived Log entry 122 added for T-1.S-4409 ID 0xf9332923 LAD:1
2021-06-10T15:28:17.743941-04:00
rfs (PID:159573): Selected LNO:9 for T-1.S-4411 dbid 3995036533 branch 1066899250
2021-06-10T15:28:17.933792-04:00
ARC1 (PID:86431): Archived Log entry 123 added for T-1.S-4410 ID 0xf9332923 LAD:1
2021-06-10T15:58:18.456869-04:00
rfs (PID:159573): Selected LNO:10 for T-1.S-4412 dbid 3995036533 branch 1066899250
2021-06-10T15:58:18.620391-04:00
ARC2 (PID:86433): Archived Log entry 124 added for T-1.S-4411 ID 0xf9332923 LAD:1
2021-06-10T16:28:16.480240-04:00
rfs (PID:159573): Selected LNO:9 for T-1.S-4413 dbid 3995036533 branch 1066899250
2021-06-10T16:28:16.662667-04:00
ARC3 (PID:86436): Archived Log entry 125 added for T-1.S-4412 ID 0xf9332923 LAD:1