Hi,
Primary = ODA Box
DR = OCI DB System(hybrid data guard solution)
DR syncing has suddenly stopped without making any changes. Please refer to the below alert log errors primary and DR.
Primary
----------
2024-04-09T11:20:57.427408+05:00
TMON (PID:27755): Process (PID:59718) hung on an I/O to LAD:3 after 247 seconds with threshold of 240 at [krsb.c:4435]
TMON (PID:27755): WARN: Terminating process hung on an operation (PID:59718)
2024-04-09T11:21:01.662557+05:00
TMON (PID:27755): Killing 1 processes (PIDS:59718) (Process by index) in order to remove hung processes. Requested by OS process 27755
2024-04-09T11:21:01.662758+05:00
Process termination requested for pid 59718 [source = rdbms], [info = 2] [request issued by pid: 27755, uid: 102]
2024-04-09T11:21:03.682827+05:00
TMON (PID:27755): Error 16198 due to hung I/O operation to LOG_ARCHIVE_DEST_3
2024-04-09T11:21:57.331351+05:00
TMON (PID:27755): Process (PID:60792) hung on an I/O to LAD:3 after 241 seconds with threshold of 240 at [krsb.c:4435]
TMON (PID:27755): WARN: Terminating process hung on an operation (PID:60792)
2024-04-09T11:21:59.442408+05:00
TMON (PID:27755): Process (PID:60802) hung on an I/O to LAD:3 after 243 seconds with threshold of 240 at [krsb.c:4435]
TMON (PID:27755): WARN: Terminating process hung on an operation (PID:60802)
2024-04-09T11:22:01.552504+05:00
TMON (PID:27755): Process (PID:60812) hung on an I/O to LAD:3 after 245 seconds with threshold of 240 at [krsb.c:4435]
TMON (PID:27755): WARN: Terminating process hung on an operation (PID:60812)
2024-04-09T11:22:03.716449+05:00
TMON (PID:27755): Process (PID:60822) hung on an I/O to LAD:2 after 246 seconds with threshold of 240 at [krsb.c:4435]
TMON (PID:27755): WARN: Terminating process hung on an operation (PID:60822)
2024-04-09T11:22:07.934546+05:00
TMON (PID:27755): Killing 1 processes (PIDS:60792) (Process by index) in order to remove hung processes. Requested by OS process 27755
2024-04-09T11:22:07.934714+05:00
Process termination requested for pid 60792 [source = rdbms], [info = 2] [request issued by pid: 27755, uid: 102]
2024-04-09T11:22:12.028454+05:00
TMON (PID:27755): Killing 1 processes (PIDS:60802) (Process by index) in order to remove hung processes. Requested by OS process 27755
2024-04-09T11:22:12.028624+05:00
Process termination requested for pid 60802 [source = rdbms], [info = 2] [request issued by pid: 27755, uid: 102]
2024-04-09T11:22:16.107027+05:00
TMON (PID:27755): Killing 1 processes (PIDS:60812) (Process by index) in order to remove hung processes. Requested by OS process 27755
2024-04-09T11:22:16.107259+05:00
Process termination requested for pid 60812 [source = rdbms], [info = 2] [request issued by pid: 27755, uid: 102]
2024-04-09T11:22:20.214022+05:00
TMON (PID:27755): Killing 1 processes (PIDS:60822) (Process by index) in order to remove hung processes. Requested by OS process 27755
2024-04-09T11:22:20.214348+05:00
Process termination requested for pid 60822 [source = rdbms], [info = 2] [request issued by pid: 27755, uid: 102]
2024-04-09T11:22:22.216095+05:00
TMON (PID:27755): Error 16198 due to hung I/O operation to LOG_ARCHIVE_DEST_2
TMON (PID:27755): Detected ARCH process failure
TMON (PID:27755): Detected ARCH process failure
TMON (PID:27755): Detected ARCH process failure
TMON (PID:27755): STARTING ARCH PROCESSES
Starting background process ARC1
2024-04-09T11:22:22.250712+05:00
ARC1 started with pid=827, OS id=76545
Starting background process ARC2
2024-04-09T11:22:22.284652+05:00
ARC2 started with pid=938, OS id=76555
Starting background process ARC3
2024-04-09T11:22:22.318336+05:00
ARC3 started with pid=952, OS id=76565
TMON (PID:27755): ARC1: Archival started
TMON (PID:27755): ARC2: Archival started
2024-04-09T11:22:22.318585+05:00
ARC1 (PID:76545): FAIL: Reclaiming FAL entry from dead process (PID:60812)
2024-04-09T11:22:22.321176+05:00
ARC2 (PID:76555): FAIL: Reclaiming FAL entry from dead process (PID:60792)
2024-04-09T11:22:22.328622+05:00
TMON (PID:27755): ARC3: Archival started
TMON (PID:27755): STARTING ARCH PROCESSES COMPLETE
2024-04-09T11:22:22.328779+05:00
ARC3 (PID:76565): FAIL: Reclaiming FAL entry from dead process (PID:60802)
2024-04-09T11:22:22.330646+05:00
ARC1 (PID:76545): FAL archive failed with error 16198 (see trace for details)
2024-04-09T11:22:22.343575+05:00
ARC2 (PID:76555): FAL archive failed with error 16198 (see trace for details)
2024-04-09T11:22:22.351849+05:00
ARC3 (PID:76565): FAL archive failed with error 16198 (see trace for details)
2024-04-09T11:22:22.357227+05:00
ARC1 (PID:76545): FAL archive failed, archiver continuing
2024-04-09T11:22:22.357271+05:00
ORACLE Instance dbscdb, archival error, archiver continuing
2024-04-09T11:22:22.357319+05:00
ORA-16055: FAL request rejected
2024-04-09T11:22:22.357396+05:00
Errors in file /u01/app/oracle/diag/rdbms/dbscdbpr/dbscdb/trace/dbscdb_arc1_76545.trc:
ORA-16055: FAL request rejected
2024-04-09T11:22:22.374491+05:00
ARC2 (PID:76555): FAL archive failed, archiver continuing
2024-04-09T11:22:22.374554+05:00
ORACLE Instance dbscdb, archival error, archiver continuing
2024-04-09T11:22:22.374619+05:00
ORA-16055: FAL request rejected
2024-04-09T11:22:22.374723+05:00
DR
-----
2024-04-09T06:06:53.745420+00:00
rfs (PID:57341): krsr_rfs_atc: Identified database type as 'PHYSICAL STANDBY': Client is ASYNC (PID:41796)
rfs (PID:57341): Primary database is in MAXIMUM PERFORMANCE mode
rfs (PID:57341): No SRLs created for T-1
2024-04-09T06:06:53.892119+00:00
rfs (PID:57341): Opened log for T-1.S-15656 dbid 2288436639 branch 1142511394
2024-04-09T06:08:02.906231+00:00
rfs (PID:57332): Killing 1 processes (PIDS:53069) (idle RFS in sequence range) in order to allow the in-transit gap to be queued. Requested by OS process 57332
2024-04-09T06:08:02.906467+00:00
Process termination requested for pid 53069 [source = rdbms], [info = 2] [request issued by pid: 57332, uid: 101]
2024-04-09T06:08:03.923211+00:00
Deleted Oracle managed file +RECO/DBSCDB_DRDB/ARCHIVELOG/2024_04_09/thread_1_seq_15571.1966.1165816863
2024-04-09T06:11:13.558439+00:00
TMON (PID:15500): Process (PID:57341) hung on an I/O after 259 seconds with threshold of 240 at [krsb.c:3471]
TMON (PID:15500): WARN: Terminating process hung on an operation (PID:57341)
2024-04-09T06:11:17.700474+00:00
TMON (PID:15500): Killing 1 processes (PIDS:57341) (Process by index) in order to remove hung processes. Requested by OS process 15500
2024-04-09T06:11:17.700647+00:00
Process termination requested for pid 57341 [source = rdbms], [info = 2] [request issued by pid: 15500, uid: 101]
2024-04-09T06:11:18.720885+00:00
Deleted Oracle managed file +RECO/DBSCDB_DRDB/ARCHIVELOG/2024_04_09/thread_1_seq_15656.1922.1165817213
2024-04-09T06:17:20.369475+00:00
rfs (PID:64445): krsr_rfs_atc: Identified database type as 'PHYSICAL STANDBY': Client is Foreground (PID:28139)
Deleted Oracle managed file +RECO/DBSCDB_DRDB/ARCHIVELOG/2024_04_09/thread_0_seq_0.1922.1165817841
2024-04-09T06:17:22.359053+00:00