Both DB are located on individual standalone AIX server and using oracle12c.
Checked with network team, they mention there is no unusual activity inside the network log file.
When check for the archive sequence between primary and standby, the standby db sequence is not sync.
I have attached a portion of the alert log file for both DB, if any guidance on finding the root cause greatly appreciated.
Production DB alert log file.
ORA-16198: LGWR received timedout error from KSR
LGWR: Attempting destination LOG_ARCHIVE_DEST_2 network reconnect (16198)
LGWR: Destination LOG_ARCHIVE_DEST_2 network reconnect abandoned
Wed Jul 26 14:19:56 2017
Errors in file /a/trace/bnm_lgwr_6684888.trc:
ORA-16198: Timeout incurred on internal channel during remote archival
Error 16198 for archive log file 3 to 'bnmdr'
ORA-16198: LGWR received timedout error from KSR
LGWR: Error 16198 disconnecting from destination LOG_ARCHIVE_DEST_2 standby host 'bnmdr'
Wed Jul 26 14:19:56 2017
Destination LOG_ARCHIVE_DEST_2 is UNSYNCHRONIZED
LGWR: Failed to archive log 3 thread 1 sequence 175024 (16198)
Wed Jul 26 14:19:56 2017
Thread 1 advanced to log sequence 175025 (LGWR switch)
Current log# 1 seq# 175025 mem# 0: /b/redo01.log
Standby DB alert log file.
Wed Jul 26 14:26:16 2017
Killing 1 processes (PIDS:44302584) (idle RFS in sequence range) in order to allow the in-transit gap to be queued. Requested by OS process 14418150
Wed Jul 26 14:26:18 2017
RFS[250]: Assigned to RFS process (PID:16384172)
RFS[250]: Opened log for thread 1 sequence 175025 dbid 4265992023 branch 905082010
Wed Jul 26 14:26:19 2017
Primary database is in MAXIMUM AVAILABILITY mode
Changing standby controlfile to RESYNCHRONIZATION level
Standby controlfile consistent with primary
RFS[251]: Assigned to RFS process (PID:36503656)
RFS[251]: Selected log 4 for thread 1 sequence 175027 dbid 4265992023 branch 905082010
Wed Jul 26 14:26:19 2017
RFS[252]: Assigned to RFS process (PID:38404128)
RFS[252]: Selected log 6 for thread 1 sequence 175026 dbid 4265992023 branch 905082010
Wed Jul 26 14:26:20 2017
Archived Log entry 6718 added for thread 1 sequence 175025 rlc 905082010 ID 0x6ffaab dest 2:
RFS[250]: Selected log 5 for thread 1 sequence 175024 dbid 4265992023 branch 905082010
Wed Jul 26 14:26:20 2017
Archived Log entry 6719 added for thread 1 sequence 175026 ID 0x6ffaab dest 1:
Wed Jul 26 14:26:21 2017
Archived Log entry 6720 added for thread 1 sequence 175024 ID 0x6ffaab dest 1:
Wed Jul 26 14:26:21 2017
Media Recovery Log /h03/bnm/standbylog/1_175025_905082010.dbf
Incomplete read from log member '/c/standbylog/1_175025_905082010.dbf'. Trying next member.
Incomplete read from log member '/c/1_175025_905082010.dbf'. Trying next member.
Errors in file /h01/oracle/diag/rdbms/bnmdr/bnm/trace/bnm_pr00_37814282.trc (incident=48029):
ORA-00353: log corruption near block 96 change 5036005979 time 07/26/2017 14:19:56
ORA-00334: archived log: '/c/standbylog/1_175025_905082010.dbf'
Incident details in: /h01/oracle/diag/rdbms/bnmdr/bnm/incident/incdir_48029/bnm_pr00_37814282_i48029.trc
Errors with log /h03/bnm/standbylog/1_175025_905082010.dbf
MRP0: Detected read corruption! Retry recovery once log is re-fetched...
Wed Jul 26 14:26:22 2017
Errors in file /a/trace/bnm_pr00_37814282.trc:
ORA-00368: checksum error in redo log block
ORA-00353: log corruption near block 96 change 5036005979 time 07/26/2017 14:19:56
ORA-00334: archived log: '/h03/bnm/standbylog/1_175025_905082010.dbf'
Managed Standby Recovery not using Real Time Apply
Recovery interrupted!
Wed Jul 26 14:26:22 2017
Errors in file /a/trace/bnm_m001_12845242.trc:
ORA-48132: requested file lock is busy, [stg2_48029_inc.swp] [/a/lck/SW_48029_1.lck]
ORA-48170: unable to lock file - already in use
IBM AIX RISC System/6000 Error: 13: Permission denied
Additional information: 8
Additional information: 42860738
Recovered data files to a consistent state at change 5036005936
Wed Jul 26 14:26:22 2017
Errors in file /a/trace/bnm_pr00_37814282.trc:
ORA-00368: checksum error in redo log block
ORA-00353: log corruption near block 96 change 5036005979 time 07/26/2017 14:19:56
ORA-00334: archived log: '/c/standbylog/1_175025_905082010.dbf'
Wed Jul 26 14:26:22 2017
Sweep [inc][48029]: completed
Wed Jul 26 14:26:23 2017
Incomplete read from log member '/c/standbylog/1_175025_905082010.dbf'. Trying next member.
Incomplete read from log member '/c/standbylog/1_175025_905082010.dbf'. Trying next member.
Errors in file /a/trace/bnm_m000_42860738.trc (incident=32355):
ORA-00353: log corruption near block 96 change 5036005979 time 07/26/2017 14:19:56
ORA-00334: archived log: '/c/standbylog/1_175025_905082010.dbf'
Incident details in: /h01/oracle/diag/rdbms/bnmdr/bnm/incident/incdir_32355/bnm_m000_42860738_i32355.trc
Wed Jul 26 14:26:23 2017
Dumping diagnostic data in directory=[cdmp_20170726142623], requested by (instance=1, osid=122748929 (PR00)), summary=[incident=48029].
Checker run found 1 new persistent data failures
Wed Jul 26 14:26:24 2017
Dumping diagnostic data in directory=[cdmp_20170726142624], requested by (instance=1, osid=121503967 (M000)), summary=[incident=32355].
Wed Jul 26 14:26:43 2017
Started logmerger process