Standby problem - log numbers out of sequence
610916Dec 19 2008 — edited Dec 19 2008Hi everyone! I am on 10.2.0.3.
My physical standby stopped applying logs because all of a sudden the logs went out of sequence. See errors below - after seq 7975 & 7976 seq# 7974 showed up again:
Recovery of Online Redo Log: Thread 1 Group 200 Seq 7974 Reading mem 0
Mem# 0: /db/data3/FSPRD/stby_redo02.log
Mem# 1: /db/data4/FSPRD/stby_redo02_02.log
Thu Dec 18 16:14:58 2008
RFS[23]: Successfully opened standby log 100: '/db/data1/FSPRD/stby_redo01.log'
Thu Dec 18 16:15:01 2008
RFS[25]: Successfully opened standby log 300: '/db/data5/FSPRD/stby_redo03.log'
Thu Dec 18 16:15:01 2008
Primary database is in MAXIMUM PERFORMANCE mode
RFS[37]: Successfully opened standby log 400: '/db/data7/FSPRD/stby_redo04.log'
Thu Dec 18 16:15:02 2008
RFS[24]: Successfully opened standby log 200: '/db/data3/FSPRD/stby_redo02.log'
Thu Dec 18 16:15:03 2008
Media Recovery Waiting for thread 1 sequence 7975 (in transit)
Thu Dec 18 16:15:03 2008
Recovery of Online Redo Log: Thread 1 Group 100 Seq 7975 Reading mem 0
Mem# 0: /db/data1/FSPRD/stby_redo01.log
Mem# 1: /db/data2/FSPRD/stby_redo01_02.log
Media Recovery Waiting for thread 1 sequence 7976 (in transit)
Thu Dec 18 16:15:07 2008
Recovery of Online Redo Log: Thread 1 Group 200 Seq 7974 Reading mem 0
Mem# 0: /db/data3/FSPRD/stby_redo02.log
Mem# 1: /db/data4/FSPRD/stby_redo02_02.log
MRP0: Background Media Recovery terminated with error 355
Thu Dec 18 16:15:12 2008
Errors in file /db/dbdump/FSPRD/bdump/fsprd_mrp0_11351.trc:
ORA-00355: change numbers out of order
ORA-00353: log corruption near block 2 change 4994299896 time 12/18/2008 16:14:57
ORA-00312: online log 200 thread 1: '/db/data4/FSPRD/stby_redo02_02.log'
ORA-00312: online log 200 thread 1: '/db/data3/FSPRD/stby_redo02.log'
Managed Standby Recovery not using Real Time Apply
Thu Dec 18 16:15:12 2008
Primary database is in MAXIMUM PERFORMANCE mode
RFS[37]: Successfully opened standby log 100: '/db/data1/FSPRD/stby_redo01.log'
Thu Dec 18 16:15:13 2008
Recovery interrupted!
Thu Dec 18 16:15:13 2008
Errors in file /db/dbdump/FSPRD/bdump/fsprd_mrp0_11351.trc:
ORA-00355: change numbers out of order
ORA-00353: log corruption near block 2 change 4994299896 time 12/18/2008 16:14:57
ORA-00312: online log 200 thread 1: '/db/data4/FSPRD/stby_redo02_02.log'
ORA-00312: online log 200 thread 1: '/db/data3/FSPRD/stby_redo02.log'
Thu Dec 18 16:15:13 2008
MRP0: Background Media Recovery process shutdown (FSPRD)
Thu Dec 18 16:15:19 2008
I of course restarted the managed recovery and everything went fine from this point on - but I could not find the root cause of this problem.
Anybody has any ideas - or experienced this before?
TIA.