Setup: - Oracle 10.2.0.4 EE on HP-UX 11.23
Two weeks ago we got our DG physical standby setup working, initial tests looked good, I could tail both primary and standby alert logs and see results of log switches, everything looked good for about a week. Then ..
One morning I came in and logs weren't shipping. Checked the standby site and it had lost its mounts to the NAS. So far out setup was more proof of concept and I was getting ready to take a week off. So on the primary I set log arch dest 2 status to DEFER. In my absence our SA fixed the problem with mounting the fs and restarted the standby db (he's not a dba and just did a normal startup). I returned to the office this morning and find dest2 status back to ENABLE. But it appears that archivelogs are not applying.
Prior to the crash, as redo was received at the standy, its alert would show
Wed May 25 18:38:11 2011
Primary database is in MAXIMUM PERFORMANCE mode
RFS[39]: Successfully opened standby log 5: '/oradata/ora_redo/BOSTON/sbyredo05a.rdo'
Wed May 25 18:38:17 2011
Media Recovery Waiting for thread 1 sequence 3123 (in transit)
Wed May 25 18:38:17 2011
Recovery of Online Redo Log: Thread 1 Group 5 Seq 3123 Reading mem 0
Mem# 0: /oradata/ora_redo/BOSTON/sbyredo05a.rdo
Mem# 1: /archive/ora_redo/BOSTON/sbyredo05b.rdo
After the standby was restarted, I don't see the above sequence, but instead get
Fri Jun 3 07:27:09 2011
Primary database is in MAXIMUM PERFORMANCE mode
RFS[28]: Successfully opened standby log 4: '/oradata/ora_redo/BOSTON/sbyredo04a.rdo'
Fri Jun 3 07:28:49 2011
Primary database is in MAXIMUM PERFORMANCE mode
RFS[28]: Successfully opened standby log 5: '/oradata/ora_redo/BOSTON/sbyredo05a.rdo'
Fri Jun 3 07:31:15 2011
Primary database is in MAXIMUM PERFORMANCE mode
RFS[28]: Successfully opened standby log 4: '/oradata/ora_redo/BOSTON/sbyredo04a.rdo'
Fri Jun 3 08:39:47 2011
Primary database is in MAXIMUM PERFORMANCE mode
RFS[28]: Successfully opened standby log 5: '/oradata/ora_redo/BOSTON/sbyredo05a.rdo'
It appears that I have exceeded control_file_record_keep_time. This showed up in the standby alert
Mon Jun 6 08:25:47 2011
FAL[client]: Failed to request gap sequence
GAP - thread 1 sequence 3126-3224
DBID 2542058214 branch 737468006
FAL[client]: All defined FAL servers have been attempted.
-------------------------------------------------------------
Check that the CONTROL_FILE_RECORD_KEEP_TIME initialization
parameter is defined to a value that is sufficiently large
enough to maintain adequate log switch information to resolve
archivelog gaps.
-------------------------------------------------------------
On primary:
SQL> show parameter CONTROL_FILE_RECORD_KEEP_TIME
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
control_file_record_keep_time integer 7
SQL>
I could take the "easy" way out and rebuild the standby from a fresh backup of primary, but before doing that would like to learn what I can from this, as I'm still pretty new to managaing a DB setup.