Database Software

DataGuard Broker failing every few hours ... have to stop and restart. ORA-16766, ORA-01237. I cannot find any information on the failure apply recovery marker (opscode 17.4).


The thought would be from the errors that the volume is out of space, we are running NETAPP and the volumes do auto expand but when I looked the volume it was only at 44% full at 1600 yesterday when I restarted and today I it is at 28%.

These are BIGFILE tablespace. I have restarted multiple times and keep getting the same errors daily and I just end up restarting and everything works again w/o intervention.

I don't want to daily have to check dataguard. If I don't run dg and just run "recover standby database until cancel auto" ... it just keeps going w/o issue, any ideas on the problem?

Please advise:

OS: Solaris 5.10
RDBMS:Oracle 11.2.0.2 64 bit
OMF
BIGFILE Tablespaces
VLDB - Standby DG Database
Created via RMAN Duplicate Database

Dataguard Alert shows the original startup information, does some health check ... becomes quite for a few hours and then I start seeing the Redo Apply is stopped.
####


2011-09-28 18:45:20.824 00000000 24342 Operation HEALTH_CHECK canceled during phase 1, error = ORA-16766
2011-09-28 18:46:20.929 DMON: HEALTH CHECK ERROR: ORA-16766: Redo Apply is stopped

#####
Alert Log showing 1237 on the extend but volume is not out of space ... bigfile tablespace. I cannot find any information on the failure apply recovery marker (opscode 17.4):
#####

Errors in file /u02/d002/oracle/bdump/diag/rdbms/DBUNIQNAME/DBNAME/trace/DBNAME_pr00_6331.trc:
ORA-01237: cannot extend datafile 670
ORA-01110: data file 670: '/u07/d101/oracle/DBUNIQNAME/datafile/o1_mf_ftk_31_c_784ql3gh_.dbf'
Managed Standby Recovery not using Real Time Apply
Wed Sep 28 18:43:57 2011
Recovery interrupted!
Wed Sep 28 18:44:29 2011
Archived Log entry 2805 added for thread 1 sequence 135638 ID 0x249d4159 dest 1:
Wed Sep 28 18:44:41 2011
Recovery stopped due to failure in applying recovery marker (opcode 17.4).
Datafiles are recovered to a consistent state at change 22941677700 but controlfile could be ahead of datafiles.
Wed Sep 28 18:44:42 2011
MRP0: Background Media Recovery process shutdown (DBNAME)
Wed Sep 28 18:46:04 2011

###########
Trace FIle showing the same problem as the alert ...

########################
Started Parallel Media Recovery
*** 2011-09-28 16:14:52.074 4265 krsh.c
Managed Standby Recovery starting Real Time Apply
....
*** 2011-09-28 18:43:08.759 4265 krsh.c
MRP0: Background Media Recovery terminated with error 1237
ORA-01237: cannot extend datafile 670
ORA-01110: data file 670: '/u07/d101/oracle/DBUNIQUNAME/datafile/o1_mf_ftk_31_c_784ql3gh_.dbf'
*** 2011-09-28 18:43:08.779 4265 krsh.c
Managed Standby Recovery not using Real Time Apply----- Redo read statistics for thread 1 -----
Read rate (ASYNC): 54581570Kb in 8888.95s => 6.00 Mb/sec
Total redo bytes: 54636866Kb Longest record: 81Kb, moves: 49633/139704932 moved: 130Mb (0%)
Longest LWN: 118127Kb, reads: 45176
Last redo scn: 0x0005.576e3884 (22941677700)
Change vector header moves = 18603031/273716057 (6%)
----------------------------------------------

*** 2011-09-28 18:43:08.816
Media Recovery drop redo thread 1

*** 2011-09-28 18:43:56.660
Completed Media Recovery

*** 2011-09-28 18:44:00.869
Checking to start in-flux buffer recovery from SCN 5.1465762230 to SCN (non-inclusive) 5.1466841220
Influx recovery found in-flux buffers

*** 2011-09-28 18:44:00.874
Influx Media Recovery add redo thread 1

*** 2011-09-28 18:44:36.493
Resized overflow buffer to 52335K (for 52335K LWN)

*** 2011-09-28 18:44:38.887
Resized overflow buffer to 63165K (for 63165K LWN)

*** 2011-09-28 18:44:41.686
Managed Recovery: Not Active posted.