Hi All,
I have created standby database  on 2 node rac  and it was working fine for couple of months. Suddently it  stopped applying the logs . The archives are getting shipped but not applied on standby database. I don't see any error  in the alert log in both standby database/primary.
I have tried  no of times cancelling and applyin the logs.  I have even rollfoward the standby database . But still the  archives are not getting applied.
Primary
select thread#, max(sequence#) "Last Standby Seq Received"
from v$archived_log val, v$database vdb
where val.resetlogs_change# = vdb.resetlogs_change#
group by thread# order by 1;
  2    3    4
   THREAD# Last Standby Seq Received
---------- -------------------------
         1                      9110
         2                      7407
when queried standby 
select thread#, max(sequence#) "Last Standby Seq Received"
from v$archived_log val, v$database vdb
where val.resetlogs_change# = vdb.resetlogs_change#
group by thread# order by 1;
  2    3    4
   THREAD# Last Standby Seq Received
---------- -------------------------
         1                      9110
         2                      7407
select thread#, max(sequence#) "Last Standby Seq Applied"
from v$archived_log val, v$database vdb
where val.resetlogs_change# = vdb.resetlogs_change#
and val.applied in ('YES','IN-MEMORY')
group by thread# order by 1;
  2    3    4    5
   THREAD# Last Standby Seq Applied
---------- ------------------------
         1                     5761
         2                     6273
 
alter database recover managed standby database disconnect from session;
Database altered.
select PROCESS,STATUS,THREAD#,SEQUENCE#,BLOCK#,BLOCKS
from v$managed_standby
where process like '%MRP%';
  2    3
PROCESS   STATUS          THREAD#  SEQUENCE#     BLOCK#     BLOCKS
--------- ------------ ---------- ---------- ---------- ----------
MRP0      WAIT_FOR_GAP          2       6274          0          0
When I started the MRP service , I see the following mesg in standby alert log ( These are older archives )..  As mentioned I have already done the  incremental backup and restored on standby  . But didn't create the standby control file once again . Is this step necessary . Please advise.
All non-current ORLs have been archived.
Completed: alter database recover managed standby database disconnect from session
Mon Dec 12 10:07:51 2016
Media Recovery Waiting for thread 2 sequence 6274
Fetching gap sequence in thread 2, gap sequence 6274-6373
Mon Dec 12 10:09:44 2016
FAL[client]: Failed to request gap sequence
 GAP - thread 2 sequence 6274-6373
 DBID 3025880229 branch 923070693
FAL[client]: All defined FAL servers have been attempted.
------------------------------------------------------------
Check that the CONTROL_FILE_RECORD_KEEP_TIME initialization
parameter is defined to a value that's sufficiently large
enough to maintain adequate log switch information to resolve
archivelog gaps.
Thanks
Thanks in advance
Sri