I had a very odd issue over the weekend that i have resolved, but I am still unsure what happened.
Long Story Short: Over the weekend we did a datacenter failover. We had to shutdown 3 of our production dataguard standby databases (not primaries, just standbys) for a few hours. Dataguard was running perfectly and everything was up to snuff before we shutdown the standbys.
After the failover testing was completed, we brought the 3 standbys back online and oddly enough 2 out of the 3 reported an unresolveable archive log gap. The log that both were looking for was one from 4 weeks ago (when we last switched over to these instances while applying security patches). Looking at v$archive_gap it showed no rows and everything i found showed the last archive to be applied was the one that was active when we shutdown the database. I could find no reason why it was seeking this archive log from 4 weeks ago.
I attempted to manually resolve the gap, but that was unsuccessful as well. My only recourse was to drop the standby databases and recreate them from scratch. Once i did that everything is good again.
What I am not sure of is why these databases were looking for an archive that had long since been applied and why these 2 databases were affected and not the 3rd one.
These are all running 18.7 on Solaris. I have done a lot of poking around on support, but i cannot find any answer. I have worked around the issue by rebuilding my standbys, but I am concerned that just restarting my standby instances caused such an issue. This kind of thing has never happened before. Any ideas? I will include some of my alert log from one of the restarts after the outage..