I have a production Oracle server SitaA running Oracle 11.2.0.4 on Windows Server 2012 R2 with two databases (DB-A and DB-BP). I also have a secondary Oracle server SiteB also running Oracle 11.2.0.4 on Windows Server 2012 R2 with two databases (DB-C and DB-BS). Databases DB-A and DB-C are standalone and reside only on their respective servers while DB-BP and DB-BS are a primary/standby pair connected via Data Guard.
I needed to install the Oracle 12.2.0.1 binaries on the two servers but only upgrade the DB-BP/DB-BS databases to 12c and leave the other two at 11g for awhile more.
My production application points to DB-BP while I perform RMAN backups from DB-BS. Prior to upgrading DB-BP, I shutdown the DB-BS because I wanted to use the standby database as a fallback in case there was an issue with the primary database upgrade. I successfully upgraded DB-BP from 11g to 12c and it is working fine. Next I copied the PFILE and PWD file for DB-BS from the 11g Home to the 12c Home on SiteB and mounted DB-BS from the 12c Home. I started MRP on DB-BS but no archivelog files got transferred over. The alert file on DB-BP shows the error:
FAL[server]: Version mismatch (FAL client ver: 3, FAL server ver: 100)
So I manually copied and registered the archivelog files generated during the upgrade of DB-BP (100 files) from SiteA to SiteB. Those got applied on DB-BS and eventually all the other files got transferred over and applied via Data Guard. What I did not notice is that the Oracle services running on SiteB for DB-BS were not changed like they were for DB-BP on SiteA. Especially the service OracleServiceDB-BS is still pointing to the 11g Home. Also in a TRM file for DB-BP I see repeated entries like:
*** 2020-05-12T12:01:51.265925-04:00
Software version mismatch: found 2 wanted 300
ORA-16063: destination evicted from collective process
*** 2020-05-12 12:01:51.656591 5634 krsh.c
Error 16063 for archive log file 3 to 'DB-BS'
*** 2020-05-12 12:01:51.656591 8506 krsi.c
krsi_dst_fail_caller: LAD:3 err:16063 force:0 blast:1
krsd_multidest_clear: Clearing Multidest for LAD:3
*** 2020-05-12 12:01:51.656591 1440 krss.c
krss_req_xpt_async_dgrd: Downgraded LAD:3 to single destination
ORA-16063: destination evicted from collective process
ORA-16063: destination evicted from collective process
What does this mean? Was there some other post-upgrade steps that I missed to make sure that DB-BS was truly upgraded?