Apparently, my physical standby database decided that it didn't want to deal with any new archive logs, back on August 2nd.
Fortunately, I can view my alert logs to see what happened. I've copied the relevant parts below (sorry it's so long!):
Primary (Zeus):
--- snip ---
Thu Aug 2 00:11:51 2007
ARCH: Connecting to console port...
Committing creation of archivelog '/home/oracle/backup/ZEUS/archivelog/2007_08_02/o1_mf_1_32060_3c2pcwcq_.arc'
Created Oracle managed file /home/oracle/backup/ZEUS/archivelog/2007_08_02/o1_mf_1_32060_3c2pcwcq_.arc
Invoking non-expedited destination LOG_ARCHIVE_DEST_2 thread 1 sequence 32060 host (DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=tcp)(HOST=hera)(PORT=1521)))(CONNECT_DATA=(SERVICE_NAME=hera_XPT.contemporaryrealty.com)(INSTANCE_NAME=hera)(SERVER=dedicated)))
Thu Aug 2 00:11:52 2007
ARC1: Completed archiving thread 1 sequence 32060 (424934712-424975107) (zeus)
ARCH: Connecting to console port...
FAL[server, ARC1]: Begin FAL noexpedite archive (branch 554751951 thread 1 sequence 32060 dest (DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=tcp)(HOST=hera)(PORT=1521)))(CONNECT_DATA=(SERVICE_NAME=hera_XPT.contemporaryrealty.com)(INSTANCE_NAME=hera)(SERVER=dedicated))))
Thu Aug 2 00:11:52 2007
ARC1: Creating remote archive destination LOG_ARCHIVE_DEST_2: '(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=tcp)(HOST=hera)(PORT=1521)))(CONNECT_DATA=(SERVICE_NAME=hera_XPT.contemporaryrealty.com)(INSTANCE_NAME=hera)(SERVER=dedicated)))' (thread 1 sequence 32060)
(zeus)
ARCH: Connecting to console port...
Thu Aug 2 00:31:52 2007
ARC1: Closing remote archive destination LOG_ARCHIVE_DEST_2: '(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=tcp)(HOST=hera)(PORT=1521)))(CONNECT_DATA=(SERVICE_NAME=hera_XPT.contemporaryrealty.com)(INSTANCE_NAME=hera)(SERVER=dedicated)))'
(zeus)
Thu Aug 2 00:31:52 2007
ARCH: Connecting to console port...
FAL[server, ARC1]: Complete FAL noexpedite archive (thread 1 sequence 32060 destination (DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=tcp)(HOST=hera)(PORT=1521)))(CONNECT_DATA=(SERVICE_NAME=hera_XPT.contemporaryrealty.com)(INSTANCE_NAME=hera)(SERVER=dedicated))))
Thu Aug 2 16:01:07 2007
Private_strands 48 at log switch
Thread 1 advanced to log sequence 32062
Current log# 9 seq# 32062 mem# 0: +DATA/zeus/onlinelog/group_9.4239.608225221
Current log# 9 seq# 32062 mem# 1: /home/oracle/backup/ZEUS/onlinelog/o1_mf_9_2q6ds4yp_.log
Thu Aug 2 16:01:07 2007
ARC1: Evaluating archive thread 1 sequence 32061
ARC1: Destination LOG_ARCHIVE_DEST_2 archival not expedited
Thu Aug 2 16:01:07 2007
ARC1: Beginning to archive thread 1 sequence 32061 (424975107-425075630) (zeus)
ARCH: Connecting to console port...
Thu Aug 2 16:01:07 2007
ARC1: Creating local archive destination LOG_ARCHIVE_DEST_1: '/home/oracle/backup/ZEUS/archivelog/2007_08_02/o1_mf_1_32061_%u_.arc' (thread 1 sequence 32061)
(zeus)
ARCH: Connecting to console port...
Thu Aug 2 16:01:18 2007
ARC1: Closing local archive destination LOG_ARCHIVE_DEST_1: '/home/oracle/backup/ZEUS/archivelog/2007_08_02/o1_mf_1_32061_3c4g0449_.arc'
(zeus)
Thu Aug 2 16:01:18 2007
ARCH: Connecting to console port...
Committing creation of archivelog '/home/oracle/backup/ZEUS/archivelog/2007_08_02/o1_mf_1_32061_3c4g0449_.arc'
Created Oracle managed file /home/oracle/backup/ZEUS/archivelog/2007_08_02/o1_mf_1_32061_3c4g0449_.arc
Invoking non-expedited destination LOG_ARCHIVE_DEST_2 thread 1 sequence 32061 host (DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=tcp)(HOST=hera)(PORT=1521)))(CONNECT_DATA=(SERVICE_NAME=hera_XPT.contemporaryrealty.com)(INSTANCE_NAME=hera)(SERVER=dedicated)))
Thu Aug 2 16:01:20 2007
ARC1: Completed archiving thread 1 sequence 32061 (424975107-425075630) (zeus)
ARCH: Connecting to console port...
FAL[server, ARC1]: Begin FAL noexpedite archive (branch 554751951 thread 1 sequence 32061 dest (DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=tcp)(HOST=hera)(PORT=1521)))(CONNECT_DATA=(SERVICE_NAME=hera_XPT.contemporaryrealty.com)(INSTANCE_NAME=hera)(SERVER=dedicated))))
Thu Aug 2 16:01:20 2007
ARC1: Creating remote archive destination LOG_ARCHIVE_DEST_2: '(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=tcp)(HOST=hera)(PORT=1521)))(CONNECT_DATA=(SERVICE_NAME=hera_XPT.contemporaryrealty.com)(INSTANCE_NAME=hera)(SERVER=dedicated)))' (thread 1 sequence 32061)
(zeus)
ARCH: Connecting to console port...
Thu Aug 2 16:26:56 2007
ARC1: Closing remote archive destination LOG_ARCHIVE_DEST_2: '(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=tcp)(HOST=hera)(PORT=1521)))(CONNECT_DATA=(SERVICE_NAME=hera_XPT.contemporaryrealty.com)(INSTANCE_NAME=hera)(SERVER=dedicated)))'
(zeus)
Thu Aug 2 16:26:56 2007
ARCH: Connecting to console port...
FAL[server, ARC1]: Complete FAL noexpedite archive (thread 1 sequence 32061 destination (DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=tcp)(HOST=hera)(PORT=1521)))(CONNECT_DATA=(SERVICE_NAME=hera_XPT.contemporaryrealty.com)(INSTANCE_NAME=hera)(SERVER=dedicated))))
FAL[server, ARC1]: Begin FAL archive (branch 554751951 thread 1 sequence 32061 dest (DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=tcp)(HOST=hera)(PORT=1521)))(CONNECT_DATA=(SERVICE_NAME=hera_XPT.contemporaryrealty.com)(INSTANCE_NAME=hera)(SERVER=dedicated))))
Thu Aug 2 16:27:00 2007
Errors in file /u01/app/oracle/admin/olympus/zeus_arc1_2917.trc:
ORA-16401: archivelog rejected by RFS
FAL[server, ARC1]: Complete FAL archive (thread 1 sequence 32061 destination (DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=tcp)(HOST=hera)(PORT=1521)))(CONNECT_DATA=(SERVICE_NAME=hera_XPT.contemporaryrealty.com)(INSTANCE_NAME=hera)(SERVER=dedicated))))
Thu Aug 2 16:27:30 2007
FAL[server, ARC1]: Begin FAL archive (branch 554751951 thread 1 sequence 32061 dest (DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=tcp)(HOST=hera)(PORT=1521)))(CONNECT_DATA=(SERVICE_NAME=hera_XPT.contemporaryrealty.com)(INSTANCE_NAME=hera)(SERVER=dedicated))))
Thu Aug 2 16:27:30 2007
ARC1: Creating remote archive destination LOG_ARCHIVE_DEST_2: '(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=tcp)(HOST=hera)(PORT=1521)))(CONNECT_DATA=(SERVICE_NAME=hera_XPT.contemporaryrealty.com)(INSTANCE_NAME=hera)(SERVER=dedicated)))' (thread 1 sequence 32061)
(zeus)
ARCH: Connecting to console port...
Thu Aug 2 16:27:30 2007
Errors in file /u01/app/oracle/admin/olympus/zeus_arc1_2917.trc:
ORA-16401: archivelog rejected by RFS
FAL[server, ARC1]: Complete FAL archive (thread 1 sequence 32061 destination (DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=tcp)(HOST=hera)(PORT=1521)))(CONNECT_DATA=(SERVICE_NAME=hera_XPT.contemporaryrealty.com)(INSTANCE_NAME=hera)(SERVER=dedicated))))
Thu Aug 2 16:28:01 2007
FAL[server, ARC1]: Begin FAL archive (branch 554751951 thread 1 sequence 32061 dest (DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=tcp)(HOST=hera)(PORT=1521)))(CONNECT_DATA=(SERVICE_NAME=hera_XPT.contemporaryrealty.com)(INSTANCE_NAME=hera)(SERVER=dedicated))))
Thu Aug 2 16:28:01 2007
ARC1: Creating remote archive destination LOG_ARCHIVE_DEST_2: '(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=tcp)(HOST=hera)(PORT=1521)))(CONNECT_DATA=(SERVICE_NAME=hera_XPT.contemporaryrealty.com)(INSTANCE_NAME=hera)(SERVER=dedicated)))' (thread 1 sequence 32061)
(zeus)
ARCH: Connecting to console port...
--- snip ---
Standby (Hera):
--- snip ---
Tue Jul 31 18:19:21 2007
Media Recovery Waiting for thread 1 sequence 32057
Wed Aug 1 00:39:00 2007
RFS[6]: No standby redo logfiles created
RFS[6]: Archived Log: '/home/oracle/backup/HERA/archivelog/2007_08_01/o1_mf_1_32057_3bzok64f_.arc'
Committing creation of archivelog '/home/oracle/backup/HERA/archivelog/2007_08_01/o1_mf_1_32057_3bzok64f_.arc'
Created Oracle managed file /home/oracle/backup/HERA/archivelog/2007_08_01/o1_mf_1_32057_3bzok64f_.arc
Wed Aug 1 01:07:49 2007
Media Recovery Log /home/oracle/backup/HERA/archivelog/2007_08_01/o1_mf_1_32057_3bzok64f_.arc
Wed Aug 1 01:10:30 2007
Media Recovery Waiting for thread 1 sequence 32058
Wed Aug 1 07:19:32 2007
RFS[6]: No standby redo logfiles created
RFS[6]: Archived Log: '/home/oracle/backup/HERA/archivelog/2007_08_01/o1_mf_1_32058_3c0f09qz_.arc'
Committing creation of archivelog '/home/oracle/backup/HERA/archivelog/2007_08_01/o1_mf_1_32058_3c0f09qz_.arc'
Created Oracle managed file /home/oracle/backup/HERA/archivelog/2007_08_01/o1_mf_1_32058_3c0f09qz_.arc
Wed Aug 1 07:39:38 2007
Media Recovery Log /home/oracle/backup/HERA/archivelog/2007_08_01/o1_mf_1_32058_3c0f09qz_.arc
Wed Aug 1 07:40:19 2007
Media Recovery Waiting for thread 1 sequence 32059
Wed Aug 1 23:05:28 2007
RFS[6]: No standby redo logfiles created
RFS[6]: Archived Log: '/home/oracle/backup/HERA/archivelog/2007_08_01/o1_mf_1_32059_3c24fshp_.arc'
Committing creation of archivelog '/home/oracle/backup/HERA/archivelog/2007_08_01/o1_mf_1_32059_3c24fshp_.arc'
Created Oracle managed file /home/oracle/backup/HERA/archivelog/2007_08_01/o1_mf_1_32059_3c24fshp_.arc
Wed Aug 1 23:25:38 2007
Media Recovery Log /home/oracle/backup/HERA/archivelog/2007_08_01/o1_mf_1_32059_3c24fshp_.arc
Wed Aug 1 23:26:04 2007
Media Recovery Waiting for thread 1 sequence 32060
Thu Aug 2 04:11:52 2007
RFS[6]: No standby redo logfiles created
RFS[6]: Archived Log: '/home/oracle/backup/HERA/archivelog/2007_08_02/o1_mf_1_32060_3c2pdchh_.arc'
Committing creation of archivelog '/home/oracle/backup/HERA/archivelog/2007_08_02/o1_mf_1_32060_3c2pdchh_.arc'
Created Oracle managed file /home/oracle/backup/HERA/archivelog/2007_08_02/o1_mf_1_32060_3c2pdchh_.arc
Thu Aug 2 04:31:56 2007
Media Recovery Log /home/oracle/backup/HERA/archivelog/2007_08_02/o1_mf_1_32060_3c2pdchh_.arc
Thu Aug 2 04:32:45 2007
Media Recovery Waiting for thread 1 sequence 32061
Thu Aug 2 20:01:20 2007
RFS[6]: No standby redo logfiles created
RFS[6]: Archived Log: '/home/oracle/backup/HERA/archivelog/2007_08_02/o1_mf_1_32061_3c4g0kjo_.arc'
Committing creation of archivelog '/home/oracle/backup/HERA/archivelog/2007_08_02/o1_mf_1_32061_3c4g0kjo_.arc'
Created Oracle managed file /home/oracle/backup/HERA/archivelog/2007_08_02/o1_mf_1_32061_3c4g0kjo_.arc
Thu Aug 2 20:26:59 2007
Fetching gap sequence in thread 1, gap sequence 32061-32061
FAL[client]: Trying FAL server: (DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=tcp)(HOST=zeus)(PORT=1521)))(CONNECT_DATA=(SERVICE_NAME=zeus_XPT.contemporaryrealty.com)(SERVER=dedicated)))
Thu Aug 2 20:27:00 2007
RFS[6]: Archivelog thread 1 sequence 32061 cannot be reused
Thu Aug 2 20:27:00 2007
Errors in file /u01/app/oracle/admin/olympus/hera_rfs_6367.trc:
ORA-16401: archivelog rejected by RFS
Thu Aug 2 20:27:30 2007
RFS[6]: Archivelog thread 1 sequence 32061 cannot be reused
Thu Aug 2 20:27:30 2007
Errors in file /u01/app/oracle/admin/olympus/hera_rfs_6367.trc:
ORA-16401: archivelog rejected by RFS
--- snip ---
And on both of them, the last few lines repeat over and over again. The trace files mentioned do not give any more info than the alert logs do, for some reason.
* To the best of my knowledge and records, no one was using the database at the time, nor were any changes made to the configuration.
* This setup has been working fine for about 9 months. There was another problem about month ago involving a screwy datafile (see details here:
2019964 but I don't see that being directly related right now.
* Dataguard Broker is set up and doesn't say that anything is wrong. I know that DGB changes several init params itself, so I'm not sure if I should touch any of those myself or not. If they're relevant here, I will post the related init params upon request.
I've searched through OTN and elsewhere, but was not able to find any useful info, so I'm at a loss of where to go from here. Any help would be greatly appreciated!