I have installed OEM 12.1.0.3.0 with the OMS and repository on separate servers. Occasionally I see flurries of messages from all of our monitored targets stating the following -
Host=aporia.ucdavis.edu
Target type=Host
Target name=aporia.ucdavis.edu
Categories=Availability
Message=Agent is unable to communicate with the OMS. (REASON = Agent is Unreachable (REASON : Agent to OMS Communication is brokenFailure connecting to https://oracle-emrep1.ucdavis.edu:4904/empbs/upload , err Connection refused ).
.
.
Availability status=Agent Unreachable
.
.
and I am trying to understand what is causing the problem. Looking at the gcagent.log file (it is too large to post) on the OMS server I see the following -
2013-10-22 06:11:11,785 [24607:A4C14822] INFO - There was no change in dynamic properties for target [j2ee_application./EMGC_GCDomain/GCDomain/EMGC_OMS1/empbs$1945]
2013-10-22 06:11:55,025 [24603:237B90F9:GC.Executor.987 (oracle_apache:/EMGC_GCDomain/instance1/ohs1:Response)] INFO - Target [oracle_apache./EMGC_GCDomain/instance1/ohs1] is marked as in DOWN state
2013-10-22 06:11:55,025 [24603:237B90F9] INFO - Target [oracle_apache./EMGC_GCDomain/instance1/ohs1] is marked as in DOWN state
2013-10-22 06:11:55,151 [39:C09F1B94:GC.Upload[1]] WARN - Communication error with repository URL = https://oracle-emrep1.ucdavis.edu:4904/empbs/upload
and shortly thereafter -
2013-10-22 06:12:40,441 [1:3305B9] INFO - Target Events are shutting down
2013-10-22 06:12:40,442 [1:3305B9] INFO - Done: SHUTDOWN on Target Event Manager
2013-10-22 06:12:40,443 [1:3305B9] INFO - Invoking SHUTDOWN (0) on CollectionItemTracker
and again a short time later
2013-10-22 06:27:19,301 [1:3305B9] INFO - Invoking STARTUP_P1 on Request Dispatcher
2013-10-22 06:27:19,466 [88:E3AE5994:GC.Executor.3 (j2ee_application:/EMGC_GCDomain/GCDomain/EMGC_OMS1/empbs:Response)] INFO - Target [j2ee_application./EMGC_GCDomain/GCDomain/EMGC_OMS1/empbs] is marked as in DOWN state
2013-10-22 06:27:19,474 [88:E3AE5994] INFO - Target [j2ee_application./EMGC_GCDomain/GCDomain/EMGC_OMS1/empbs] is marked as in DOWN state
In the gcagent.log file on target aporia (this information is reported on other monitored targets as well) I see -
2013-10-22 06:11:19,355 [476:A1B91698] INFO - attempting another heartbeat
2013-10-22 06:11:19,373 [476:A1B91698] WARN - Ping communication error
o.s.emSDK.agent.comm.exception.ConnectException [Failure connecting to https://oracle-emrep1.ucdavis.edu:4904/empbs/upload , err Connection refused]
java.net.ConnectException [Connection refused]
2013-10-22 06:11:22,758 [76:673B8035:GC.SysExecutor.12 (AgentSystemMonitorTask)] WARN - Subsystem (Upload Manager) returned bad status of {+ Upload Manager: *Warning* +}
Also while this is going on attempts to log in to the OEM console fail with a message indicating the communication with the OMS fails.
Ultimately this resolves and everything comes back and restarts but I don't understand what is going on. The problem is intermittent and was noticed initially after upgrading to 12.1.0.1.0 but I recently upgraded to 12.1.0.3.0 and the problem persists. Based on this statement in the log '(oracle_apache:/EMGC_GCDomain/instance1/ohs1:Response)' I suspect that weblogic may be involved but at this point I am not sure where else to look or what else to look for. If anyone has any ideas I would greatly appreciate hearing them.
Thank you.
Bill Wagman