We have OEM Grid Control 11g installed on Linux server. And we have Oracle 11gR2 RAC database on HP-UX server. I installed and configured OEM Agent on both RAC servers and it was working with out any issue, I was able to monitor from Grid Control. We have 3 RAC databases on the RAC server and I was able to monitor all 3.
Since yesterday I am getting the following error for on Grid Control for first node of the RAC for one database, other 2 databases are fine.
Status: Failed
Details: Failed to connect to database instance: ORA-12537: TNS:connection closed (DBD ERROR: OCIServerAttach).
And I checked Grid Control Server log (/app/oracle/gc_inst/em/EMGC_OMS1/sysman/log/emoms.trc) and I see the following warnings for that particular database
2012-01-31 09:32:13,272 [EMUI_09_32_13_/console/database/instance/sitemap] WARN console.SwbUtil logp.251 - Unable to communicate with agent for target with name eaiprod_eaiprod1 and type oracle_database
2012-01-31 09:48:13,950 [EMUI_09_48_12_/console/database/config] WARN emd.targets setContents.92 - ShortName ShortName Average Synchronous Single-Block Read Latency (ms) very long. Has been truncated.
2012-01-31 09:48:14,003 [EMUI_09_48_12_/console/database/config] WARN emd.targets setContents.101 - Label Total messages processed per queue per subscriber per minute in the last interval very long. Has been truncated.
2012-01-31 09:48:14,006 [EMUI_09_48_12_/console/database/config] WARN emd.targets setContents.101 - Label Total messages received per queue per subscriber per minute in the last interval very long. Has been truncated.
2012-01-31 09:48:14,008 [EMUI_09_48_12_/console/database/config] WARN emd.targets setContents.101 - Label Messages processed per queue (%) per subscriber per minute in the last interval very long. Has been truncated.
2012-01-31 09:48:14,011 [EMUI_09_48_12_/console/database/config] WARN emd.targets setContents.101 - Label Age of the first message in persistent queue per subscriber (seconds) very long. Has been truncated.
2012-01-31 09:48:14,014 [EMUI_09_48_12_/console/database/config] WARN emd.targets setContents.101 - Label Age of the first message in the buffered queue per queue (seconds) very long. Has been truncated.
2012-01-31 09:48:31,698 [Thread-353] WARN emd.targets setContents.92 - ShortName ShortName Average Synchronous Single-Block Read Latency (ms) very long. Has been truncated.
2012-01-31 09:48:31,774 [Thread-353] WARN emd.targets setContents.101 - Label Total messages processed per queue per subscriber per minute in the last interval very long. Has been truncated.
2012-01-31 09:48:31,775 [Thread-353] WARN emd.targets setContents.101 - Label Total messages received per queue per subscriber per minute in the last interval very long. Has been truncated.
2012-01-31 09:48:31,775 [Thread-353] WARN emd.targets setContents.101 - Label Messages processed per queue (%) per subscriber per minute in the last interval very long. Has been truncated.
2012-01-31 09:48:31,778 [Thread-353] WARN emd.targets setContents.101 - Label Age of the first message in persistent queue per subscriber (seconds) very long. Has been truncated.
2012-01-31 09:48:31,780 [Thread-353] WARN emd.targets setContents.101 - Label Age of the first message in the buffered queue per queue (seconds) very long. Has been truncated.
28762,1 Bot
But I am able to connect the database (both the instances) using TOAD or some other SQL Client. I stopped and restarted the agent but no changes.
I am not sure where to look into it, can some one help me to understand the issue.