Hi all,
this help request is divided in three parts: a SCENARIO where you find information about the environment, a PROBLEM where you find information about the problem and DIAGNOSIS where you find information about my diagnostic attempts so far.
SCENARIO:
I have successfully installed EM13c (on Windows Server 2008 R2) and deployed its 13.1.1.0 agent on a test target server (Windows Server 2008 R2) where 6 Oracle database instances (11.2.0.3) are running. EM13c is also monitoring its own Oracle12c database (which is running on the same server where OMS is running).
The status of the agent in the target machine is as follows:
F:\>C:\Oracle\OEM13c\agents\agent_13.1.0.0.0\agent_13.1.0.0.0\bin\emctl.bat status agent
Oracle Enterprise Manager Cloud Control 13c Release 1
Copyright (c) 1996, 2015 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
Agent Version : 13.1.0.0.0
OMS Version : 13.1.0.0.0
Protocol Version : 12.1.0.1.0
Agent Home : C:/Oracle/OEM13c/agents/agent_13.1.0.0.0/agent_inst
Agent Log Directory : C:/Oracle/OEM13c/agents/agent_13.1.0.0.0/agent_inst\sysman\log
Agent Binaries : C:\Oracle\OEM13c\agents\agent_13.1.0.0.0\agent_13.1.0.0.0
Core JAR Location : C:\Oracle\OEM13c\agents\agent_13.1.0.0.0\agent_13.1.0.0.0\jlib
Agent Process ID : 5484
Parent Process ID : 5920
Agent URL : https://MYSERVER.MYADDRESS:3872/emd/main/
Local Agent URL in NAT : https://MYSERVER.MYADDRESS:3872/emd/main/
Repository URL : https://MYSERVER.MYADDRESS:1159/empbs/upload
Started at : 2016-03-08 16:18:00
Started by user : MYSERVER$
Operating System : Windows version 6.1 (amd64)
Number of Targets : 11
Last Reload : (none)
Last successful upload : 2016-03-10 10:00:43
Last attempted upload : 2016-03-10 10:00:43
Total Megabytes of XML files uploaded so far : 2.17
Number of XML files pending upload : 0
Size of XML files pending upload(MB) : 0
Available disk space on upload filesystem : 29.05%
Collection Status : Collections enabled
Heartbeat Status : Ok
Last attempted heartbeat to OMS : 2016-03-10 10:03:12
Last successful heartbeat to OMS : 2016-03-10 10:03:12
Next scheduled heartbeat to OMS : 2016-03-10 10:04:12
---------------------------------------------------------------
Agent is Running and Ready
PROBLEM:
3 instances are being perfectly monitored while the other 3 ones have the following errors:
1) Status is "n/a" (I've already read many threads about it but...no, there's no firewall problem in my case). If I put the mouse pointer over the n/a icon I can read "Availability Evaluation Error Since 09 March 2016 15:06:57"
2) Metrics are not shown
3) If you want to manage the three instances objects (e.g. tablespaces, views and so on) you cannot get connected to them as sys (while you can do it with the other 3 instances which are perfectly being monitored). The error shown is "ORA-01017: invalid username/password; logon denied" although the logon credentials ARE correct.
4) If you want to manage the three instance objects (e.g. tablespaces, views and so on) you can only do it by getting connected to them as dbsnmp. However the connection is shown as being UNencrypted (whereas the connection is encrypted when you get connected, either as sys or dbsnmp, to the other 3 instances which are perfectly being monitored)
DIAGNOSIS:
All the 6 instances are working well (and the agent is fine as you read above and also on account of the fact that three intstances are perfectly being monitored),
However the following can be found from EM13c point of view:
A) I checked all the logs (at least all that I was able to find) and only found a hint within gcagent.log (found at C:\Oracle\OEM13c\agents\agent_13.1.0.0.0\agent_inst\sysman\log)
B) In gcagent.log, for all the three instances which are suffering from the above problems, there are multiple occurrences [starting from the date shown as described in the problem number 1) ] of the following:
2016-03-09 15:06:53,440 [5792:D4C6EF27] INFO - Target: [oracle_database.DBAX] is fully broken: Dynamic Category property error (code=0x400)
There is also this message:
2016-03-09 15:12:32,666 [5951:FCA89A76] INFO - >>> Reporting exception: oracle.sysman.emSDK.agent.client.exception.NoSuchTargetException: the oracle_database target "DBAX" does not exist (request id 1) <<<
C) I found this document describing my exact case. ORACLE suggested to run this query:
select v.version "DBVersion", p.value "DBDomain" from
(select nvl((select version from (select version || '.'|| id version
from dba_registry_history where NAMESPACE = 'SERVER'
and BUNDLE_SERIES='PSU'
order by ACTION_TIME desc) where rownum = 1),
(select version from v$instance)) version from dual) v, v$parameter p
where p.name='db_domain'
and BUNDLE_SERIES='PSU' ;
Unfortunately I found that that BUNDLE_SERIES column does not exist in EITHER the three isntances in "fully broken" state AND in the ones which are being perfectly monitored (in fact I can also see "database target does not exist" messages for the three perfectly monitored instances as well!!).
Therefore the solution described in the above document doesn't really seem to fit my case.
Would anyone of you have a good idea about this trouble or I'd better open a SR?
Thank you very much in advance for any precious help from you, kind regards
Michele