3.1.1 upgrade breaks OEM 12c Infrastructure Cloud
Greetings,
After upgrading from 3.0.3 to 3.1.1 (manager first then servers) all of the Oracle VM targets in OEM 12c Infrastructure Cloud have metric collection errors, specifically "ERROR - Failed to connect to OVM Manager1010", i.e. oracle.sysman.vt.ovm.integration.api.EMOVMCommunicationServiceException: 1010: Error while performing the operation: Connection to the remote OVM Manager url tcp://ovm-laforge.sf.mokum.com:54321 failed. Please make sure that the OVM Manager version is 3.0.2 or greater and the service url and the credentials are correct.Check exception cause and logs for more details.[com.oracle.ovm.mgr.api.exception.FailedOperationException:OVMAPI_6000E Internal Error: Connection refused Connection refused.
We walked through the check in "Oracle VM Manager Login Fails with OVMAPI_6000E [ID 1416443.1" , all systems go....
sh ./ovm_upgrade.sh --dbuser=ovs --dbpass=xxxxxxx --dbhost=localhost --dbport=1521 --dbsid=orcl
COMMAND: ./ovm_upgrade.sh --dbuser=ovs --dbpass=xxxxxxx --dbhost=localhost --dbport=1521 --dbsid=orcl
find: /tmp/ovmpatches: No such file or directory
Copying xsl files to patch path location
`./transform_003000002000_010.xsl' -> `/tmp/ovmpatches/transform_003000002000_010.xsl'
`./transform_003000003000_010.xsl' -> `/tmp/ovmpatches/transform_003000003000_010.xsl'
`./transform_003001001000_010.xsl' -> `/tmp/ovmpatches/transform_003001001000_010.xsl'
`./transform_003001001000_020.xsl' -> `/tmp/ovmpatches/transform_003001001000_020.xsl'
Copying deleted classes files to patch path location
`./deletedClasses.xml' -> `/tmp/ovmpatches/deletedClasses.xml'
INFO (OvmUpgrade.java:206) Oracle OVM Manager Upgrade Processor
INFO (OvmUpgrade.java:207)
INFO (OvmUpgrade.java:488) Upgrade Initialization Starting
INFO (OdofDirector.java:220) Oracle Distributed Object Fabric (ODOF): Copyright (C) 2007, 2012 Oracle. All rights reserved.
INFO (OdofDirector.java:221) ODOF Version: 1.0.0.0
INFO (OdofDirector.java:222) Initializing...
INFO (OdofDirector.java:279) Initialization Complete!
INFO (OvmUpgrade.java:490) Upgrade Initialization Complete
The /u01/app/oracle/ovm-manager-3/machine1/base_adf_domain/servers/AdminServer/logs/AdminServer.log logs are clean....
This log is quickly filling up our free space (5G worth of logs :-): /u01/app/oracle/product/agent/agent_inst/sysman/log/gcagent_errors.log with:
2012-05-12 17:43:38,405 [129:GC.Executor.7 (oracle_vm_zone:Page:Response) (oracle_vm_zone:Page:Response:Response)] ERROR - Failed to connect to OVM Manager1010: Error while performing the operation: Connection to the remote OVM Manager url tcp://ovm-laforge.sf.mokum.com:54321 failed. Please make sure that the OVM Manager version is 3.0.2 or greater and the service url and the credentials are correct.Check exception cause and logs for more details.[com.oracle.ovm.mgr.api.exception.FailedOperationException:OVMAPI_6000E Internal Error: Connection refused Connection refused
From the manager:
# lsof -i|grep 54321
java 3067 oracle 454u IPv6 710576 0t0 TCP localhost.localdomain:43620->localhost.localdomain:54321 (ESTABLISHED)
java 3067 oracle 511u IPv6 19638 0t0 TCP localhost.localdomain:54321 (LISTEN)
java 3067 oracle 710u IPv6 19639 0t0 TCP localhost.localdomain:64163->localhost.localdomain:54321 (ESTABLISHED)
java 3067 oracle 711u IPv6 19640 0t0 TCP localhost.localdomain:54321->localhost.localdomain:64163 (ESTABLISHED)
java 3067 oracle 712u IPv6 19641 0t0 TCP localhost.localdomain:54321->localhost.localdomain:64164 (ESTABLISHED)
java 3067 oracle 713u IPv6 19642 0t0 TCP localhost.localdomain:64164->localhost.localdomain:54321 (ESTABLISHED)
java 3067 oracle 714u IPv6 19643 0t0 TCP localhost.localdomain:54321->localhost.localdomain:64166 (ESTABLISHED)
java 3067 oracle 715u IPv6 19645 0t0 TCP localhost.localdomain:64166->localhost.localdomain:54321 (ESTABLISHED)
java 3067 oracle 716u IPv6 19646 0t0 TCP localhost.localdomain:54321->localhost.localdomain:64168 (ESTABLISHED)
java 3067 oracle 717u IPv6 19649 0t0 TCP localhost.localdomain:64168->localhost.localdomain:54321 (ESTABLISHED)
java 3067 oracle 718u IPv6 19650 0t0 TCP localhost.localdomain:54321->localhost.localdomain:64173 (ESTABLISHED)
java 3067 oracle 720u IPv6 19781 0t0 TCP localhost.localdomain:64173->localhost.localdomain:54321 (ESTABLISHED)
java 3067 oracle 721u IPv6 19782 0t0 TCP localhost.localdomain:54321->localhost.localdomain:64175 (ESTABLISHED)
java 3067 oracle 723u IPv6 19798 0t0 TCP localhost.localdomain:64175->localhost.localdomain:54321 (ESTABLISHED)
java 3067 oracle 749u IPv6 20742 0t0 TCP localhost.localdomain:64235->localhost.localdomain:54321 (ESTABLISHED)
java 3067 oracle 750u IPv6 20743 0t0 TCP localhost.localdomain:54321->localhost.localdomain:64240 (ESTABLISHED)
java 3067 oracle 757u IPv6 20149 0t0 TCP localhost.localdomain:54321->localhost.localdomain:64207 (ESTABLISHED)
java 3067 oracle 758u IPv6 20761 0t0 TCP localhost.localdomain:64240->localhost.localdomain:54321 (ESTABLISHED)
java 3067 oracle 759u IPv6 20167 0t0 TCP localhost.localdomain:64207->localhost.localdomain:54321 (ESTABLISHED)
java 3067 oracle 760u IPv6 20168 0t0 TCP localhost.localdomain:54321->localhost.localdomain:64209 (ESTABLISHED)
java 3067 oracle 761u IPv6 20185 0t0 TCP localhost.localdomain:64209->localhost.localdomain:54321 (ESTABLISHED)
java 3067 oracle 762u IPv6 20186 0t0 TCP localhost.localdomain:54321->localhost.localdomain:64235 (ESTABLISHED)
java 3067 oracle 763u IPv6 20762 0t0 TCP localhost.localdomain:54321->localhost.localdomain:64247 (ESTABLISHED)
java 3067 oracle 765u IPv6 20802 0t0 TCP localhost.localdomain:64247->localhost.localdomain:54321 (ESTABLISHED)
java 3067 oracle 770u IPv6 464327 0t0 TCP localhost.localdomain:54321->localhost.localdomain:43620 (ESTABLISHED)
java 3067 oracle 771u IPv6 21578 0t0 TCP localhost.localdomain:54321->localhost.localdomain:64313 (ESTABLISHED)
java 3067 oracle 779u IPv6 21617 0t0 TCP localhost.localdomain:64313->localhost.localdomain:54321 (ESTABLISHED)
java 3067 oracle 781u IPv6 25612 0t0 TCP localhost.localdomain:54321->localhost.localdomain:11597 (ESTABLISHED)
java 3067 oracle 783u IPv6 74799 0t0 TCP localhost.localdomain:11597->localhost.localdomain:54321 (ESTABLISHED)
I recollect in 12.0, not 12.1, there was a required OVM password deep in a hidden nook in the OEM Infrastructure Cloud GUI, I cant find it in 12.1, could this be the issue?
Respectfully,
Roddy