Hi guys,
I Installed oracle 11gr2 on VM (OS RHEL Linux) ,then created database using dbca which also configured OEM and listeners
post which OEM works for few minutes and goes down on its own when checked showing agent is down,restarting using >>emctl start dbconsole
not work ,it bring up oem but agent not coming up,when i shutdown and restart VM(OS,Linux) again OEM and agent work for few minutes and i could use all features but again goes
down after few minutes,
emctl status dbconsole shows as below:
EM Daemon is not running
or
em agent not running
after oracle 11gr2 installation,i changed hostname of vm/host,checked required files have changed hostname if this issue can be related to this and any specific file i could check if hostname is showing old(but no error related to hostname)
please guide me how to fix this
,so far below troubleshooting steps is done:
1)-Reconfiguring(dropping,recreating oem/repos multiple times)
emca -config dbcontrol db -repos create
below was message after first installation and recreation
----------------------------------------------------------------------------------------------
Error securing Database Control, Database Control has been brought up in non-secure mode. To secure the Database Control execute the following command(s):
1) Set the environment variable ORACLE_SID to
2) /u01/app/oracle/product/11.2.0/dbhome_1/bin/emctl stop dbconsole
3) /u01/app/oracle/product/11.2.0/dbhome_1/bin/emctl config emkey -repos -sysman_pwd <pwd>
4) /u01/app/oracle/product/11.2.0/dbhome_1/bin/emctl secure dbconsole -sysman_pwd <pwd>
5) /u01/app/oracle/product/11.2.0/dbhome_1/bin/emctl start dbconsole
To secure Em Key, run /u01/app/oracle/product/11.2.0/dbhome_1/bin/emctl config emkey -remove_from_repos -sysman_pwd < Password for SYSMAN user >
----------------------------------------------------------------------------------------------------
2)-Timezone related changes
emctl config agent updateTZ
cho $TZ
[oracle@RHENODE2 log]$ export TZ=Etc/GMT+
3)-checking and changing listener
4)- Replacing server.xml file from other location after backing it up
5) Bouncing OEM,Listener multiple times,bouncing db,OS/VM
6)changing password of sysman,sys (to rule out if any secured connection)
account_status for users is open
also
config emkey -repos -sysman_pwd <pwd>
emctl secure dbconsole -sysman_pwd <pwd>
emctl start dbconsole
I am uploading related log and trace files output
------------------------------------------------------------------------------------------------------------------------------------- --------------------------------------------
emctl.log
15933 :: Wed Oct 25 04:03:31 2017::AgentStatus.pm:Processing start agent
emagent.log
017-10-25 03:26:23,947 Thread-2331187008 Starting Agent 10.2.0.4.2 from /u01/app/oracle/product/11.2.0/dbhome_1 (00701)
2017-10-25 03:26:23,952 Thread-2331187008 <Agent Startup> : Startup of HTTP LISTENER failure (00716)
2017-10-25 03:42:09,318 Thread-3664656192 Starting Agent 10.2.0.4.2 from /u01/app/oracle/product/11.2.0/dbhome_1 (00701)
2017-10-25 03:42:09,319 Thread-3664656192 <Agent Startup> : Startup of HTTP LISTENER failure (00716)
2017-10-25 03:46:58,100 Thread-3298998080 Starting Agent 10.2.0.4.2 from /u01/app/oracle/product/11.2.0/dbhome_1 (00701)
2017-10-25 03:46:58,100 Thread-3298998080 <Agent Startup> : Startup of HTTP LISTENER failure (00716)
2017-10-25 04:12:48,675 Thread-3472807744 Starting Agent 10.2.0.4.2 from /u01/app/oracle/product/11.2.0/dbhome_1 (00701)
2017-10-25 04:12:48,676 Thread-3472807744 <Agent Startup> : Startup of HTTP LISTENER failure (00716)
2017-10-25 21:42:56,615 Thread-4101580608 Starting Agent 10.2.0.4.2 from /u01/app/oracle/product/11.2.0/dbhome_1 (00701)
2017-10-25 21:42:59,434 Thread-4101580608 [Oracle Exadata Storage Server] InstanceProperty (MgmtIPAddr2) is marked OPTIONAL but is being used (00506)
2017-10-25 21:43:05,336 Thread-4101580608 [Load Balancer Switch] InstanceProperty (snmpTimeout) is marked OPTIONAL but is being used (00506)
2017-10-25 21:43:54,990 Thread-4101580608 EMAgent started successfully (00702)
emoms.log
017-10-25 21:56:58,006 [OC4J Launcher] ERROR app.ContextInitializer contextInitialized.420 - Integration Class not found: oracle.sysman.emCLI.CLIIntg
2017-10-25 21:56:58,008 [OC4J Launcher] ERROR app.ContextInitializer contextInitialized.420 - Integration Class not found: oracle.sysman.emCLI.CLIDownloadIntg
2017-10-25 21:56:58,615 [OC4J Launcher] ERROR app.ContextInitializer contextInitialized.420 - Integration Class not found: oracle.sysman.eml.target.slb.common.SLBIntegration
2017-10-25 21:57:02,127 [OC4J Launcher] ERROR app.ContextInitializer contextInitialized.420 - Integration Class not found: oracle.sysman.pp.asprov.ui.intg.ASProvisioningIntegration
2017-10-25 21:57:02,128 [OC4J Launcher] ERROR app.ContextInitializer contextInitialized.420 - Integration Class not found: oracle.sysman.pp.paf.sample.ui.intg.PAFDemoIntegration
2017-10-25 21:57:02,173 [OC4J Launcher] ERROR app.ContextInitializer contextInitialized.420 - Integration Class not found: oracle.sysman.pp.sidb.ui.intg.SIDBProvisioningIntegration
2017-10-25 21:57:02,291 [OC4J Launcher] ERROR app.ContextInitializer contextInitialized.420 - Integration Class not found: oracle.sysman.pp.racprov.ui.intg.RACProvIntegration
2017-10-25 21:57:02,292 [OC4J Launcher] ERROR app.ContextInitializer contextInitialized.420 - Integration Class not found: oracle.sysman.pp.ec.ui.intg.ExtendClusterIntegration
2017-10-25 21:57:02,293 [OC4J Launcher] ERROR app.ContextInitializer contextInitialized.420 - Integration Class not found: oracle.sysman.pp.dn.ui.intg.DltNodeIntegration
2017-10-25 21:57:02,293 [OC4J Launcher] ERROR app.ContextInitializer contextInitialized.420 - Integration Class not found: oracle.sysman.pp.common.ui.intg.ProvCommonIntegration
2017-10-25 21:57:02,294 [OC4J Launcher] ERROR app.ContextInitializer contextInitialized.420 - Integration Class not found: oracle.sysman.pp.bpelprov.ui.intg.BPELProvisioningIntegratio
emagent.trc
-10-25 22:59:01,298 Thread-3633866496 ERROR pingManager: nmepm_pingReposURL: Cannot connect to http://RHENODE2:1158/em/upload/: retStatus=-1
2017-10-25 22:59:01,299 Thread-3633866496 WARN http: nmehl_connect_internal: connect failed to (RHENODE2:1158): Connection refused (error = 111)
2017-10-25 22:59:01,299 Thread-3633866496 ERROR pingManager: nmepm_pingReposURL: Cannot connect to http://RHENODE2:1158/em/upload/: retStatus=-1
2017-10-25 22:59:31,308 Thread-3633866496 WARN http: nmehl_connect_internal: connect failed to (RHENODE2:1158): Connection refused (error = 111)
2017-10-25 22:59:31,308 Thread-3633866496 ERROR pingManager: nmepm_pingReposURL: Cannot connect to http://RHENODE2:1158/em/upload/: retStatus=-1
2017-10-25 22:59:31,310 Thread-3633866496 WARN http: nmehl_connect_internal: connect failed to (RHENODE2:1158): Connection refused (error = 111)
2017-10-25 22:59:31,310 Thread-3633866496 ERROR pingManager: nmepm_pingReposURL: Cannot connect to http://RHENODE2:1158/em/upload/: retStatus=-1
2017-10-25 23:00:01,325 Thread-3633866496 WARN http: nmehl_connect_internal: connect failed to (RHENODE2:1158): Connection refused (error = 111)
2017-10-25 23:00:01,325 Thread-3633866496 ERROR pingManager: nmepm_pingReposURL: Cannot connect to http://RHENODE2:1158/em/upload/: retStatus=-1
2017-10-25 23:00:01,327 Thread-3633866496 WARN http: nmehl_connect_internal: connect failed to (RHENODE2:1158): Connection refused (error = 111)
2017-10-25 23:00:01,327 Thread-3633866496 ERROR pingManager: nmepm_pingReposURL: Cannot connect to http://RHENODE2:1158/em/upload/: retStatus=-1
emoms.trc
an.pp.racprov.ui.intg.RACProvIntegration
2017-10-25 21:57:02,292 [OC4J Launcher] ERROR app.ContextInitializer contextInitialized.420 - Integration Class not found: oracle.sysman.pp.ec.ui.intg.ExtendClusterIntegration
2017-10-25 21:57:02,293 [OC4J Launcher] ERROR app.ContextInitializer contextInitialized.420 - Integration Class not found: oracle.sysman.pp.dn.ui.intg.DltNodeIntegration
2017-10-25 21:57:02,293 [OC4J Launcher] ERROR app.ContextInitializer contextInitialized.420 - Integration Class not found: oracle.sysman.pp.common.ui.intg.ProvCommonIntegration
2017-10-25 21:57:02,294 [OC4J Launcher] ERROR app.ContextInitializer contextInitialized.420 - Integration Class not found: oracle.sysman.pp.bpelprov.ui.intg.BPELProvisioningIntegration
emagent_perl.trc
lertlogAdr.pl: Wed Oct 25 21:45:07 2017: WARN: First stack timestamp for /u01/app/oracle/diag/rdbms/test1/TEST1/alert/log.xml: 2017-10-09T09:18:15.382-07:00
alertlogAdr.pl: Wed Oct 25 21:45:10 2017: WARN: First stack timestamp for /u01/app/oracle/diag/rdbms/test1/TEST1/alert/log.xml: 2017-10-09T09:18:15.382-07:00
alertlogAdr.pl: Wed Oct 25 21:45:23 2017: WARN: First stack timestamp for /u01/app/oracle/diag/rdbms/test1/TEST1/alert/log.xml: 2017-10-09T09:18:15.382-07:00
has_metrics.pl: Wed Oct 25 21:45:23 2017: WARN: WARN:has::Common:: has::Common::hasGetScanInformation:This is a pre 11gR2 cluster, binary emcrsp is not found, no Scan information at /u01/app/oracle/product/11.2.0/dbhome_1/sysman/admin/scripts/has/HasCluster.pm line 5578.
alertlogAdr.pl: Wed Oct 25 21:45:26 2017: WARN: First stack timestamp for /u01/app/oracle/diag/rdbms/test1/TEST1/alert/log.xml: 2017-10-09T09:18:15.382-07:00
alertlogAdr.pl: Wed Oct 25 21:45:27 2017: WARN: First stack timestamp for /u01/app/oracle/diag/rdbms/test1/TEST1/alert/log.xml: 2017-10-09T09:18:15.382-07:00
alertlogAdr.pl: Wed Oct 25 21:45:27 2017: WARN: First stack timestamp for /u01/app/oracle/diag/rdbms/test1/TEST1/alert/log.xml: 2017-10-09T09:18:15.382-07:00
alertlogAdr.pl: Wed Oct 25 21:50:06 2017: WARN: First stack timestamp for /u01/app/oracle/diag/rdbms/test1/TEST1/alert/log.xml: 2017-10-09T09:18:15.382-07:00
pwdexpiry.pl: Wed Oct 25 21:51:28 2017: ERROR: pwdexpiry: number of args is 1
pwdexpiry.pl: Wed Oct 25 21:51:28 2017: ERROR: pwdexpiry: Invalid number of arguments passed
alertlogAdr.pl: Wed Oct 25 21:55:05 2017: WARN: First stack timestamp for /u01/app/oracle/diag/rdbms/test1/TEST1/alert/log.xml: 2017-10-09T09:18:15.382-07:00
alertlogAdr.pl: Wed Oct 25 22:00:07 2017: WARN: First stack timestamp for /u01/app/oracle/diag/rdbms/test1/TEST1/alert/log.xml: 2017-10-09T09:18:15.382-07:00
alertlogAdr.pl: Wed Oct 25 22:05:05 2017: WARN: First stack timestamp for /u01/app/oracle/diag/rdbms/test1/TEST1/alert/log.xml: 2017-10-09T09:18:15.382-07:00
alertlogAdr.pl: Wed Oct 25 22:10:05 2017: WARN: First stack timestamp for /u01/app/oracle/diag/rdbms/test1/TEST1/alert/log.xml: 2017-10-09T09:18:15.382-07:00
alertlogAdr.pl: Wed Oct 25 22:15:05 2017: WARN: First stack timestamp for /u01/app/oracle/diag/rdbms/test1/TEST1/alert/log.xml: 2017-10-09T09:18:15.382-07:00
alertlogAdr.pl: Wed Oct 25 22:20:05 2017: WARN: First stack timestamp for /u01/app/oracle/diag/rdbms/test1/TEST1/alert/log.xml: 2017-10-09T09:18:15.382-07:00
alertlogAdr.pl: Wed Oct 25 22:25:05 2017: WARN: First stack timestamp for /u01/app/oracle/diag/rdbms/test1/TEST1/alert/log.xml: 2017-10-09T09:18:15.382-07:00
alertlogAdr.pl: Wed Oct 25 22:30:05 2017: WARN: First stack timestamp for /u01/app/oracle/diag/rdbms/test1/TEST1/alert/log.xml: 2017-10-09T09:18:15.382-07:00
alertlogAdr.pl: Wed Oct 25 22:35:05 2017: WARN: First stack timestamp for /u01/app/oracle/diag/rdbms/test1/TEST1/alert/log.xml: 2017-10-09T09:18:15.382-07:00
alertlogAdr.pl: Wed Oct 25 22:40:05 2017: WARN: First stack timestamp for /u01/app/oracle/diag/rdbms/test1/TEST1/alert/log.xml: 2017-10-09T09:18:15.382-07:00
alertlogAdr.pl: Wed Oct 25 22:45:06 2017: WARN: First stack timestamp for /u01/app/oracle/diag/rdbms/test1/TEST1/alert/log.xml: 2017-10-09T09:18:15.382-07:00
alertlogAdr.pl: Wed Oct 25 22:50:05 2017: WARN: First stack timestamp for /u01/app/oracle/diag/rdbms/test1/TEST1/alert/log.xml: 2017-10-09T09:18:15.382-07:00
alertlogAdr.pl: Wed Oct 25 22:55:05 2017: WARN: First stack timestamp for /u01/app/oracle/diag/rdbms/test1/TEST1/alert/log.xml: 2017-10-09T09:18:15.382-07:00
alertlogAdr.pl: Wed Oct 25 23:00:05 2017: WARN: First stack timestamp for /u01/app/oracle/diag/rdbms/test1/TEST1/alert/log.xml: 2017-10-09T09:18:15.382-07:00