Skip to Main Content
Forums

Enterprise Manager

Announcement

For appeals, questions and feedback about Oracle Forums, please email oracle-forums-moderators_us@oracle.com. Technical questions should be asked in the appropriate category. Thank you!

Interested in getting your voice heard by members of the Developer Marketing team at Oracle? Check out this post for AppDev or this post for AI focus group information.

Troubleshooting Enterprise Manager emctl start dbconsole -- emc

User_7L9DTDec 9 2016 — edited Dec 11 2016

Hello everyone,
platform: AIX 6.1

Oracle Database 10g
Oracle RAC / 2 nodes

I have searched on the forums, some users had similar problems (posted here and here and here) but no solution provided. Any help would be appreciated.


Now I have two envrionments, Production and development and both of them are having problems with EMC, nothing has been changed, it was working well and suddenly it stopped
(have not changed the hostname or

let us start with the development environment

two nodes, now the first node: dbnode1

oracle@dbnode1 /home/oracle> . ./.demop
ORACLE_HOME=/u01/app/oracle/product/10.2.0/db_1
ORACLE_SID=DEMO1
LIBPATH=/u01/app/oracle/product/10.2.0/db_1/lib:/u01/app/oracle/product/10.2.0/db_1/lib32:
oracle@dbnode1 /home/oracle>  emctl start dbconsole
Oracle Enterprise Manager 10g Database Control Release 10.2.0.4.0
Copyright (c) 1996, 2007 Oracle Corporation.  All rights reserved.
https://dbnode1:1158/em/console/aboutApplication
ps: 0509-048 Flag -o was used with invalid list.
ps: Not a recognized flag: -
Usage: ps [-ANPaedfklmMZ] [-n namelist] [-F Format] [-o specifier[=header],...]
                [-p proclist][-G|-g grouplist] [-t termlist] [-U|-u userlist] [-c classlist] [ -T pid] [ -L pidlist]
Usage: ps [aceglnsuvwxU] [t tty] [processnumber]
Agent Version     : 10.1.0.6.0
OMS Version       : 10.1.0.6.0
Protocol Version  : 10.1.0.2.0
Agent Home        : /u01/app/oracle/product/10.2.0/db_1/dbnode1_DEMO1
Agent binaries    : /u01/app/oracle/product/10.2.0/db_1
Agent Process ID  : 471534
Parent Process ID : 679958
Agent URL         : https://dbnode1:3938/emd/main
Started at        : 2016-12-08 15:47:02
Started by user   : oracle
Last Reload       : 2016-12-08 15:47:02
Last successful upload                       : (none)
Last attempted upload                        : 2016-12-09 23:48:54
Total Megabytes of XML files uploaded so far :     0.00
Number of XML files pending upload           :     3671
Size of XML files pending upload(MB)         :    59.18
Available disk space on upload filesystem    :    38.31%
Agent is already started. Will restart the agent
Stopping agent ... stopped.
Starting Oracle Enterprise Manager 10g Database Control .... started.
------------------------------------------------------------------
Logs are generated in directory /u01/app/oracle/product/10.2.0/db_1/dbnode1_DEMO1/sysman/log

Second node dbnode2

oracle@dbnode2 /home/oracle> . ./.demop
ORACLE_HOME=/u01/app/oracle/product/10.2.0/db_1
ORACLE_SID=DEMO2
LIBPATH=/u01/app/oracle/product/10.2.0/db_1/lib:/u01/app/oracle/product/10.2.0/db_1/lib32:
oracle@dbnode2 /home/oracle> emctl start dbconsole
Oracle Enterprise Manager 10g Database Control Release 10.2.0.4.0
Copyright (c) 1996, 2007 Oracle Corporation.  All rights reserved.
https://dbnode1:1158/em/console/aboutApplication
Agent Version     : 10.1.0.6.0
OMS Version       : 10.1.0.6.0
Protocol Version  : 10.1.0.2.0
Agent Home        : /u01/app/oracle/product/10.2.0/db_1/dbnode2_DEMO2
Agent binaries    : /u01/app/oracle/product/10.2.0/db_1
Agent Process ID  : 524394
Parent Process ID : 659626
Agent URL         : https://dbnode2:3938/emd/main
Started at        : 2016-12-08 15:34:59
Started by user   : oracle
Last Reload       : 2016-12-08 15:34:59
Last successful upload                       : (none)
Last attempted upload                        : 2016-12-09 23:53:28
Total Megabytes of XML files uploaded so far :     0.00
Number of XML files pending upload           :     2793
Size of XML files pending upload(MB)         :    50.05
Available disk space on upload filesystem    :    41.76%
Agent is already started. Will restart the agent
This will stop the Oracle Enterprise Manager 10g Database Control process. Continue [y/n] :y
Stopping Oracle Enterprise Manager 10g Database Control ...
 ...  Stopped.
Agent is not running.
Starting Oracle Enterprise Manager 10g Database Control .... started.
------------------------------------------------------------------
Logs are generated in directory /u01/app/oracle/product/10.2.0/db_1/dbnode2_DEMO2/sysman/log

when i browse to the URL from the server itself

https://dbnode1:3938/emd/main

https://dbnode2:3938/emd/main

EMAgent10.1.0.6.0
Congratulations, EMAgent is working!

oracle@dbnode1 /home/oracle>  emctl status dbconsole
Oracle Enterprise Manager 10g Database Control Release 10.2.0.4.0
Copyright (c) 1996, 2007 Oracle Corporation.  All rights reserved.
https://dbnode1:1158/em/console/aboutApplication
Oracle Enterprise Manager 10g is not running.
------------------------------------------------------------------
Logs are generated in directory /u01/app/oracle/product/10.2.0/db_1/dbnode1_DEMO1/sysman/log
on the other node
oracle@dbnode2 /home/oracle> emctl status dbconsole
Oracle Enterprise Manager 10g Database Control Release 10.2.0.4.0
Copyright (c) 1996, 2007 Oracle Corporation.  All rights reserved.
https://dbnode1:1158/em/console/aboutApplication
EM Daemon is running.
------------------------------------------------------------------
Logs are generated in directory /u01/app/oracle/product/10.2.0/db_1/dbnode2_DEMO2/sysman/log
oracle@dbnode2 /home/oracle>
on let's look at the logs on DBNODE2
oracle@dbnode2 /u01/app/oracle/product/10.2.0/db_1/dbnode2_DEMO2/sysman/log> ls -ltra
total 36216
-rw-r-----   1 oracle   dba               0 Sep 21 2015  emdctl.log
drwxr-S---   7 oracle   dba             512 Sep 21 2015  ..
-rw-r-----   1 oracle   dba               0 Sep 21 2015  emagentfetchlet.trc
-rw-r-----   1 oracle   dba               0 Sep 21 2015  emagentfetchlet.log
drwxrwx---   2 oracle   dba             512 Sep 21 2015  nmcDEMO21521
-rw-r-----   1 oracle   dba         4194412 Feb 05 2016  emagent.trc.4
-rw-r-----   1 oracle   dba         4194532 Feb 06 2016  emagent.trc.3
-rw-r-----   1 oracle   dba         4194419 Feb 07 2016  emagent.trc.2
-rw-r-----   1 oracle   dba         4194318 Dec 09 17:04 emagent.trc.1
drwxr-S---   3 oracle   dba             512 Dec 09 17:04 .
-rw-r-----   1 oracle   dba             632 Dec 09 23:54 emdctl.trc
-rw-r-----   1 oracle   dba            1548 Dec 09 23:54 emdb.nohup.lr
-rw-r-----   1 oracle   dba             561 Dec 09 23:54 emagent_perl.trc
-rw-r-----   1 oracle   dba             745 Dec 09 23:54 emagent.log
-rw-r-----   1 oracle   dba            1862 Dec 09 23:54 emdb.nohup
-rw-r-----   1 oracle   dba         1206531 Dec 10 01:22 emagent.trc
oracle@dbnode2 /u01/app/oracle/product/10.2.0/db_1/dbnode2_DEMO2/sysman/log> tail -n60 emagent.trc
2016-12-10 01:22:14 Thread-1930 ERROR upload: FxferSend: Cannot connect to: https://dbnode1:1158/em/upload/. retStatus=-32
2016-12-10 01:22:14 Thread-1930 ERROR upload: Failed to upload file A0000001.xml, ret = -2
2016-12-10 01:22:14 Thread-1930 WARN  http: snmehl_connect: connect failed to (dbnode1:1158): A remote host refused an attempted connect operation. (error = 79)
2016-12-10 01:22:14 Thread-1930 ERROR upload: FxferSend: Cannot connect to: https://dbnode1:1158/em/upload/. retStatus=-32
2016-12-10 01:22:14 Thread-1930 ERROR upload: Failed to upload file A0000001.xml, ret = -2
2016-12-10 01:22:14 Thread-1930 ERROR upload: 3 Failures in a row for A0000001.xml, we give up
2016-12-10 01:22:14 Thread-1930 ERROR upload: Error in uploadXMLFiles.  Trying again in 300.00 seconds.
2016-12-10 01:22:17 Thread-1931 WARN  http: snmehl_connect: connect failed to (dbnode1:1158): A remote host refused an attempted connect operation. (error = 79)
2016-12-10 01:22:17 Thread-1931 ERROR upload: FxferSend: Cannot connect to: https://dbnode1:1158/em/upload/. retStatus=-32
2016-12-10 01:22:17 Thread-1931 ERROR upload: Failed to upload file A0000001.xml, ret = -2
2016-12-10 01:22:17 Thread-1931 WARN  http: snmehl_connect: connect failed to (dbnode1:1158): A remote host refused an attempted connect operation. (error = 79)
2016-12-10 01:22:17 Thread-1931 ERROR upload: FxferSend: Cannot connect to: https://dbnode1:1158/em/upload/. retStatus=-32
2016-12-10 01:22:17 Thread-1931 ERROR upload: Failed to upload file A0000001.xml, ret = -2
2016-12-10 01:22:17 Thread-1931 WARN  http: snmehl_connect: connect failed to (dbnode1:1158): A remote host refused an attempted connect operation. (error = 79)
2016-12-10 01:22:17 Thread-1931 ERROR upload: FxferSend: Cannot connect to: https://dbnode1:1158/em/upload/. retStatus=-32
2016-12-10 01:22:17 Thread-1931 ERROR upload: Failed to upload file A0000001.xml, ret = -2
2016-12-10 01:22:17 Thread-1931 ERROR upload: 3 Failures in a row for A0000001.xml, we give up
2016-12-10 01:22:17 Thread-1931 ERROR upload: Error in uploadXMLFiles.  Trying again in 300.00 seconds.
2016-12-10 01:22:19 Thread-1932 WARN  http: snmehl_connect: connect failed to (dbnode1:1158): A remote host refused an attempted connect operation. (error = 79)
2016-12-10 01:22:19 Thread-1932 ERROR pingManager: nmepm_pingReposURL: Cannot connect to https://dbnode1:1158/em/upload/: retStatus=-32
2016-12-10 01:22:19 Thread-1932 WARN  http: snmehl_connect: connect failed to (dbnode1:1158): A remote host refused an attempted connect operation. (error = 79)
2016-12-10 01:22:19 Thread-1932 ERROR pingManager: nmepm_pingReposURL: Cannot connect to https://dbnode1:1158/em/upload/: retStatus=-32
2016-12-10 01:22:49 Thread-1933 WARN  http: snmehl_connect: connect failed to (dbnode1:1158): A remote host refused an attempted connect operation. (error = 79)
2016-12-10 01:22:49 Thread-1933 ERROR pingManager: nmepm_pingReposURL: Cannot connect to https://dbnode1:1158/em/upload/: retStatus=-32
2016-12-10 01:22:49 Thread-1933 WARN  http: snmehl_connect: connect failed to (dbnode1:1158): A remote host refused an attempted connect operation. (error = 79)
2016-12-10 01:22:49 Thread-1933 ERROR pingManager: nmepm_pingReposURL: Cannot connect to https://dbnode1:1158/em/upload/: retStatus=-32
2016-12-10 01:23:17 Thread-1935 WARN  http: snmehl_connect: connect failed to (dbnode1:1158): A remote host refused an attempted connect operation. (error = 79)
2016-12-10 01:23:17 Thread-1935 ERROR upload: FxferSend: Cannot connect to: https://dbnode1:1158/em/upload/. retStatus=-32
2016-12-10 01:23:17 Thread-1935 ERROR upload: Failed to upload file A0000001.xml, ret = -2
2016-12-10 01:23:17 Thread-1935 WARN  http: snmehl_connect: connect failed to (dbnode1:1158): A remote host refused an attempted connect operation. (error = 79)
2016-12-10 01:23:17 Thread-1935 ERROR upload: FxferSend: Cannot connect to: https://dbnode1:1158/em/upload/. retStatus=-32
2016-12-10 01:23:17 Thread-1935 ERROR upload: Failed to upload file A0000001.xml, ret = -2
2016-12-10 01:23:17 Thread-1935 WARN  http: snmehl_connect: connect failed to (dbnode1:1158): A remote host refused an attempted connect operation. (error = 79)
2016-12-10 01:23:17 Thread-1935 ERROR upload: FxferSend: Cannot connect to: https://dbnode1:1158/em/upload/. retStatus=-32
2016-12-10 01:23:17 Thread-1935 ERROR upload: Failed to upload file A0000001.xml, ret = -2
2016-12-10 01:23:17 Thread-1935 ERROR upload: 3 Failures in a row for A0000001.xml, we give up
2016-12-10 01:23:17 Thread-1935 ERROR upload: Error in uploadXMLFiles.  Trying again in 300.00 seconds.
2016-12-10 01:23:19 Thread-1936 WARN  http: snmehl_connect: connect failed to (dbnode1:1158): A remote host refused an attempted connect operation. (error = 79)
2016-12-10 01:23:19 Thread-1936 ERROR pingManager: nmepm_pingReposURL: Cannot connect to https://dbnode1:1158/em/upload/: retStatus=-32
2016-12-10 01:23:19 Thread-1936 WARN  http: snmehl_connect: connect failed to (dbnode1:1158): A remote host refused an attempted connect operation. (error = 79)
2016-12-10 01:23:19 Thread-1936 ERROR pingManager: nmepm_pingReposURL: Cannot connect to https://dbnode1:1158/em/upload/: retStatus=-32
2016-12-10 01:23:50 Thread-1937 ERROR pingManager: nmepm_pingReposURL: Error in request response. code = 400. text =
2016-12-10 01:23:50 Thread-1029 ERROR command: nmejcn: error receiving response headers from https://dbnode1:1158/em/upload/
2016-12-10 01:23:50 Thread-1937 WARN  http: snmehl_connect: connect failed to (dbnode1:1158): A remote host refused an attempted connect operation. (error = 79)
2016-12-10 01:23:50 Thread-1937 ERROR pingManager: nmepm_pingReposURL: Cannot connect to https://dbnode1:1158/em/upload/: retStatus=-32
2016-12-10 01:24:17 Thread-1939 WARN  http: snmehl_connect: connect failed to (dbnode1:1158): A remote host refused an attempted connect operation. (error = 79)
2016-12-10 01:24:17 Thread-1939 ERROR upload: FxferSend: Cannot connect to: https://dbnode1:1158/em/upload/. retStatus=-32
2016-12-10 01:24:17 Thread-1939 ERROR upload: Failed to upload file A0000001.xml, ret = -2
2016-12-10 01:24:17 Thread-1939 WARN  http: snmehl_connect: connect failed to (dbnode1:1158): A remote host refused an attempted connect operation. (error = 79)
2016-12-10 01:24:17 Thread-1939 ERROR upload: FxferSend: Cannot connect to: https://dbnode1:1158/em/upload/. retStatus=-32
2016-12-10 01:24:17 Thread-1939 ERROR upload: Failed to upload file A0000001.xml, ret = -2
2016-12-10 01:24:17 Thread-1939 WARN  http: snmehl_connect: connect failed to (dbnode1:1158): A remote host refused an attempted connect operation. (error = 79)
2016-12-10 01:24:17 Thread-1939 ERROR upload: FxferSend: Cannot connect to: https://dbnode1:1158/em/upload/. retStatus=-32
2016-12-10 01:24:17 Thread-1939 ERROR upload: Failed to upload file A0000001.xml, ret = -2
2016-12-10 01:24:17 Thread-1939 ERROR upload: 3 Failures in a row for A0000001.xml, we give up
2016-12-10 01:24:17 Thread-1939 ERROR upload: Error in uploadXMLFiles.  Trying again in 300.00 seconds.
2016-12-10 01:24:20 Thread-1940 WARN  http: snmehl_connect: connect failed to (dbnode1:1158): A remote host refused an attempted connect operation. (error = 79)
2016-12-10 01:24:20 Thread-1940 ERROR pingManager: nmepm_pingReposURL: Cannot connect to https://dbnode1:1158/em/upload/: retStatus=-32
2016-12-10 01:24:20 Thread-1940 WARN  http: snmehl_connect: connect failed to (dbnode1:1158): A remote host refused an attempted connect operation. (error = 79)
2016-12-10 01:24:20 Thread-1940 ERROR pingManager: nmepm_pingReposURL: Cannot connect to https://dbnode1:1158/em/upload/: retStatus=-32
oracle@dbnode2 /u01/app/oracle/product/10.2.0/db_1/dbnode2_DEMO2/sysman/log>
oracle@dbnode2 /u01/app/oracle/product/10.2.0/db_1/dbnode2_DEMO2/sysman/log> tail -n50 emdb.nohup
----- Mon Sep 21 14:41:02 2015::DBConsole. Monitoring Agent+Console -----
----- Mon Sep 21 14:41:02 2015::Remote OMS configured, starting agent only. -----
----- Mon Sep 21 14:41:02 2015::Property 'agentTZRegion' is  missing from  /u01/app/oracle/product/10.2.0/db_1/dbnode2_DEMO2/sysman/config/emd.properties. This is normal when the agent is started for the very first time.Updating it... -----
----- Mon Sep 21 14:41:02 2015::/u01/app/oracle/product/10.2.0/db_1/dbnode2_DEMO2/sysman/config/emd.properties copied to /u01/app/oracle/product/10.2.0/db_1/dbnode2_DEMO2/sysman/config/emd.properties.2015-09-21-14-41-02 while updating the property  'agentTZRegion' -----
----- Mon Sep 21 14:41:02 2015::An agentTZregion of 'Asia' is installed in /u01/app/oracle/product/10.2.0/db_1/dbnode2_DEMO2/sysman/config/emd.properties. -----
----- Mon Sep 21 14:41:02 2015::Agent Launched with PID 970786 at time Mon Sep 21 14:41:02 2015 -----
(pid=970786): starting emagent version 10.1.0.6.0
(pid=970786): Exiting normally due to signal SIGTERM
----- Thu Dec  8 15:34:58 2016::DBConsole. Monitoring Agent+Console -----
----- Thu Dec  8 15:34:58 2016::Remote OMS configured, starting agent only. -----
----- Thu Dec  8 15:34:59 2016::Agent Launched with PID 524394 at time Thu Dec  8 15:34:59 2016 -----
(pid=524394): starting emagent version 10.1.0.6.0
(pid=524394): emagent now exiting normally
----- Fri Dec  9 23:54:13 2016::DBConsole. Monitoring Agent+Console -----
----- Fri Dec  9 23:54:13 2016::Remote OMS configured, starting agent only. -----
----- Fri Dec  9 23:54:14 2016::Agent Launched with PID 627192 at time Fri Dec  9 23:54:14 2016 -----
(pid=627192): starting emagent version 10.1.0.6.0
----- Fri Dec  9 23:54:41 2016::EMAgent exited at Fri Dec  9 23:54:41 2016 with return value 0. -----
----- Fri Dec  9 23:54:41 2016::Exiting watchdog loop -----
oracle@dbnode2 /u01/app/oracle/product/10.2.0/db_1/dbnode2_DEMO2/sysman/log> tail -n40 emagent.log
2015-09-21 14:41:02 Thread-1 Starting Agent 10.1.0.6.0 from /u01/app/oracle/product/10.2.0/db_1 (00701)
2015-09-21 14:41:04 Thread-1 EMAgent started successfully (00702)
2015-12-23 15:43:52 Thread-1302 Disable collector (00406)
2016-12-08 15:34:59 Thread-1 Starting Agent 10.1.0.6.0 from /u01/app/oracle/product/10.2.0/db_1 (00701)
2016-12-08 15:35:03 Thread-1 EMAgent started successfully (00702)
2016-12-08 15:35:03 Thread-1808 Disable collector (00406)
2016-12-09 23:54:12 Thread-1 EMAgent normal shutdown (00703)
2016-12-09 23:54:14 Thread-1 Starting Agent 10.1.0.6.0 from /u01/app/oracle/product/10.2.0/db_1 (00701)
2016-12-09 23:54:16 Thread-1 EMAgent started successfully (00702)
2016-12-09 23:54:16 Thread-1807 Disable collector (00406)
oracle@dbnode2 /u01/app/oracle/product/10.2.0/db_1/dbnode2_DEMO2/sysman/log> tail -n40 emdctl.trc
2015-09-21 14:41:01 Thread-1 WARN  http: snmehl_connect: connect failed to (dbnode2:3938): A remote host refused an attempted connect operation. (error = 79)
2016-12-08 15:34:57 Thread-1 WARN  http: snmehl_connect: connect failed to (dbnode2:3938): A remote host refused an attempted connect operation. (error = 79)
2016-12-09 23:54:12 Thread-1 WARN  http: snmehl_connect: connect failed to (dbnode2:3938): A remote host refused an attempted connect operation. (error = 79)
2016-12-09 23:54:13 Thread-1 WARN  http: snmehl_connect: connect failed to (dbnode2:3938): A remote host refused an attempted connect operation. (error = 79)
oracle@dbnode2 /u01/app/oracle/product/10.2.0/db_1/dbnode2_DEMO2/sysman/log>
Going to DBNODE1
oracle@dbnode1 /u01/app/oracle/product/10.2.0/db_1/dbnode1_DEMO1/sysman/log> tail -n40 emagent.log
2015-09-21 14:30:55 Thread-1 Starting Agent 10.1.0.6.0 from /u01/app/oracle/product/10.2.0/db_1 (00701)
2015-09-21 14:30:59 Thread-1 EMAgent started successfully (00702)
2015-12-23 09:07:31 Thread-1928 Disable collector (00406)
2016-11-01 14:24:36 Thread-1 Starting Agent 10.1.0.6.0 from /u01/app/oracle/product/10.2.0/db_1 (00701)
2016-11-01 14:24:45 Thread-1 EMAgent started successfully (00702)
2016-11-01 14:24:46 Thread-1550 Disable collector (00406)
2016-11-01 14:26:46 Thread-1556 enable collector (00407)
2016-11-01 14:26:46 Thread-1815 Disable collector (00406)
2016-11-01 14:28:45 Thread-1569 enable collector (00407)
2016-11-01 14:28:45 Thread-2341 Disable collector (00406)
2016-11-01 14:30:45 Thread-1582 enable collector (00407)
2016-11-01 14:30:46 Thread-1841 Disable collector (00406)
2016-12-08 15:47:02 Thread-1 Starting Agent 10.1.0.6.0 from /u01/app/oracle/product/10.2.0/db_1 (00701)
2016-12-08 15:47:06 Thread-1 EMAgent started successfully (00702)
2016-12-08 15:47:06 Thread-1807 Disable collector (00406)
2016-12-09 23:49:39 Thread-1 EMAgent normal shutdown (00703)
2016-12-09 23:51:16 Thread-1 Starting Agent 10.1.0.6.0 from /u01/app/oracle/product/10.2.0/db_1 (00701)
2016-12-09 23:51:18 Thread-1 EMAgent started successfully (00702)
2016-12-09 23:51:18 Thread-1807 Disable collector (00406)
oracle@dbnode1 /u01/app/oracle/product/10.2.0/db_1/dbnode1_DEMO1/sysman/log> tail -n50 emdb.nohup
16/12/10 01:10:09 Oracle Application Server Containers for J2EE 10g (9.0.4.1.0) initialized
----- Sat Dec 10 01:12:32 2016::DBConsole exited at Sat Dec 10 01:12:32 2016 with return value 3. -----
----- Sat Dec 10 01:12:32 2016::DBConsole has requested a restart. -----
----- Sat Dec 10 01:12:32 2016::Restarting DBConsole. -----
----- Sat Dec 10 01:12:32 2016::Console Launched with PID 643174 at time Sat Dec 10 01:12:32 2016 -----
16/12/10 01:12:39 securePort = 1820
16/12/10 01:12:39 isOMSSecure = 1 isOMSSecureLocked = 1
16/12/10 01:12:41 Oracle Application Server Containers for J2EE 10g (9.0.4.1.0) initialized
----- Sat Dec 10 01:15:03 2016::DBConsole exited at Sat Dec 10 01:15:03 2016 with return value 3. -----
----- Sat Dec 10 01:15:03 2016::DBConsole has requested a restart. -----
----- Sat Dec 10 01:15:03 2016::Restarting DBConsole. -----
----- Sat Dec 10 01:15:03 2016::Console Launched with PID 651358 at time Sat Dec 10 01:15:03 2016 -----
16/12/10 01:15:11 securePort = 1820
16/12/10 01:15:11 isOMSSecure = 1 isOMSSecureLocked = 1
16/12/10 01:15:12 Oracle Application Server Containers for J2EE 10g (9.0.4.1.0) initialized
----- Sat Dec 10 01:17:35 2016::DBConsole exited at Sat Dec 10 01:17:35 2016 with return value 3. -----
----- Sat Dec 10 01:17:35 2016::DBConsole has requested a restart. -----
----- Sat Dec 10 01:17:35 2016::Restarting DBConsole. -----
----- Sat Dec 10 01:17:35 2016::Console Launched with PID 672088 at time Sat Dec 10 01:17:35 2016 -----
16/12/10 01:17:42 securePort = 1820
16/12/10 01:17:42 isOMSSecure = 1 isOMSSecureLocked = 1
16/12/10 01:17:44 Oracle Application Server Containers for J2EE 10g (9.0.4.1.0) initialized
----- Sat Dec 10 01:20:06 2016::DBConsole exited at Sat Dec 10 01:20:06 2016 with return value 3. -----
----- Sat Dec 10 01:20:06 2016::DBConsole has requested a restart. -----
----- Sat Dec 10 01:20:06 2016::Restarting DBConsole. -----
----- Sat Dec 10 01:20:06 2016::Console Launched with PID 672090 at time Sat Dec 10 01:20:06 2016 -----
16/12/10 01:20:14 securePort = 1820
16/12/10 01:20:14 isOMSSecure = 1 isOMSSecureLocked = 1
16/12/10 01:20:15 Oracle Application Server Containers for J2EE 10g (9.0.4.1.0) initialized
----- Sat Dec 10 01:22:38 2016::DBConsole exited at Sat Dec 10 01:22:38 2016 with return value 3. -----
----- Sat Dec 10 01:22:38 2016::DBConsole has requested a restart. -----
----- Sat Dec 10 01:22:38 2016::Restarting DBConsole. -----
----- Sat Dec 10 01:22:38 2016::Console Launched with PID 672094 at time Sat Dec 10 01:22:38 2016 -----
16/12/10 01:22:45 securePort = 1820
16/12/10 01:22:45 isOMSSecure = 1 isOMSSecureLocked = 1
16/12/10 01:22:47 Oracle Application Server Containers for J2EE 10g (9.0.4.1.0) initialized
----- Sat Dec 10 01:25:10 2016::DBConsole exited at Sat Dec 10 01:25:10 2016 with return value 3. -----
----- Sat Dec 10 01:25:10 2016::DBConsole has requested a restart. -----
----- Sat Dec 10 01:25:10 2016::Restarting DBConsole. -----
----- Sat Dec 10 01:25:10 2016::Console Launched with PID 667650 at time Sat Dec 10 01:25:10 2016 -----
16/12/10 01:25:17 securePort = 1820
16/12/10 01:25:17 isOMSSecure = 1 isOMSSecureLocked = 1
16/12/10 01:25:19 Oracle Application Server Containers for J2EE 10g (9.0.4.1.0) initialized
----- Sat Dec 10 01:27:41 2016::DBConsole exited at Sat Dec 10 01:27:41 2016 with return value 3. -----
----- Sat Dec 10 01:27:41 2016::DBConsole has requested a restart. -----
----- Sat Dec 10 01:27:41 2016::Restarting DBConsole. -----
----- Sat Dec 10 01:27:41 2016::Console Launched with PID 672182 at time Sat Dec 10 01:27:41 2016 -----
16/12/10 01:27:48 securePort = 1820
16/12/10 01:27:48 isOMSSecure = 1 isOMSSecureLocked = 1
16/12/10 01:27:50 Oracle Application Server Containers for J2EE 10g (9.0.4.1.0) initialized

oracle@dbnode1 /u01/app/oracle/product/10.2.0/db_1/dbnode1_DEMO1/sysman/log>  tail -n60 emagent.trc
2016-12-10 01:25:49 Thread-1909 WARN  http: snmehl_connect: connect failed to (dbnode1:1158): A remote host refused an attempted connect operation. (error = 79)
2016-12-10 01:25:49 Thread-1909 ERROR pingManager: nmepm_pingReposURL: Cannot connect to https://dbnode1:1158/em/upload/: retStatus=-32
2016-12-10 01:25:49 Thread-1909 WARN  http: snmehl_connect: connect failed to (dbnode1:1158): A remote host refused an attempted connect operation. (error = 79)
2016-12-10 01:25:49 Thread-1909 ERROR pingManager: nmepm_pingReposURL: Cannot connect to https://dbnode1:1158/em/upload/: retStatus=-32
2016-12-10 01:26:19 Thread-1910 WARN  http: snmehl_connect: connect failed to (dbnode1:1158): A remote host refused an attempted connect operation. (error = 79)
2016-12-10 01:26:19 Thread-1910 ERROR pingManager: nmepm_pingReposURL: Cannot connect to https://dbnode1:1158/em/upload/: retStatus=-32
2016-12-10 01:26:19 Thread-1910 WARN  http: snmehl_connect: connect failed to (dbnode1:1158): A remote host refused an attempted connect operation. (error = 79)
2016-12-10 01:26:19 Thread-1910 ERROR pingManager: nmepm_pingReposURL: Cannot connect to https://dbnode1:1158/em/upload/: retStatus=-32
2016-12-10 01:26:19 Thread-2167 WARN  http: snmehl_connect: connect failed to (dbnode1:1158): A remote host refused an attempted connect operation. (error = 79)
2016-12-10 01:26:19 Thread-2167 ERROR upload: FxferSend: Cannot connect to: https://dbnode1:1158/em/upload/. retStatus=-32
2016-12-10 01:26:19 Thread-2167 ERROR upload: Failed to upload file A0000001.xml, ret = -2
2016-12-10 01:26:19 Thread-2167 WARN  http: snmehl_connect: connect failed to (dbnode1:1158): A remote host refused an attempted connect operation. (error = 79)
2016-12-10 01:26:19 Thread-2167 ERROR upload: FxferSend: Cannot connect to: https://dbnode1:1158/em/upload/. retStatus=-32
2016-12-10 01:26:19 Thread-2167 ERROR upload: Failed to upload file A0000001.xml, ret = -2
2016-12-10 01:26:19 Thread-2167 WARN  http: snmehl_connect: connect failed to (dbnode1:1158): A remote host refused an attempted connect operation. (error = 79)
2016-12-10 01:26:19 Thread-2167 ERROR upload: FxferSend: Cannot connect to: https://dbnode1:1158/em/upload/. retStatus=-32
2016-12-10 01:26:19 Thread-2167 ERROR upload: Failed to upload file A0000001.xml, ret = -2
2016-12-10 01:26:19 Thread-2167 ERROR upload: 3 Failures in a row for A0000001.xml, we give up
2016-12-10 01:26:19 Thread-2167 ERROR upload: Error in uploadXMLFiles.  Trying again in 300.00 seconds.
2016-12-10 01:26:49 Thread-1912 WARN  http: snmehl_connect: connect failed to (dbnode1:1158): A remote host refused an attempted connect operation. (error = 79)
2016-12-10 01:26:49 Thread-1912 ERROR pingManager: nmepm_pingReposURL: Cannot connect to https://dbnode1:1158/em/upload/: retStatus=-32
2016-12-10 01:26:49 Thread-1912 WARN  http: snmehl_connect: connect failed to (dbnode1:1158): A remote host refused an attempted connect operation. (error = 79)
2016-12-10 01:26:49 Thread-1912 ERROR pingManager: nmepm_pingReposURL: Cannot connect to https://dbnode1:1158/em/upload/: retStatus=-32
2016-12-10 01:27:19 Thread-2171 WARN  http: snmehl_connect: connect failed to (dbnode1:1158): A remote host refused an attempted connect operation. (error = 79)
2016-12-10 01:27:19 Thread-2171 ERROR pingManager: nmepm_pingReposURL: Cannot connect to https://dbnode1:1158/em/upload/: retStatus=-32
2016-12-10 01:27:19 Thread-2171 WARN  http: snmehl_connect: connect failed to (dbnode1:1158): A remote host refused an attempted connect operation. (error = 79)
2016-12-10 01:27:19 Thread-2171 ERROR pingManager: nmepm_pingReposURL: Cannot connect to https://dbnode1:1158/em/upload/: retStatus=-32
2016-12-10 01:27:19 Thread-1914 WARN  http: snmehl_connect: connect failed to (dbnode1:1158): A remote host refused an attempted connect operation. (error = 79)
2016-12-10 01:27:19 Thread-1914 ERROR upload: FxferSend: Cannot connect to: https://dbnode1:1158/em/upload/. retStatus=-32
2016-12-10 01:27:19 Thread-1914 ERROR upload: Failed to upload file A0000001.xml, ret = -2
2016-12-10 01:27:19 Thread-1914 WARN  http: snmehl_connect: connect failed to (dbnode1:1158): A remote host refused an attempted connect operation. (error = 79)
2016-12-10 01:27:19 Thread-1914 ERROR upload: FxferSend: Cannot connect to: https://dbnode1:1158/em/upload/. retStatus=-32
2016-12-10 01:27:19 Thread-1914 ERROR upload: Failed to upload file A0000001.xml, ret = -2
2016-12-10 01:27:19 Thread-1914 WARN  http: snmehl_connect: connect failed to (dbnode1:1158): A remote host refused an attempted connect operation. (error = 79)
2016-12-10 01:27:19 Thread-1914 ERROR upload: FxferSend: Cannot connect to: https://dbnode1:1158/em/upload/. retStatus=-32
2016-12-10 01:27:19 Thread-1914 ERROR upload: Failed to upload file A0000001.xml, ret = -2
2016-12-10 01:27:19 Thread-1914 ERROR upload: 3 Failures in a row for A0000001.xml, we give up
2016-12-10 01:27:19 Thread-1914 ERROR upload: Error in uploadXMLFiles.  Trying again in 300.00 seconds.
2016-12-10 01:27:51 Thread-1916 ERROR pingManager: nmepm_pingReposURL: Error in request response. code = 400. text =
2016-12-10 01:27:51 Thread-1916 WARN  http: snmehl_connect: connect failed to (dbnode1:1158): A remote host refused an attempted connect operation. (error = 79)
2016-12-10 01:27:51 Thread-1916 ERROR pingManager: nmepm_pingReposURL: Cannot connect to https://dbnode1:1158/em/upload/: retStatus=-32
2016-12-10 01:28:19 Thread-1917 WARN  http: snmehl_connect: connect failed to (dbnode1:1158): A remote host refused an attempted connect operation. (error = 79)
2016-12-10 01:28:19 Thread-1917 ERROR upload: FxferSend: Cannot connect to: https://dbnode1:1158/em/upload/. retStatus=-32
2016-12-10 01:28:19 Thread-1917 ERROR upload: Failed to upload file A0000001.xml, ret = -2
2016-12-10 01:28:19 Thread-1917 WARN  http: snmehl_connect: connect failed to (dbnode1:1158): A remote host refused an attempted connect operation. (error = 79)
2016-12-10 01:28:19 Thread-1917 ERROR upload: FxferSend: Cannot connect to: https://dbnode1:1158/em/upload/. retStatus=-32
2016-12-10 01:28:19 Thread-1917 ERROR upload: Failed to upload file A0000001.xml, ret = -2
2016-12-10 01:28:19 Thread-1917 WARN  http: snmehl_connect: connect failed to (dbnode1:1158): A remote host refused an attempted connect operation. (error = 79)
2016-12-10 01:28:19 Thread-1917 ERROR upload: FxferSend: Cannot connect to: https://dbnode1:1158/em/upload/. retStatus=-32
2016-12-10 01:28:19 Thread-1917 ERROR upload: Failed to upload file A0000001.xml, ret = -2
2016-12-10 01:28:19 Thread-1917 ERROR upload: 3 Failures in a row for A0000001.xml, we give up
2016-12-10 01:28:19 Thread-1917 ERROR upload: Error in uploadXMLFiles.  Trying again in 300.00 seconds.
2016-12-10 01:28:21 Thread-1918 WARN  http: snmehl_connect: connect failed to (dbnode1:1158): A remote host refused an attempted connect operation. (error = 79)
2016-12-10 01:28:21 Thread-1918 ERROR pingManager: nmepm_pingReposURL: Cannot connect to https://dbnode1:1158/em/upload/: retStatus=-32
2016-12-10 01:28:21 Thread-1918 WARN  http: snmehl_connect: connect failed to (dbnode1:1158): A remote host refused an attempted connect operation. (error = 79)
2016-12-10 01:28:21 Thread-1918 ERROR pingManager: nmepm_pingReposURL: Cannot connect to https://dbnode1:1158/em/upload/: retStatus=-32
2016-12-10 01:28:51 Thread-1919 WARN  http: snmehl_connect: connect failed to (dbnode1:1158): A remote host refused an attempted connect operation. (error = 79)
2016-12-10 01:28:51 Thread-1919 ERROR pingManager: nmepm_pingReposURL: Cannot connect to https://dbnode1:1158/em/upload/: retStatus=-32
2016-12-10 01:28:51 Thread-1919 WARN  http: snmehl_connect: connect failed to (dbnode1:1158): A remote host refused an attempted connect operation. (error = 79)
2016-12-10 01:28:51 Thread-1919 ERROR pingManager: nmepm_pingReposURL: Cannot connect to https://dbnode1:1158/em/upload/: retStatus=-32
oracle@dbnode1 /u01/app/oracle/product/10.2.0/db_1/dbnode1_DEMO1/sysman/log>
oracle@dbnode1 /u01/app/oracle/product/10.2.0/db_1/dbnode1_DEMO1/sysman/log> ls -ltra
total 245592
-rw-r-----   1 oracle   dba               0 Sep 21 2015  emdctl.log
drwxr-S---   8 oracle   dba             512 Sep 21 2015  ..
-rw-r-----   1 oracle   dba            8536 Sep 21 2015  secure.log
-rw-r-----   1 oracle   dba               0 Sep 21 2015  emagentfetchlet.trc
-rw-r-----   1 oracle   dba               0 Sep 21 2015  emagentfetchlet.log
drwxrwx---   2 oracle   dba             512 Sep 21 2015  nmcDEMO11521
-rw-r-----   1 oracle   dba         4194635 Feb 05 2016  emagent.trc.4
-rw-r-----   1 oracle   dba         5000366 Feb 05 2016  emoms.trc.10
-rw-r-----   1 oracle   dba         5000203 Feb 06 2016  emoms.trc.9
-rw-r-----   1 oracle   dba         5000154 Feb 06 2016  emoms.trc.8
-rw-r-----   1 oracle   dba         5000172 Feb 06 2016  emoms.trc.7
-rw-r-----   1 oracle   dba         5000434 Feb 06 2016  emoms.trc.6
-rw-r-----   1 oracle   dba         5000059 Feb 06 2016  emoms.trc.5
-rw-r-----   1 oracle   dba         4194393 Feb 06 2016  emagent.trc.3
-rw-r-----   1 oracle   dba         5000408 Feb 07 2016  emoms.trc.4
-rw-r-----   1 oracle   dba        20000437 Feb 07 2016  emoms.log.1
-rw-r-----   1 oracle   dba         5000183 Feb 07 2016  emoms.trc.3
-rw-r-----   1 oracle   dba         5002176 Feb 07 2016  emoms.trc.2
-rw-r-----   1 oracle   dba         4194462 Nov 01 15:58 emdctl.trc.1
-rw-r-----   1 oracle   dba         4194387 Nov 02 10:35 emagent.trc.2
-rw-r-----   1 oracle   dba         5000006 Dec 09 12:30 emoms.trc.1
-rw-r-----   1 oracle   dba         4194356 Dec 09 15:55 emagent.trc.1
drwxr-S---   3 oracle   dba            1024 Dec 09 15:55 .
-rw-r-----   1 oracle   dba            4096 Dec 09 23:51 emdb.nohup.lr
-rw-r-----   1 oracle   dba            1087 Dec 09 23:51 emagent_perl.trc
-rw-r-----   1 oracle   dba            1318 Dec 09 23:51 emagent.log
-rw-r-----   1 oracle   dba          215828 Dec 10 01:30 emdctl.trc
-rw-r-----   1 oracle   dba         1409618 Dec 10 01:30 emoms.trc
-rw-r-----   1 oracle   dba        11467120 Dec 10 01:30 emoms.log
-rw-r-----   1 oracle   dba        18341634 Dec 10 01:30 emdb.nohup
-rw-r-----   1 oracle   dba         1389602 Dec 10 01:30 emagent.trc
oracle@dbnode1 /u01/app/oracle/product/10.2.0/db_1/dbnode1_DEMO1/sysman/log> tail -n40 emoms.log
2016-12-10 01:32:53,270 [Orion Launcher] ERROR app.ContextInitializer contextInitialized.272 - Integration Class not found: oracle.tip.oem.central.instance.ProcessConnectInstanceIntg
2016-12-10 01:32:53,271 [Orion Launcher] ERROR app.ContextInitializer contextInitialized.272 - Integration Class not found: oracle.webdb.admin.em.PortalIntegration
2016-12-10 01:32:53,273 [Orion Launcher] ERROR app.ContextInitializer contextInitialized.272 - Integration Class not found: oracle.webdb.admin.em.SSOIntegration
2016-12-10 01:32:53,275 [Orion Launcher] ERROR app.ContextInitializer contextInitialized.272 - Integration Class not found: oracle.reports.em.RepIntg
2016-12-10 01:32:53,276 [Orion Launcher] ERROR app.ContextInitializer contextInitialized.272 - Integration Class not found: oracle.sysman.ocs.mntr.target.OcsEmailIntegration
2016-12-10 01:32:53,278 [Orion Launcher] ERROR app.ContextInitializer contextInitialized.272 - Integration Class not found: oracle.sysman.ocs.mntr.target.OcsOidIntegration
2016-12-10 01:32:53,279 [Orion Launcher] ERROR app.ContextInitializer contextInitialized.272 - Integration Class not found: oracle.sysman.ocs.mntr.target.OcsOvfIntegration
2016-12-10 01:32:53,280 [Orion Launcher] ERROR app.ContextInitializer contextInitialized.272 - Integration Class not found: oracle.sysman.ocs.mntr.target.OcsWebconfIntegration
2016-12-10 01:32:53,282 [Orion Launcher] ERROR app.ContextInitializer contextInitialized.272 - Integration Class not found: oracle.sysman.ocs.mntr.target.OcsWirelessIntg
2016-12-10 01:32:53,283 [Orion Launcher] ERROR app.ContextInitializer contextInitialized.272 - Integration Class not found: oracle.sysman.ocs.mntr.target.OcsCalGrpIntegration
2016-12-10 01:32:53,957 [OmsServiceDriver thread] ERROR em.failover preTask.390 - java.sql.SQLException: ORA-01422: exact fetch returns more than requested number of rows
ORA-06512: at "SYSMAN.MGMT_FAILOVER", line 35
ORA-06512: at line 1
java.sql.SQLException: ORA-01422: exact fetch returns more than requested number of rows
ORA-06512: at "SYSMAN.MGMT_FAILOVER", line 35
ORA-06512: at line 1
        at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:111)
        at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:330)
        at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:287)
        at oracle.jdbc.driver.T4C8Oall.receive(T4C8Oall.java:744)
        at oracle.jdbc.driver.T4CCallableStatement.doOall8(T4CCallableStatement.java:217)
        at oracle.jdbc.driver.T4CCallableStatement.executeForRows(T4CCallableStatement.java:968)
        at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1284)
        at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3375)
        at oracle.jdbc.driver.OraclePreparedStatement.execute(OraclePreparedStatement.java:3481)
        at oracle.jdbc.driver.OracleCallableStatement.execute(OracleCallableStatement.java:4399)
        at oracle.sysman.util.jdbc.PreparedStatementWrapper.execute(PreparedStatementWrapper.java:630)
        at oracle.sysman.emdrep.failover.OMSHeartbeatRecorder.getId(OMSHeartbeatRecorder.java:181)
        at oracle.sysman.emdrep.failover.OMSHeartbeatRecorder.preTask(OMSHeartbeatRecorder.java:386)
        at oracle.sysman.emdrep.omsservice.OmsServiceDriver.run(OmsServiceDriver.java:147)
2016-12-10 01:32:53,958 [OmsServiceDriver thread] ERROR em.failover die.210 - OMS HB recorder: causing a restart
2016-12-10 01:32:53,961 [OmsServiceDriver thread] ERROR emd.main restart.410 - HealthMonitor Dec 10, 2016 1:32:53 AM OMS Heartbeat Recorder:  error: Could not get id: java.sql.SQLException: ORA-01422: exact fetch returns more than requested number of rows
ORA-06512: at "SYSMAN.MGMT_FAILOVER", line 35
ORA-06512: at line 1
2016-12-10 01:32:53,976 [HttpRequestHandler-2013299148] ERROR eml.OMSHandshake getParameterFromDB.564 -
2016-12-10 01:32:53,977 [HttpRequestHandler-2013299148] ERROR eml.OMSHandshake processFailure.781 - OMSHandshake failed.(AGENT URL = https://dbnode2:3938/emd/main)(ERROR = INTERNAL_ERROR)
oracle@dbnode1 /u01/app/oracle/product/10.2.0/db_1/dbnode1_DEMO1/sysman/log>
oracle@dbnode1 /u01/app/oracle/product/10.2.0/db_1/dbnode1_DEMO1/sysman/log> tail -n40 emoms.trc
2016-12-10 01:32:53,270 [Orion Launcher] ERROR app.ContextInitializer contextInitialized.272 - Integration Class not found: oracle.tip.oem.central.instance.ProcessConnectInstanceIntg
2016-12-10 01:32:53,271 [Orion Launcher] ERROR app.ContextInitializer contextInitialized.272 - Integration Class not found: oracle.webdb.admin.em.PortalIntegration
2016-12-10 01:32:53,273 [Orion Launcher] ERROR app.ContextInitializer contextInitialized.272 - Integration Class not found: oracle.webdb.admin.em.SSOIntegration
2016-12-10 01:32:53,275 [Orion Launcher] ERROR app.ContextInitializer contextInitialized.272 - Integration Class not found: oracle.reports.em.RepIntg
2016-12-10 01:32:53,276 [Orion Launcher] ERROR app.ContextInitializer contextInitialized.272 - Integration Class not found: oracle.sysman.ocs.mntr.target.OcsEmailIntegration
2016-12-10 01:32:53,278 [Orion Launcher] ERROR app.ContextInitializer contextInitialized.272 - Integration Class not found: oracle.sysman.ocs.mntr.target.OcsOidIntegration
2016-12-10 01:32:53,279 [Orion Launcher] ERROR app.ContextInitializer contextInitialized.272 - Integration Class not found: oracle.sysman.ocs.mntr.target.OcsOvfIntegration
2016-12-10 01:32:53,280 [Orion Launcher] ERROR app.ContextInitializer contextInitialized.272 - Integration Class not found: oracle.sysman.ocs.mntr.target.OcsWebconfIntegration
2016-12-10 01:32:53,282 [Orion Launcher] ERROR app.ContextInitializer contextInitialized.272 - Integration Class not found: oracle.sysman.ocs.mntr.target.OcsWirelessIntg
2016-12-10 01:32:53,283 [Orion Launcher] ERROR app.ContextInitializer contextInitialized.272 - Integration Class not found: oracle.sysman.ocs.mntr.target.OcsCalGrpIntegration
2016-12-10 01:32:53,957 [OmsServiceDriver thread] ERROR em.failover preTask.390 - java.sql.SQLException: ORA-01422: exact fetch returns more than requested number of rows
ORA-06512: at "SYSMAN.MGMT_FAILOVER", line 35
ORA-06512: at line 1
java.sql.SQLException: ORA-01422: exact fetch returns more than requested number of rows
ORA-06512: at "SYSMAN.MGMT_FAILOVER", line 35
ORA-06512: at line 1
        at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:111)
        at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:330)
        at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:287)
        at oracle.jdbc.driver.T4C8Oall.receive(T4C8Oall.java:744)
        at oracle.jdbc.driver.T4CCallableStatement.doOall8(T4CCallableStatement.java:217)
        at oracle.jdbc.driver.T4CCallableStatement.executeForRows(T4CCallableStatement.java:968)
        at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1284)
        at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3375)
        at oracle.jdbc.driver.OraclePreparedStatement.execute(OraclePreparedStatement.java:3481)
        at oracle.jdbc.driver.OracleCallableStatement.execute(OracleCallableStatement.java:4399)
        at oracle.sysman.util.jdbc.PreparedStatementWrapper.execute(PreparedStatementWrapper.java:630)
        at oracle.sysman.emdrep.failover.OMSHeartbeatRecorder.getId(OMSHeartbeatRecorder.java:181)
        at oracle.sysman.emdrep.failover.OMSHeartbeatRecorder.preTask(OMSHeartbeatRecorder.java:386)
        at oracle.sysman.emdrep.omsservice.OmsServiceDriver.run(OmsServiceDriver.java:147)
2016-12-10 01:32:53,958 [OmsServiceDriver thread] ERROR em.failover die.210 - OMS HB recorder: causing a restart
2016-12-10 01:32:53,961 [OmsServiceDriver thread] ERROR emd.main restart.410 - HealthMonitor Dec 10, 2016 1:32:53 AM OMS Heartbeat Recorder:  error: Could not get id: java.sql.SQLException: ORA-01422: exact fetch returns more than requested number of rows
ORA-06512: at "SYSMAN.MGMT_FAILOVER", line 35
ORA-06512: at line 1
2016-12-10 01:32:53,976 [HttpRequestHandler-2013299148] ERROR eml.OMSHandshake getParameterFromDB.564 -
2016-12-10 01:32:53,977 [HttpRequestHandler-2013299148] ERROR eml.OMSHandshake processFailure.781 - OMSHandshake failed.(AGENT URL = https://dbnode2:3938/emd/main)(ERROR = INTERNAL_ERROR)
oracle@dbnode1 /u01/app/oracle/product/10.2.0/db_1/dbnode1_DEMO1/sysman/log>

oracle@dbnode1 /u01/app/oracle/product/10.2.0/db_1/dbnode1_DEMO1/sysman/log> ps -ef |  grep emagent      <-- OK
  oracle 671978 635258   0 23:51:16  pts/1  0:05 /u01/app/oracle/product/10.2.0/db_1/bin/emagent
  oracle 704680 651710   0 01:40:31  pts/1  0:00 grep emagent

oracle@dbnode1 /u01/app/oracle/product/10.2.0/db_1/dbnode1_DEMO1/sysman/log>  ps -ef | grep DEMS    <--- Not running
  oracle 700548 651710   0 01:40:45  pts/1  0:00 grep DEMS
oracle@dbnode1 /u01/app/oracle/product/10.2.0/db_1/dbnode1_DEMO1/sysman/log>
oracle@dbnode1 /u01/app/oracle/product/10.2.0/db_1/dbnode1_DEMO1/sysman/log> emctl status agent
Oracle Enterprise Manager 10g Database Control Release 10.2.0.4.0
Copyright (c) 1996, 2007 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
Agent Version     : 10.1.0.6.0
OMS Version       : 10.1.0.6.0
Protocol Version  : 10.1.0.2.0
Agent Home        : /u01/app/oracle/product/10.2.0/db_1/dbnode1_DEMO1
Agent binaries    : /u01/app/oracle/product/10.2.0/db_1
Agent Process ID  : 671978
Parent Process ID : 635258
Agent URL         : https://dbnode1:3938/emd/main
Started at        : 2016-12-09 23:51:16
Started by user   : oracle
Last Reload       : 2016-12-09 23:51:16
Last successful upload                       : (none)
Last attempted upload                        : 2016-12-10 01:43:28
Total Megabytes of XML files uploaded so far :     0.00
Number of XML files pending upload           :     3673
Size of XML files pending upload(MB)         :    59.21
Available disk space on upload filesystem    :    38.31%
---------------------------------------------------------------
Agent is Running and Ready
oracle@dbnode1 /u01/app/oracle/product/10.2.0/db_1/dbnode1_DEMO

but trying to connect to EM console

https://dbnode1:1158/em/console/aboutApplication

https://dbnode2:1158/em/console/aboutApplication

I get the message: cannot connect to the

Unable to connect      
        
          
Firefox can't establish a connection to the server at dbnode2:1158.
        
                
    *   The site could be temporarily unavailable or too busy. Try again in a few
          moments.
    *   If you are unable to load any pages, check your computer's network
          connection.
    *   If your computer or network is protected by a firewall or proxy, make sure
          that Firefox is permitted to access the Web.

Message was edited by: 984556

Comments

Locked Post
New comments cannot be posted to this locked post.

Post Details

Locked on Jan 8 2017
Added on Dec 9 2016
#em, #oracle
6 comments
1,092 views