Hello all! Recently I have performed a fresh installation of Oracle Cloud Control 13.3. The first thing I have noticed was that the Weblogic admin server appears to be down according to Cloud Control.
When querying via "emctl status oms -details" everything seems ok:
[oracle@ora-cloud-control nodemanager]$ emctl status oms -details
Oracle Enterprise Manager Cloud Control 13c Release 3
Copyright (c) 1996, 2018 Oracle Corporation. All rights reserved.
Enter Enterprise Manager Root (SYSMAN) Password :
Console Server Host : ora-cloud-control.localdomain
HTTP Console Port : 7788
HTTPS Console Port : 7803
HTTP Upload Port : 4889
HTTPS Upload Port : 4903
EM Instance Home : /u01/app/oracle/gc_inst/em/EMGC_OMS1
OMS Log Directory Location : /u01/app/oracle/gc_inst/em/EMGC_OMS1/sysman/log
OMS is not configured with SLB or virtual hostname
Agent Upload is locked.
OMS Console is locked.
Active CA ID: 1
Console URL: https://ora-cloud-control.localdomain:7803/em
Upload URL: https://ora-cloud-control.localdomain:4903/empbs/upload
WLS Domain Information
Domain Name : GCDomain
Admin Server Host : ora-cloud-control.localdomain
Admin Server HTTPS Port: 7102
Admin Server is RUNNING
Oracle Management Server Information
Managed Server Instance Name: EMGC_OMS1
Oracle Management Server Instance Host: ora-cloud-control.localdomain
WebTier is Up
Oracle Management Server is Up
JVMD Engine is Up
BI Publisher Server Information
BI Publisher Managed Server Name: BIP
BI Publisher Server is Up
BI Publisher HTTP Managed Server Port : 9701
BI Publisher HTTPS Managed Server Port : 9803
BI Publisher HTTP OHS Port : 9788
BI Publisher HTTPS OHS Port : 9851
BI Publisher is locked.
BI Publisher Server named 'BIP' running at URL: https://ora-cloud-control.localdomain:9851/xmlpserver/servlet/home
BI Publisher Server Logs: /u01/app/oracle/gc_inst/user_projects/domains/GCDomain/servers/BIP/logs/
BI Publisher Log : /u01/app/oracle/gc_inst/user_projects/domains/GCDomain/servers/BIP/logs/bipublisher/bipublisher.log
However, the admin server cannot be reached at: https://ora-cloud-control.localdomain:7102/console nor via WLST:
wls:/offline> connect('weblogic','*******','t3s://ora-cloud-control.localdomain:7102')
Connecting to t3s://ora-cloud-control.localdomain:7102 with userid weblogic ...
<Apr 1, 2021 10:02:29 PM CEST> <Info> <Security> <BEA-090905> <Disabling the CryptoJ JCE Provider self-integrity check for better startup performance. To enable this check, specify -Dweblogic.security.allowCryptoJDefaultJCEVerification=true.>
<Apr 1, 2021 10:02:29 PM CEST> <Info> <Security> <BEA-090906> <Changing the default Random Number Generator in RSA CryptoJ from ECDRBG128 to FIPS186PRNG. To disable this change, specify -Dweblogic.security.allowCryptoJDefaultPRNG=true.>
<Apr 1, 2021 10:02:29 PM CEST> <Info> <Security> <BEA-090908> <Using the default WebLogic SSL Hostname Verifier implementation.>
Traceback (innermost last):
File "<console>", line 1, in ?
File "<iostream>", line 19, in connect
File "<iostream>", line 552, in raiseWLSTException
WLSTException: Error occurred while performing connect : Cannot connect via t3s or https. If using demo certs, verify that the -Dweblogic.security.TrustKeyStore=DemoTrust system property is set. : t3s://ora-cloud-control.localdomain:7102: Destination 127.0.0.1, 7102 unreachable; nested exception is:
javax.net.ssl.SSLHandshakeException: General SSLEngine problem; No available router to destination
Use dumpStack() to view the full stacktrace :
dumpStack follows:
wls:/offline> dumpStack()
This Exception occurred at Thu Apr 01 22:02:30 CEST 2021.
javax.naming.CommunicationException: t3s://ora-cloud-control.localdomain:7102: Destination 127.0.0.1, 7102 unreachable; nested exception is:
javax.net.ssl.SSLHandshakeException: General SSLEngine problem; No available router to destination [Root exception is java.net.ConnectException: t3s://ora-cloud-control.localdomain:7102: Destination 127.0.0.1, 7102 unreachable; nested exception is:
javax.net.ssl.SSLHandshakeException: General SSLEngine problem; No available router to destination]
at weblogic.jndi.internal.ExceptionTranslator.toNamingException(ExceptionTranslator.java:40)
at weblogic.jndi.WLInitialContextFactoryDelegate.toNamingException(WLInitialContextFactoryDelegate.java:808)
at weblogic.jndi.WLInitialContextFactoryDelegate.getInitialContext(WLInitialContextFactoryDelegate.java:363)
at weblogic.jndi.Environment.getContext(Environment.java:319)
at weblogic.jndi.Environment.getContext(Environment.java:288)
at weblogic.jndi.WLInitialContextFactory.getInitialContext(WLInitialContextFactory.java:117)
at javax.naming.spi.NamingManager.getInitialContext(NamingManager.java:684)
at javax.naming.InitialContext.getDefaultInitCtx(InitialContext.java:307)
at javax.naming.InitialContext.init(InitialContext.java:242)
at javax.naming.InitialContext.<init>(InitialContext.java:216)
at weblogic.management.scripting.WLSTHelper.populateInitialContext(WLSTHelper.java:519)
at weblogic.management.scripting.WLSTHelper.initDeprecatedConnection(WLSTHelper.java:570)
at weblogic.management.scripting.WLSTHelper.initConnections(WLSTHelper.java:310)
at weblogic.management.scripting.WLSTHelper.connect(WLSTHelper.java:200)
at weblogic.management.scripting.WLScriptContext.connect(WLScriptContext.java:67)
at weblogic.management.scripting.utils.WLSTUtil.initializeOnlineWLST(WLSTUtil.java:188)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.python.core.PyReflectedFunction.__call__(Unknown Source)
at org.python.core.PyMethod.__call__(Unknown Source)
at org.python.core.PyObject.__call__(Unknown Source)
at org.python.core.PyObject.invoke(Unknown Source)
at org.python.pycode._pyx7.connect$1(<iostream>:13)
at org.python.pycode._pyx7.call_function(<iostream>)
at org.python.core.PyTableCode.call(Unknown Source)
at org.python.core.PyTableCode.call(Unknown Source)
at org.python.core.PyTableCode.call(Unknown Source)
at org.python.core.PyFunction.__call__(Unknown Source)
at org.python.pycode._pyx87.f$0(<console>:1)
at org.python.pycode._pyx87.call_function(<console>)
at org.python.core.PyTableCode.call(Unknown Source)
at org.python.core.PyCode.call(Unknown Source)
at org.python.core.Py.runCode(Py.java:1226)
at org.python.core.Py.exec(Py.java:1252)
at org.python.util.PythonInterpreter.exec(Unknown Source)
at org.python.util.InteractiveInterpreter.runcode(Unknown Source)
at org.python.util.InteractiveInterpreter.runsource(Unknown Source)
at org.python.util.InteractiveInterpreter.runsource(Unknown Source)
at weblogic.management.scripting.utils.WLSTInterpreter.runsource(WLSTInterpreter.java:910)
at weblogic.management.scripting.WLST.main(WLST.java:217)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at weblogic.WLST.main(WLST.java:29)
Caused by: java.net.ConnectException: t3s://ora-cloud-control.localdomain:7102: Destination 127.0.0.1, 7102 unreachable; nested exception is:
javax.net.ssl.SSLHandshakeException: General SSLEngine problem; No available router to destination
at weblogic.rjvm.RJVMFinder.findOrCreateInternal(RJVMFinder.java:241)
at weblogic.rjvm.RJVMFinder.findOrCreate(RJVMFinder.java:169)
at weblogic.rjvm.ServerURL.findOrCreateRJVM(ServerURL.java:177)
at weblogic.jndi.WLInitialContextFactoryDelegate$1.run(WLInitialContextFactoryDelegate.java:342)
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:363)
at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:146)
at weblogic.jndi.WLInitialContextFactoryDelegate.getInitialContext(WLInitialContextFactoryDelegate.java:337)
... 44 more
Caused by: java.rmi.ConnectException: Destination 127.0.0.1, 7102 unreachable; nested exception is:
javax.net.ssl.SSLHandshakeException: General SSLEngine problem; No available router to destination
at weblogic.rjvm.ConnectionManager.bootstrap(ConnectionManager.java:489)
at weblogic.rjvm.ConnectionManager.bootstrap(ConnectionManager.java:327)
at weblogic.rjvm.RJVMManager.findOrCreateRemoteInternal(RJVMManager.java:309)
at weblogic.rjvm.RJVMManager.findOrCreate(RJVMManager.java:213)
at weblogic.rjvm.RJVMFinder.findOrCreateRemoteServer(RJVMFinder.java:263)
at weblogic.rjvm.RJVMFinder.findOrCreateInternal(RJVMFinder.java:225)
... 50 more
javax.naming.CommunicationException: t3s://ora-cloud-control.localdomain:7102: Destination 127.0.0.1, 7102 unreachable; nested exception is:
javax.net.ssl.SSLHandshakeException: General SSLEngine problem; No available router to destination [Root exception is java.net.ConnectException: t3s://ora-cloud-control.localdomain:7102: Destination 127.0.0.1, 7102 unreachable; nested exception is:
javax.net.ssl.SSLHandshakeException: General SSLEngine problem; No available router to destination]
wls:/offline>
I have tried three things to resolve the issue according to what I have found browsing the web:
-appending **-Dweblogic.security.TrustKeyStore=DemoTrust** to JAVA_OPTIONS string in /u01/app/oracle/gc_inst/user_projects/domains/GCDomain/bin/startWebLogic.sh file
-extracting the SSL certificate from https://ora-cloud-control.localdomain:7803 and importing it into the following keystores using [[https://stackoverflow.com/questions/11617210/how-to-properly-import-a-selfsigned-certificate-into-java-keystore-that-is-avail][1]](https://stackoverflow.com/questions/11617210/how-to-properly-import-a-selfsigned-certificate-into-java-keystore-that-is-avail][1]) as a reference:
/u01/app/oracle/middleware/oracle_common/jdk/jre/lib/security/cacerts
/u01/app/oracle/middleware/wlserver/server/lib/cacerts
/u01/app/oracle/agent/agent_13.3.0.0.0/oracle_common/jdk/jre/lib/security/cacerts
-importing the same certificate into /u01/app/oracle/middleware/wlserver/server/lib/DemoTrust.jks trustore
I have tried to restart the whole OMS stack after each of the actions described above but with no luck, still getting the very same error message :-(
Any ideas please?
Kind regards,
Michal