Hi,
NOTE: DID NOT FIND THE CATEGORY FOR SOA/WEBLOGIC and hence chose JAVA CATEGORY. PLEASE REDIRECT TO THE SOA/WEBLOGIC CATEGORY.
We had an earlier implementation where we were producing a message out of a SOA webservice using the JNDI "eis/tibjms/Queue" and hence we were using the queue "eis/tibjms/Queue" under Deployments-JMS Adapater->Configuration->Outbound Connection Pools, where in we had provided the third party details for the attributes like ConnectionFactoryLocation, Factory Properties, Password and UserName and it was working fine. Since this is a TIBCO JMS queue, we even had a JAR file with the name "tibjms.jar" in our SOA path "<SOAInstall_DIR>/user_projects/domains/<DOMAIN_NAME>/lib".
The setup mentioned above was identical in our DEV/TEST and PROD SOA Weblogic servers. All the 3 environments were working with TIBCO setup without any issues.
Now, the third party is getting away from the TIBCO implementation and are moving to a WLS queue. So, in order to be compatible and be able to produce the message correctly to the right type of the new WLS queue, we have started using the JNDI "eis/wls/Queue", instead of the above mentioned JNDI name for TIBCO, by changing our web service code in the partner link for JMS.
Also, the credentials and the values for the properties "ConnectionFactoryLocation, Factory Properties" were changed by the other team and have provided them to us for configuring them on our end.
We have configured them and were able to produce message to the new WLS queue in our Test SOA weblogic without issues and integration testing was successful.
Now, we are trying to make the same shift in our SOA Production environment (which is equivalent to the TEST SOA instance where things are working, with all the patches etc.,), it is not working and the Message production and publishing is Timing out and is also causing the entire SOA server to hog and causing stuck threads etc.,
We have tried to compare all the attributes of the queue "eis/wls/Queue" in both TEST and PROD SOA Weblogic and they are all same.
We have even tried to put in the TEST credentials of the Third Party WLS queue into Production SOA Weblogic queue setup on our side, but it does not work.
We are unable to find what could be so different on the Production SOA Weblogic server that can cause this which is working fine on the TEST SOA Weblogic server.
The "tibjms.jar" file is still there on all the 3 DEV/TEST/PROD SOA paths.
Any hints here would be of great help.
When the hogging happens, the SOA server spits out a log as follows, which in a way is pointing us to the tibjms.jar, but the same JAR is there on Test SOA Server also, where it works.
SOA Server Log:
2022-10-17 14:05:17
Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.341-b10 mixed mode):
"soa_coherence_gar:soa.instanceTrackingSideCacheServiceWorker:0x05B7:1209474" #9399 daemon prio=5 os_prio=0 tid=0x00007fb150020000 nid=0x618b in Object.wait() [0x00007fb10b10a000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:460)
at com.oracle.common.base.Blocking.wait(Blocking.java:97)
at com.oracle.common.base.Blocking.wait(Blocking.java:80)
at com.tangosol.coherence.component.util.Queue.await(Queue.CDB:9)
- locked <0x00000007bdeabf58> (a com.tangosol.coherence.component.util.queue.concurrentQueue.dualQueue.AssociationQueue)
at com.tangosol.coherence.component.util.queue.ConcurrentQueue.await(ConcurrentQueue.CDB:12)
at com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:13)
at com.tangosol.coherence.component.util.DaemonPool$Daemon.onWait(DaemonPool.CDB:3)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid$DaemonPool$Daemon.onWait(Grid.CDB:3)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:43)
at java.lang.Thread.run(Thread.java:750)
"TIBCO EMS Connections Pinger" #471 daemon prio=9 os_prio=0 tid=0x00007fb19c109000 nid=0x734 waiting on condition [0x00007fb10bb10000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at com.tibco.tibjms.TibjmsxLinkTcp$Pinger.run(TibjmsxLinkTcp.java:146)
at java.lang.Thread.run(Thread.java:750)
"DaemonWorkThread: '7' of WorkManager: 'default_Adapters'" #417 daemon prio=10 os_prio=0 tid=0x00007fb179e0e000 nid=0x65b0 runnable [0x00007fb10b60d000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:197)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:378)
- locked <0x00000007bd5a0af0> (a java.lang.Object)
at oracle.net.nt.TimeoutSocketChannel.read(TimeoutSocketChannel.java:145)
at oracle.net.ns.NIOHeader.readHeaderBuffer(NIOHeader.java:83)
at oracle.net.ns.NIOPacket.readFromSocketChannel(NIOPacket.java:140)
at oracle.net.ns.NIOPacket.readFromSocketChannel(NIOPacket.java:102)
at oracle.net.ns.NIONSDataChannel.readDataFromSocketChannel(NIONSDataChannel.java:81)
at oracle.jdbc.driver.T4CMAREngineNIO.prepareForReading(T4CMAREngineNIO.java:99)
at oracle.jdbc.driver.T4CMAREngineNIO.unmarshalUB1(T4CMAREngineNIO.java:535)
at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:486)
at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:253)
at oracle.jdbc.driver.T4Caqdq.doOAQDQ(T4Caqdq.java:191)
at oracle.jdbc.driver.T4CConnection.doDequeue(T4CConnection.java:4736)
- locked <0x00000007bd32e2f0> (a oracle.jdbc.driver.T4CConnection)
at oracle.jdbc.driver.PhysicalConnection.dequeue(PhysicalConnection.java:8161)
at oracle.jdbc.driver.PhysicalConnection.dequeue(PhysicalConnection.java:8217)
at oracle.jdbc.OracleConnectionWrapper.dequeue(OracleConnectionWrapper.java:1584)
at weblogic.jdbc.wrapper.XAConnection_oracle_jdbc_driver_LogicalConnection.dequeue(Unknown Source)
at weblogic.jdbc.wrapper.JTAConnection_weblogic_jdbc_wrapper_XAConnection_oracle_jdbc_driver_LogicalConnection.dequeue(Unknown Source)
at oracle.tip.adapter.aq.v2.database.AbstractDequeuer.dequeue(AbstractDequeuer.java:54)
at oracle.tip.adapter.aq.v2.database.AppsDequeueAgent.dequeue(AppsDequeueAgent.java:159)
at oracle.tip.adapter.aq.v2.database.AbstractDequeueAgent.dequeueMessage(AbstractDequeueAgent.java:397)
at oracle.tip.adapter.aq.v2.database.AbstractDequeueAgent.getInputRecord(AbstractDequeueAgent.java:370)
at oracle.tip.adapter.aq.v2.database.AbstractDequeueAgent.runInbound(AbstractDequeueAgent.java:156)
at oracle.tip.adapter.aq.v2.database.AbstractDequeueAgent.run(AbstractDequeueAgent.java:106)
at oracle.integration.platform.blocks.executor.WorkManagerExecutor$1.run(WorkManagerExecutor.java:184)
at weblogic.work.j2ee.J2EEWorkManager$WorkWithListener.run(J2EEWorkManager.java:209)
at weblogic.invocation.ComponentInvocationContextManager._runAs(ComponentInvocationContextManager.java:352)
at weblogic.invocation.ComponentInvocationContextManager.runAs(ComponentInvocationContextManager.java:337)
at weblogic.work.LivePartitionUtility.doRunWorkUnderContext(LivePartitionUtility.java:57)
at weblogic.work.PartitionUtility.runWorkUnderContext(PartitionUtility.java:41)
at weblogic.work.SelfTuningWorkManagerImpl.runWorkUnderContext(SelfTuningWorkManagerImpl.java:652)
at weblogic.work.SelfTuningWorkManagerImpl.runWorkUnderContext(SelfTuningWorkManagerImpl.java:630)
at weblogic.work.DaemonWorkThread.run(DaemonWorkThread.java:39)
"DaemonWorkThread: '6' of WorkManager: 'default_Adapters'" #416 daemon prio=10 os_prio=0 tid=0x00007fb17a05c000 nid=0x6580 runnable [0x00007fb111003000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:197)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:378)
- locked <0x0000000758977510> (a java.lang.Object)
at oracle.net.nt.TimeoutSocketChannel.read(TimeoutSocketChannel.java:145)
at oracle.net.ns.NIOHeader.readHeaderBuffer(NIOHeader.java:83)
at oracle.net.ns.NIOPacket.readFromSocketChannel(NIOPacket.java:140)
at oracle.net.ns.NIOPacket.readFromSocketChannel(NIOPacket.java:102)
at oracle.net.ns.NIONSDataChannel.readDataFromSocketChannel(NIONSDataChannel.java:81)
at oracle.jdbc.driver.T4CMAREngineNIO.prepareForReading(T4CMAREngineNIO.java:99)
at oracle.jdbc.driver.T4CMAREngineNIO.unmarshalUB1(T4CMAREngineNIO.java:535)
at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:486)
at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:253)
at oracle.jdbc.driver.T4Caqdq.doOAQDQ(T4Caqdq.java:191)
at oracle.jdbc.driver.T4CConnection.doDequeue(T4CConnection.java:4736)
- locked <0x000000075896f528> (a oracle.jdbc.driver.T4CConnection)
at oracle.jdbc.driver.PhysicalConnection.dequeue(PhysicalConnection.java:8161)
at oracle.jdbc.driver.PhysicalConnection.dequeue(PhysicalConnection.java:8217)
at oracle.jdbc.OracleConnectionWrapper.dequeue(OracleConnectionWrapper.java:1584)
at weblogic.jdbc.wrapper.XAConnection_oracle_jdbc_driver_LogicalConnection.dequeue(Unknown Source)
at weblogic.jdbc.wrapper.JTAConnection_weblogic_jdbc_wrapper_XAConnection_oracle_jdbc_driver_LogicalConnection.dequeue(Unknown Source)
at oracle.tip.adapter.aq.v2.database.AbstractDequeuer.dequeue(AbstractDequeuer.java:54)
at oracle.tip.adapter.aq.v2.database.AppsDequeueAgent.dequeue(AppsDequeueAgent.java:159)
at oracle.tip.adapter.aq.v2.database.AbstractDequeueAgent.dequeueMessage(AbstractDequeueAgent.java:397)
at oracle.tip.adapter.aq.v2.database.AbstractDequeueAgent.getInputRecord(AbstractDequeueAgent.java:370)
at oracle.tip.adapter.aq.v2.database.AbstractDequeueAgent.runInbound(AbstractDequeueAgent.java:156)
at oracle.tip.adapter.aq.v2.database.AbstractDequeueAgent.run(AbstractDequeueAgent.java:106)
at oracle.integration.platform.blocks.executor.WorkManagerExecutor$1.run(WorkManagerExecutor.java:184)
at weblogic.work.j2ee.J2EEWorkManager$WorkWithListener.run(J2EEWorkManager.java:209)
at weblogic.invocation.ComponentInvocationContextManager._runAs(ComponentInvocationContextManager.java:352)
at weblogic.invocation.ComponentInvocationContextManager.runAs(ComponentInvocationContextManager.java:337)
at weblogic.work.LivePartitionUtility.doRunWorkUnderContext(LivePartitionUtility.java:57)
at weblogic.work.PartitionUtility.runWorkUnderContext(PartitionUtility.java:41)
at weblogic.work.SelfTuningWorkManagerImpl.runWorkUnderContext(SelfTuningWorkManagerImpl.java:652)
at weblogic.work.SelfTuningWorkManagerImpl.runWorkUnderContext(SelfTuningWorkManagerImpl.java:630)
at weblogic.work.DaemonWorkThread.run(DaemonWorkThread.java:39)
"ClusterNodeHeartbeat" #415 daemon prio=10 os_prio=0 tid=0x00007fb1790cd800 nid=0x657f waiting on condition [0x00007fb10d6ec000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at com.collaxa.cube.cluster.ClusterService$1.run(ClusterService.java:218)
Regards,
Vijay Keerthi