we are encountering a stuck thread issue in production due to ADF Active Data Service polling calls.
We make use of ADS for collaboration in our project. Below is the ADS config that we use:-
<ads:adf-activedata-config xmlns="http://xmlns.oracle.com/adf/activedata/config">
<transport>long-polling</transport>
<keep-alive-interval>20000</keep-alive-interval>
</ads:adf-activedata-config>
In production, occasionally, where we are seeing below kind of exceptions at multiple places:
####<Jul 20, 2021 6:06:59,110 AM UTC> <Error> <WebLogicServer> <912d0de7-0008-4a97-896d-96d559b49e5e-ms2.dpapp.oic.oraclevcn.com> <oic_server2> <[ACTIVE] ExecuteThread: '129' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <d87f52bf-4b70-4a3d-9fb6-46d6e7245563-0049c06c> <1626761219110> <[severity-value: 8] [rid: 0] [partition-id: 0] [partition-name: DOMAIN] > <BEA-000337> <[STUCK] ExecuteThread: '137' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "702" seconds working on the request "Http Request Information: weblogic.servlet.internal.ServletRequestImpl@5e39ace9[GET /ic/process/composer/adwwls/wls.ads]
", which is more than the configured time (StuckThreadMaxTime) of "600" seconds in "server-failure-trigger". Stack trace:
oracle.adfinternal.view.faces.activedata.AdsProcessor.doProcess(AdsProcessor.java:208)
oracle.adfinternal.view.faces.activedata.AdsServlet.doGet(AdsServlet.java:116)
javax.servlet.http.HttpServlet.service(HttpServlet.java:687)
javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelp
I matched the timestamp in access logs, I can see some calls going to wls.ads reconnect action like these:
2021-07-20 06:02:02 859.951 0 GET /ic/process/composer/adwwls/wls.ads?action=reconnect&windowId=w1bnu1p6dtt&replaceMasterWindowId=w1bnu1p6dtt&pageId=2&replaceMasterPageId=2&messageCount=1&Adf-Window-Id=w1bnu1p6dtt 200 "d87f52bf-4b70-4a3d-9fb6-46d6e7245563-0049b0ed" - - - -
As we can see, the above call took 859 seconds to complete, which I believe, led to the stuck thread. I tried to simulate the behavior on my chrome browser by throttling to absurdly high latency, still could not get such high response time. Most I could simulate was 60 seconds.
Any ideas, what could cause such large response times and stuck threads? Any kind of help would be much appreciated.