Hi,
we are getting struck thread error on customer on demand environment. we can only tell the customer what could be the problem and they will fix it. Please find the logs below.
The error is not seen on the UI but takes of around 18 to 20 mins to complete the process. I was wondering whether below error could be the reason why it is taking so much time.
####<Dec 16, 2011 7:00:24 PM CST> <Error> <WebLogicServer> <vmfsnpfgr014> <CompensationServer_1> <[ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1324083624614> <BEA-000337> <[STUCK] ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "637" seconds working on the request "weblogic.servlet.internal.ServletRequestImpl@1ab284fd[
POST /hcmCompensation/faces/HcmCompWorkbenchWorkarea?_adf.ctrl-state=z79ftjyg4_4 HTTP/1.1
Accept: */*
Accept-Language: en-us
Referer: https://hcm-pfgrtest.oracleoutsourcing.com/hcmCompensation/faces/HcmCompWorkbenchWorkarea?_adf.ctrl-state=z79ftjyg4_4&_afrLoop=51666628422117
adf-rich-message: true
Content-Type: application/x-www-form-urlencoded; charset=UTF-8
adf-ads-page-id: 1
Accept-Encoding: gzip, deflate
User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Trident/4.0; .NET CLR 2.0.50727; .NET CLR 1.1.4322; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; InfoPath.2)
Content-Length: 2021
Cache-Control: no-cache
Cookie: JSESSIONID=qv9JTrmcLlQMF0pvR7QTchTm044fxXYdzMbWP9C88xQgkT2vBLcF!173893753!-372509416; _WL_AUTHCOOKIE_JSESSIONID=1iRpGAwfALHZCjdv4YUc; BIGipServerhcm-pfgrtest_oracleoutsourcing_com_http=2935575948.31785.0000; ORA_FUSION_PREFS=v1.0~bG9jYWxlPWVufmRlZmF1bHRMYW5ndWFnZU1hcmtlcj10cnVl; ORA_FND_SESSION_PPFGR010=DEFAULT_PILLAR:ryB1JgPzP5mciFeRcKC6uuj4JQBt8TQ8WmlY5Wl0g7PP4ZadAe8Ffj8ayzSgKdBI:1323666564228; ORA_FND_SESSION_DPFGR0FA=DEFAULT_PILLAR:sZesvQrAxVEHIxBRIzK4ZnQCoLuUV2hwd6i9AoGyZ0dg68LOrfjRiufpaKG+abuW:1324020481682; ObSSOCookie=BSrWFHmoM5g9AT7nOXe%2FhL2xRX8By3TT2aUlefbNIwNXEPMQaH6Kl2fUj9yQzabq5L4tXxGq4sowmn9afJVCqcpYz9bxaVA%2FqL5J232dC8ThHmEUl%2FyOcquT9ZxmtDyg6n1D6KigJrq0fYu7Uh7qLmBTR0aM5V3fw1ad6B1oZUngMOpeE2OU%2B8kn8eAvrtB%2FKDplrVIfNuDpPLHZ3j4M5wWg9oXrPHVD2AEIEKY7jkD4ZCQWYuvspENZnh9Eiy9XTPfdVdlpXx1y9YgY5T5NC60n29oDSqgqA26MjqUDrr8k7xSk%2BP85Sd%2F3VYn7Qzv3GiB8rqVcWYiApEXJb6mdwP2OpF4c0VUaPi8%2BHeJUsgwNYaXEzJhm2GD40ppcRlme; ORA_FND_SESSION_TPFGR0FA=DEFAULT_PILLAR:e5aaeFBIq8ySue1itQXyyAeTSphs9dSfSmr82P00Er7/sKp+KOyBfml/p/EixBHN:1324082886977
OAM_IMPERSONATOR_USER:
OAM_REMOTE_USER: PFG_COMP_ADMIN
ECID-Context: 1.004h3wO4O5s37ERquWZBEH0001RC000GLq;kXjE1ZDLIPIQj8ROhJJRhGTPiJTOoPRRiVAPXJPPnV8IZTRGjTOSNVQSfIPPiJOQNVQSfHQS_HO
Connection: Keep-Alive
Proxy-Remote-User: PFG_COMP_ADMIN
Proxy-Client-IP: 141.147.44.153
X-Forwarded-For: 141.147.44.153
X-WebLogic-KeepAliveSecs: 30
X-WebLogic-Request-ClusterInfo: true
x-weblogic-cluster-hash: vkQEkVBwrYpr3/E4W8uzix+ETuQ
]", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
Thread-21 "[STUCK] ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)'" <alive, in native, suspended, priority=1, DAEMON> {
jrockit.net.SocketNativeIO.readBytesPinned(SocketNativeIO.java:???)
jrockit.net.SocketNativeIO.socketRead(SocketNativeIO.java:24)
java.net.SocketInputStream.socketRead0(SocketInputStream.java:???)
java.net.SocketInputStream.read(SocketInputStream.java:107)
oracle.net.nt.MetricsEnabledInputStream.read(TcpNTAdapter.java:707)
oracle.net.ns.Packet.receive(Packet.java:243)
oracle.net.ns.DataPacket.receive(DataPacket.java:106)
oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:309)
oracle.net.ns.NetInputStream.read(NetInputStream.java:254)
oracle.net.ns.NetInputStream.read(NetInputStream.java:187)
oracle.net.ns.NetInputStream.read(NetInputStream.java:92)
oracle.jdbc.driver.T4CSocketInputStreamWrapper.readNextPacket(T4CSocketInputStreamWrapper.java:122)
oracle.jdbc.driver.T4CSocketInputStreamWrapper.read(T4CSocketInputStreamWrapper.java:77)
oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1172)
oracle.jdbc.driver.T4CMAREngine.unmarshalSB1(T4CMAREngine.java:1153)
oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:303)
oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:181)
oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:382)
oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:70)
oracle.jdbc.driver.T4CPreparedStatement.executeForDescribe(T4CPreparedStatement.java:911)
oracle.jdbc.driver.OracleStatement.executeMaybeDescribe(OracleStatement.java:1226)
oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1330)
oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3722)
oracle.jdbc.driver.OraclePreparedStatement.executeQuery(OraclePreparedStatement.java:3789)
^-- Holding lock: oracle.jdbc.driver.T4CConnection@1705d3e5[thin lock]
oracle.jdbc.driver.OraclePreparedStatementWrapper.executeQuery(OraclePreparedStatementWrapper.java:1667)
weblogic.jdbc.wrapper.PreparedStatement.executeQuery(PreparedStatement.java:123)
oracle.jbo.server.QueryCollection.buildResultSet(QueryCollection.java:1040)
oracle.jbo.server.QueryCollection.executeQuery(QueryCollection.java:895)
oracle.jbo.server.ViewObjectImpl.executeQueryForCollection(ViewObjectImpl.java:7020)
oracle.apps.fnd.applcore.oaext.model.OAViewObjectImpl.executeQueryForCollection(OAViewObjectImpl.java:2209)
oracle.apps.hcm.compensation.common.publicModel.XchgVOImpl.executeParentQueryForCollection(XchgVOImpl.java:61)
^-- Holding lock: oracle.jbo.JboSyncLock@17f5df32[thin lock]
^-- Holding lock: oracle.jbo.JboSyncLock@17f5df32[thin lock]
oracle.apps.hcm.compensation.common.publicModel.XchgVOHelper.executeQueryForCollection(XchgVOHelper.java:32)
oracle.apps.hcm.compensation.common.publicModel.XchgVOImpl.executeQueryForCollection(XchgVOImpl.java:56)
oracle.jbo.server.ViewRowSetImpl.execute(ViewRowSetImpl.java:1052)
^-- Holding lock: oracle.jbo.JboSyncLock@17f5df32[thin lock]
oracle.jbo.server.ViewRowSetImpl.executeQueryForMasters(ViewRowSetImpl.java:1343)
oracle.jbo.server.ViewRowSetImpl.executeQueryForMode(ViewRowSetImpl.java:1270)
oracle.jbo.server.ViewRowSetImpl.executeQuery(ViewRowSetImpl.java:1253)
oracle.jbo.server.ViewObjectImpl.executeQuery(ViewObjectImpl.java:6954)
^-- Holding lock: oracle.jbo.JboSyncLock@17f5df32[thin lock]
oracle.apps.hcm.compensation.distribution.uiModel.applicationModule.ModelProcessingHelper.applyModel(ModelProcessingHelper.java:117)
oracle.apps.hcm.compensation.distribution.uiModel.applicationModule.ModelAMImpl.applyModeltoBudgets(ModelAMImpl.java:334)
sun.reflect.NativeMethodAccessorImpl.invoke0(NativeMethodAccessorImpl.java:???)
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:27)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:575)
oracle.adf.model.binding.DCInvokeMethod.invokeMethod(DCInvokeMethod.java:464)
oracle.adf.model.binding.DCDataControl.invokeMethod(DCDataControl.java:2143)
oracle.adf.model.bc4j.DCJboDataControl.invokeMethod(DCJboDataControl.java:3049)
oracle.adf.model.binding.DCInvokeMethod.callMethod(DCInvokeMethod.java:220)
oracle.jbo.uicli.binding.JUCtrlActionBinding.doIt(JUCtrlActionBinding.java:825)
^-- Holding lock: oracle.jbo.uicli.binding.JUFormBinding@1ab2854d[thin lock]
oracle.adf.model.binding.DCDataControl.invokeOperation(DCDataControl.java:2148)
oracle.jbo.uicli.binding.JUCtrlActionBinding.invoke(JUCtrlActionBinding.java:722)
oracle.adf.controller.v2.lifecycle.PageLifecycleImpl.executeEvent(PageLifecycleImpl.java:347)
oracle.adfinternal.view.faces.model.binding.FacesCtrlActionBinding._execute(FacesCtrlActionBinding.java:246)
oracle.adfinternal.view.faces.model.binding.FacesCtrlActionBinding.execute(FacesCtrlActionBinding.java:210)
sun.reflect.GeneratedMethodAccessor29454.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:575)
com.sun.el.parser.AstValue.invoke(Unknown Source)
com.sun.el.MethodExpressionImpl.invoke(Unknown Source)
oracle.apps.hcm.compensation.budgets.publicUi.util.ADFUtil.invokeEL(ADFUtil.java:115)
oracle.apps.hcm.compensation.budgets.publicUi.util.ADFUtil.invokeEL(ADFUtil.java:102)
oracle.apps.hcm.compensation.budgets.publicUi.bean.BudgetMethodBean.showBudgetMethodPopUp(BudgetMethodBean.java:49)
sun.reflect.NativeMethodAccessorImpl.invoke0(NativeMethodAccessorImpl.java:???)
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:27)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:575)
com.sun.el.parser.AstValue.invoke(Unknown Source)
com.sun.el.MethodExpressionImpl.invoke(Unknown Source)
oracle.apps.hcm.compensation.distribution.ui.common.ModelUtil.invokeEL(ModelUtil.java:62)
oracle.apps.hcm.compensation.distribution.ui.common.ModelUtil.invokeEL(ModelUtil.java:24)
oracle.apps.hcm.compensation.distribution.ui.bean.ModelPreviewBean.handleApplyPopup(ModelPreviewBean.java:118)
sun.reflect.NativeMethodAccessorImpl.invoke0(NativeMethodAccessorImpl.java:???)
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:27)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:575)
com.sun.el.parser.AstValue.invoke(Unknown Source)
com.sun.el.MethodExpressionImpl.invoke(Unknown Source)
org.apache.myfaces.trinidadinternal.taglib.util.MethodExpressionMethodBinding.invoke(MethodExpressionMethodBinding.java:53)
org.apache.myfaces.trinidad.component.UIXComponentBase.broadcastToMethodBinding(UIXComponentBase.java:1251)
org.apache.myfaces.trinidad.component.UIXCommand.broadcast(UIXCommand.java:171)
oracle.adf.view.rich.component.fragment.ContextSwitchingComponent$1.run(ContextSwitchingComponent.java:92)
oracle.adf.view.rich.component.fragment.ContextSwitchingComponent._processPhase(ContextSwitchingComponent.java:351)
oracle.adf.view.rich.component.fragment.ContextSwitchingComponent.broadcast(ContextSwitchingComponent.java:84)
oracle.adf.view.rich.component.fragment.UIXInclude.broadcast(UIXInclude.java:96)
oracle.adf.view.rich.component.fragment.ContextSwitchingComponent$1.run(ContextSwitchingComponent.java:92)
oracle.adf.view.rich.component.fragment.ContextSwitchingComponent._processPhase(ContextSwitchingComponent.java:351)
oracle.adf.view.rich.component.fragment.ContextSwitchingComponent.broadcast(ContextSwitchingComponent.java:84)
oracle.adf.view.rich.component.fragment.UIXInclude.broadcast(UIXInclude.java:96)
oracle.adf.view.rich.component.fragment.UIXRegion.broadcast(UIXRegion.java:131)
oracle.adf.view.rich.component.fragment.UIXRegion.broadcast(UIXRegion.java:131)
oracle.adf.view.rich.component.fragment.UIXInclude.broadcast(UIXInclude.java:96)
oracle.adf.view.rich.component.fragment.ContextSwitchingComponent$1.run(ContextSwitchingComponent.java:92)
oracle.adf.view.rich.component.fragment.ContextSwitchingComponent._processPhase(ContextSwitchingComponent.java:351)
oracle.adf.view.rich.component.fragment.ContextSwitchingComponent.broadcast(ContextSwitchingComponent.java:84)
oracle.adf.view.rich.component.fragment.UIXInclude.broadcast(UIXInclude.java:96)
oracle.adfinternal.view.faces.lifecycle.LifecycleImpl.broadcastEvents(LifecycleImpl.java:867)
oracle.adfinternal.view.faces.lifecycle.LifecycleImpl._executePhase(LifecycleImpl.java:253)
oracle.adfinternal.view.faces.lifecycle.LifecycleImpl.execute(LifecycleImpl.java:166)
javax.faces.webapp.FacesServlet.service(FacesServlet.java:246)
weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:224)
weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:108)
weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:206)
weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:26)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
oracle.bi.nanserver.adf.servlet.BIADFServletFilter.doFilter(BIADFServletFilter.java:45)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
oracle.webcenter.framework.events.dispatcher.EventDispatcherFilter.doFilter(EventDispatcherFilter.java:42)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
oracle.portlet.client.adapter.adf.ADFPortletFilter.doFilter(ADFPortletFilter.java:32)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
oracle.adf.view.page.editor.webapp.WebCenterComposerFilter.doFilter(WebCenterComposerFilter.java:77)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
oracle.adf.model.servlet.ADFBindingFilter.doFilter(ADFBindingFilter.java:147)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
oracle.adfinternal.view.faces.webapp.rich.RegistrationFilter.doFilter(RegistrationFilter.java:69)
org.apache.myfaces.trinidadinternal.webapp.TrinidadFilterImpl$FilterListChain.doFilter(TrinidadFilterImpl.java:437)
oracle.adfinternal.view.faces.activedata.AdsFilter.doFilter(AdsFilter.java:44)
org.apache.myfaces.trinidadinternal.webapp.TrinidadFilterImpl$FilterListChain.doFilter(TrinidadFilterImpl.java:437)
org.apache.myfaces.trinidadinternal.webapp.TrinidadFilterImpl._doFilterImpl(TrinidadFilterImpl.java:231)
org.apache.myfaces.trinidadinternal.webapp.TrinidadFilterImpl.doFilter(TrinidadFilterImpl.java:133)
org.apache.myfaces.trinidad.webapp.TrinidadFilter.doFilter(TrinidadFilter.java:91)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
oracle.apps.fnd.applcore.common.ApplSessionFilter.doFilter(ApplSessionFilter.java:122)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
oracle.security.wls.filter.SSOSessionSynchronizationFilter.doFilter(SSOSessionSynchronizationFilter.java:121)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
oracle.wcps.client.PersonalizationFilter.doFilter(PersonalizationFilter.java:74)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
oracle.webcenter.content.integration.servlets.ContentServletFilter.doFilter(ContentServletFilter.java:147)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
oracle.adf.library.webapp.LibraryFilter.doFilter(LibraryFilter.java:163)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
oracle.bpel.services.workflow.client.worklist.util.WorkflowFilter.doFilter(WorkflowFilter.java:107)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
oracle.security.jps.ee.http.JpsAbsFilter$1.run(JpsAbsFilter.java:111)
oracle.security.jps.util.JpsSubject.doAsPrivileged(JpsSubject.java:299)
oracle.security.jps.ee.util.JpsPlatformUtil.runJaasMode(JpsPlatformUtil.java:405)
oracle.security.jps.ee.http.JpsAbsFilter.runJaasMode(JpsAbsFilter.java:94)
oracle.security.jps.ee.http.JpsAbsFilter.doFilter(JpsAbsFilter.java:98)
oracle.security.jps.ee.http.JpsFilter.doFilter(JpsFilter.java:70)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
oracle.dms.servlet.DMSServletFilter.doFilter(DMSServletFilter.java:86)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
weblogic.servlet.internal.RequestEventsFilter.doFilter(RequestEventsFilter.java:25)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3687)
weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3681)
weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:308)
weblogic.security.service.SecurityManager.runAs(SecurityManager.java:116)
weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2213)
weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2135)
weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1420)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:203)
weblogic.work.ExecuteThread.run(ExecuteThread.java:170)
}