Hi Experts,
My WLS: 10.3.3.0
After our fusion app run for a while, we got warnings in wls admin console, saying that there are stuck threads, and app pages take infinite time to display. We tried several methods to tune, including increasing queue length for WM, set longer stuck max time, but didn't work.
Attached is part of server log and thread dump, any help would be highly appreciated.
Thanks,
Todd
---------------------------------------------------------------------------------------
Server Log:
####<Feb 11, 2011 11:35:23 PM PST> <Error> <WebLogicServer> <stasg15> <FOD_stasg15_MS1> <[ACTIVE] ExecuteThread: '14' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1297496123731> <BEA-000337> <[STUCK] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "1,903" seconds working on the request "weblogic.servlet.internal.ServletRequestImpl@f7febc6[
POST /StoreFrontModule/faces/myPage?_adf.ctrl-state=109nxs7t6x_4 HTTP/1.1
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.12) Gecko/20101026 Firefox/3.6.12
Accept: text/html, image/gif, image/jpeg, */*
Accept-Language: en-us
Connection: Keep-Alive
Accept-Encoding: gzip
Content-Length: 702
Content-Type: multipart/form-data; boundary=---------------------------174211871619718
Cookie: JSESSIONID=JqnJNWvC0GQdZLvflymMm9xzwVbjnJHc5tnGdv4Dmf2vdt2J22KT!-1763094287
]", which is more than the configured time (StuckThreadMaxTime) of "1,800" seconds. Stack trace:
Thread-59 "[STUCK] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'" <alive, suspended, parked, priority=1, DAEMON> {
java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747)
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:770)
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1114)
java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:183)
java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:262)
oracle.adf.model.dcframe.DataControlFrameImpl.lock(DataControlFrameImpl.java:286)
oracle.adf.model.dcframe.DataControlFrameImpl.beginRequest(DataControlFrameImpl.java:336)
oracle.adf.model.BindingContext.setCurrentFrame(BindingContext.java:2102)
oracle.adf.model.BindingContext.setCurrentDataControlFrame(BindingContext.java:2005)
oracle.adfinternal.controller.util.model.DCFrameImpl.makeCurrent(DCFrameImpl.java:125)
oracle.adfinternal.controller.state.ViewPortContextImpl.makeCurrent(ViewPortContextImpl.java:994)
oracle.adfinternal.controller.state.RequestState.setCurrentViewPortContext(RequestState.java:151)
oracle.adfinternal.controller.state.ControllerState.setRequestState(ControllerState.java:884)
oracle.adfinternal.controller.state.ControllerState.synchronizeStatePart1(ControllerState.java:227)
oracle.adfinternal.controller.application.SyncNavigationStateListener.beforePhase(SyncNavigationStateListener.java:95)
oracle.adfinternal.controller.lifecycle.ADFLifecycleImpl$PagePhaseListenerWrapper.beforePhase(ADFLifecycleImpl.java:536)
oracle.adfinternal.controller.lifecycle.LifecycleImpl.internalDispatchBeforeEvent(LifecycleImpl.java:89)
oracle.adfinternal.controller.lifecycle.LifecycleImpl.dispatchBeforePagePhaseEvent(LifecycleImpl.java:141)
oracle.adfinternal.controller.faces.lifecycle.ADFPhaseListener$PhaseInvokerImpl.dispatchBeforePagePhaseEvent(ADFPhaseListener.java:112)
oracle.adfinternal.controller.faces.lifecycle.ADFPhaseListener.beforePhase(ADFPhaseListener.java:58)
oracle.adfinternal.controller.faces.lifecycle.ADFLifecyclePhaseListener.beforePhase(ADFLifecyclePhaseListener.java:44)
oracle.adfinternal.view.faces.lifecycle.LifecycleImpl._executePhase(LifecycleImpl.java:244)
oracle.adfinternal.view.faces.lifecycle.LifecycleImpl.execute(LifecycleImpl.java:157)
javax.faces.webapp.FacesServlet.service(FacesServlet.java:246)
weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:224)
weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:108)
weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:206)
weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:26)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
oracle.adf.model.servlet.ADFBindingFilter.doFilter(ADFBindingFilter.java:134)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
oracle.adfinternal.view.faces.webapp.rich.RegistrationFilter.doFilter(RegistrationFilter.java:69)
org.apache.myfaces.trinidadinternal.webapp.TrinidadFilterImpl$FilterListChain.doFilter(TrinidadFilterImpl.java:411)
oracle.adfinternal.view.faces.activedata.AdsFilter.doFilter(AdsFilter.java:44)
org.apache.myfaces.trinidadinternal.webapp.TrinidadFilterImpl$FilterListChain.doFilter(TrinidadFilterImpl.java:411)
org.apache.myfaces.trinidadinternal.webapp.TrinidadFilterImpl._doFilterImpl(TrinidadFilterImpl.java:210)
org.apache.myfaces.trinidadinternal.webapp.TrinidadFilterImpl.doFilter(TrinidadFilterImpl.java:126)
org.apache.myfaces.trinidad.webapp.TrinidadFilter.doFilter(TrinidadFilter.java:91)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
oracle.security.jps.ee.http.JpsAbsFilter$1.run(JpsAbsFilter.java:94)
oracle.security.jps.util.JpsSubject.doAsPrivileged(JpsSubject.java:299)
oracle.security.jps.ee.util.JpsPlatformUtil.runJaasMode(JpsPlatformUtil.java:406)
oracle.security.jps.ee.http.JpsAbsFilter.doFilter(JpsAbsFilter.java:82)
oracle.security.jps.ee.http.JpsFilter.doFilter(JpsFilter.java:70)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
com.bea.content.manager.servlets.ContentServletFilter.doFilter(ContentServletFilter.java:157)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
oracle.adf.library.webapp.LibraryFilter.doFilter(LibraryFilter.java:147)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
oracle.security.jps.ee.http.JpsAbsFilter$1.run(JpsAbsFilter.java:94)
oracle.security.jps.util.JpsSubject.doAsPrivileged(JpsSubject.java:299)
oracle.security.jps.ee.util.JpsPlatformUtil.runJaasMode(JpsPlatformUtil.java:406)
oracle.security.jps.ee.http.JpsAbsFilter.doFilter(JpsAbsFilter.java:82)
oracle.security.jps.ee.http.JpsFilter.doFilter(JpsFilter.java:70)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
oracle.dms.wls.DMSServletFilter.doFilter(DMSServletFilter.java:114)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
weblogic.servlet.internal.RequestEventsFilter.doFilter(RequestEventsFilter.java:25)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.doIt(WebAppServletContext.java:3656)
weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3650)
weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:308)
weblogic.security.service.SecurityManager.runAs(SecurityManager.java:117)
weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2204)
weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2126)
weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1412)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:198)
weblogic.work.ExecuteThread.run(ExecuteThread.java:165)
}
-------------------------------------------------------------------------------------
Thread Dump:
"[STUCK] ExecuteThread: '12' for queue: 'weblogic.kernel.Default (self-tuning)'" id=134 idx=0x208 tid=28802 prio=1 alive, parked, native_blocked, daemon
-- Parking to wait for: java/util/concurrent/locks/ReentrantLock$NonfairSync@0xb13cc448
at jrockit/vm/Locks.park0(J)V(Native Method)
at jrockit/vm/Locks.park(Locks.java:2221)[inlined]
at jrockit/proxy/sun/misc/Unsafe.park(Unsafe.java:592)[inlined]
at java/util/concurrent/locks/LockSupport.park(LockSupport.java:158)[inlined]
at java/util/concurrent/locks/AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747)[inlined]
at java/util/concurrent/locks/AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:778)[inlined]
at java/util/concurrent/locks/AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1114)[inlined]
at java/util/concurrent/locks/ReentrantLock$NonfairSync.lock(ReentrantLock.java:186)[optimized]
at java/util/concurrent/locks/ReentrantLock.lock(ReentrantLock.java:262)[inlined]
at oracle/adf/model/dcframe/DataControlFrameImpl.lock(DataControlFrameImpl.java:294)[optimized]
at oracle/adf/model/dcframe/DataControlFrameImpl.beginRequest(DataControlFrameImpl.java:336)[inlined]
at oracle/adf/model/BindingContext.setCurrentFrame(BindingContext.java:2107)[optimized]
at oracle/adf/model/BindingContext.setCurrentDataControlFrame(BindingContext.java:2009)[inlined]
at oracle/adfinternal/controller/util/model/DCFrameImpl.makeCurrent(DCFrameImpl.java:126)[inlined]
at oracle/adfinternal/controller/state/ViewPortContextImpl.makeCurrent(ViewPortContextImpl.java:1006)[optimized]
at oracle/adfinternal/controller/state/RequestState.setCurrentViewPortContext(RequestState.java:159)
at oracle/adfinternal/controller/state/ControllerState.setRequestState(ControllerState.java:900)[inlined]
at oracle/adfinternal/controller/state/ControllerState.synchronizeStatePart1(ControllerState.java:355)[optimized]
at oracle/adfinternal/controller/application/SyncNavigationStateListener.beforePhase(SyncNavigationStateListener.java:105)[optimized]
at oracle/adfinternal/controller/lifecycle/ADFLifecycleImpl$PagePhaseListenerWrapper.beforePhase(ADFLifecycleImpl.java:551)[optimized]
at oracle/adfinternal/controller/lifecycle/LifecycleImpl.internalDispatchBeforeEvent(LifecycleImpl.java:100)[inlined]
at oracle/adfinternal/controller/lifecycle/LifecycleImpl.dispatchBeforePagePhaseEvent(LifecycleImpl.java:147)[inlined]
at oracle/adfinternal/controller/faces/lifecycle/ADFPhaseListener$PhaseInvokerImpl.dispatchBeforePagePhaseEvent(ADFPhaseListener.java:112)[optimized]
at oracle/adfinternal/controller/faces/lifecycle/ADFPhaseListener.beforePhase(ADFPhaseListener.java:59)[optimized]
at oracle/adfinternal/controller/faces/lifecycle/ADFLifecyclePhaseListener.beforePhase(ADFLifecyclePhaseListener.java:44)
at oracle/adfinternal/view/faces/lifecycle/LifecycleImpl._executePhase(LifecycleImpl.java:258)[optimized]
at oracle/adfinternal/view/faces/lifecycle/LifecycleImpl.execute(LifecycleImpl.java:177)[optimized]
at javax/faces/webapp/FacesServlet.service(FacesServlet.java:265)[optimized]
at weblogic/servlet/internal/StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:227)[optimized]
at weblogic/servlet/internal/StubSecurityHelper.invokeServlet(StubSecurityHelper.java:125)[inlined]
at weblogic/servlet/internal/ServletStubImpl.execute(ServletStubImpl.java:300)[inlined]
at weblogic/servlet/internal/TailFilter.doFilter(TailFilter.java:26)[optimized]
at weblogic/servlet/internal/FilterChainImpl.doFilter(FilterChainImpl.java:56)[optimized]
at oracle/adf/model/servlet/ADFBindingFilter.doFilter(ADFBindingFilter.java:191)[optimized]
at weblogic/servlet/internal/FilterChainImpl.doFilter(FilterChainImpl.java:56)[optimized]
at oracle/adfinternal/view/faces/webapp/rich/RegistrationFilter.doFilter(RegistrationFilter.java:97)
at org/apache/myfaces/trinidadinternal/webapp/TrinidadFilterImpl$FilterListChain.doFilter(TrinidadFilterImpl.java:420)
at oracle/adfinternal/view/faces/activedata/AdsFilter.doFilter(AdsFilter.java:60)
at org/apache/myfaces/trinidadinternal/webapp/TrinidadFilterImpl$FilterListChain.doFilter(TrinidadFilterImpl.java:420)
at org/apache/myfaces/trinidadinternal/webapp/TrinidadFilterImpl._doFilterImpl(TrinidadFilterImpl.java:247)[inlined]
at org/apache/myfaces/trinidadinternal/webapp/TrinidadFilterImpl.doFilter(TrinidadFilterImpl.java:157)[optimized]
at org/apache/myfaces/trinidad/webapp/TrinidadFilter.doFilter(TrinidadFilter.java:92)[optimized]
at weblogic/servlet/internal/FilterChainImpl.doFilter(FilterChainImpl.java:56)[optimized]
at oracle/security/jps/ee/http/JpsAbsFilter$1.run(JpsAbsFilter.java:94)
at jrockit/vm/AccessController.doPrivileged(AccessController.java:254)[inlined]
at oracle/security/jps/util/JpsSubject.doAsPrivileged(JpsSubject.java:313)[inlined]
at oracle/security/jps/ee/util/JpsPlatformUtil.runJaasMode(JpsPlatformUtil.java:414)[inlined]
at oracle/security/jps/ee/http/JpsAbsFilter.doFilter(JpsAbsFilter.java:138)[optimized]
at oracle/security/jps/ee/http/JpsFilter.doFilter(JpsFilter.java:71)
at weblogic/servlet/internal/FilterChainImpl.doFilter(FilterChainImpl.java:56)[optimized]
at com/bea/content/manager/servlets/ContentServletFilter.doFilter(ContentServletFilter.java:178)
at weblogic/servlet/internal/FilterChainImpl.doFilter(FilterChainImpl.java:56)[optimized]
at oracle/adf/library/webapp/LibraryFilter.doFilter(LibraryFilter.java:159)[optimized]
at weblogic/servlet/internal/FilterChainImpl.doFilter(FilterChainImpl.java:56)[optimized]
at oracle/security/jps/ee/http/JpsAbsFilter$1.run(JpsAbsFilter.java:94)
at jrockit/vm/AccessController.doPrivileged(AccessController.java:254)[inlined]
at oracle/security/jps/util/JpsSubject.doAsPrivileged(JpsSubject.java:313)[inlined]
at oracle/security/jps/ee/util/JpsPlatformUtil.runJaasMode(JpsPlatformUtil.java:414)[inlined]
at oracle/security/jps/ee/http/JpsAbsFilter.doFilter(JpsAbsFilter.java:138)[optimized]
at oracle/security/jps/ee/http/JpsFilter.doFilter(JpsFilter.java:71)
at weblogic/servlet/internal/FilterChainImpl.doFilter(FilterChainImpl.java:56)[optimized]
at oracle/dms/wls/DMSServletFilter.doFilter(DMSServletFilter.java:330)[optimized]
at weblogic/servlet/internal/FilterChainImpl.doFilter(FilterChainImpl.java:56)[optimized]
at weblogic/servlet/internal/RequestEventsFilter.doFilter(RequestEventsFilter.java:27)[optimized]
at weblogic/servlet/internal/FilterChainImpl.doFilter(FilterChainImpl.java:56)[inlined]
at weblogic/servlet/internal/WebAppServletContext$ServletInvocationAction.doIt(WebAppServletContext.java:3684)[inlined]
at weblogic/servlet/internal/WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3650)[optimized]
at weblogic/security/acl/internal/AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)[optimized]
at weblogic/security/service/SecurityManager.runAs(SecurityManager.java:121)[inlined]
at weblogic/servlet/internal/WebAppServletContext.securedExecute(WebAppServletContext.java:2268)[inlined]
at weblogic/servlet/internal/WebAppServletContext.execute(WebAppServletContext.java:2174)[optimized]
at weblogic/servlet/internal/ServletRequestImpl.run(ServletRequestImpl.java:1446)[optimized]
at weblogic/work/ExecuteThread.execute(ExecuteThread.java:201)[inlined]
at weblogic/work/ExecuteThread.run(ExecuteThread.java:173)[optimized]
at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
-- end of trace
Circular (deadlocked) lock chains
=================================
Chain 2:
"[STUCK] ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)'" id=21 idx=0x58 tid=25256 waiting for java/util/concurrent/locks/ReentrantLock$NonfairSync@0x77543e08 held by:
"[STUCK] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'" id=59 idx=0xdc tid=25408 waiting for java/util/concurrent/locks/ReentrantLock$NonfairSync@0x6e2478f8 held by:
"[STUCK] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)'" id=88 idx=0x150 tid=25523 waiting for java/util/concurrent/locks/ReentrantLock$NonfairSync@0x77543998 held by:
"[STUCK] ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)'" id=21 idx=0x58 tid=25256
Blocked lock chains
===================
Chain 4:
"ExecuteThread: '2' for queue: 'weblogic.socket.Muxer'" id=26 idx=0x64 tid=25259 waiting for java/lang/String@0x7b64a308 held by:
"ExecuteThread: '0' for queue: 'weblogic.socket.Muxer'" id=24 idx=0x5c tid=25257 in chain 3
Chain 5:
"ExecuteThread: '3' for queue: 'weblogic.socket.Muxer'" id=27 idx=0x68 tid=25260 waiting for java/lang/String@0x7b64a308 held by:
"ExecuteThread: '0' for queue: 'weblogic.socket.Muxer'" id=24 idx=0x5c tid=25257 in chain 3
Chain 6:
"[ACTIVE] ExecuteThread: '16' for queue: 'weblogic.kernel.Default (self-tuning)'" id=139 idx=0x21c tid=28904 waiting for java/util/concurrent/locks/ReentrantLock$NonfairSync@0x8dab2718 held by:
"[STUCK] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'" id=60 idx=0xe0 tid=25410 waiting for java/util/concurrent/locks/ReentrantLock$NonfairSync@0x8dab2800 held by:
"[STUCK] ExecuteThread: '10' for queue: 'weblogic.kernel.Default (self-tuning)'" id=132 idx=0x200 tid=28726 waiting for java/util/concurrent/locks/ReentrantLock$NonfairSync@0x6e2dbf18 held by:
"[STUCK] ExecuteThread: '6' for queue: 'weblogic.kernel.Default (self-tuning)'" id=89 idx=0x154 tid=25524 waiting for java/util/concurrent/locks/ReentrantLock$NonfairSync@0x6e2dc198 held by:
"[STUCK] ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)'" id=21 idx=0x58 tid=25256 in chain 2
Chain 7:
"[ACTIVE] ExecuteThread: '21' for queue: 'weblogic.kernel.Default (self-tuning)'" id=144 idx=0x230 tid=28909 waiting for java/util/concurrent/locks/ReentrantLock$NonfairSync@0xb13cc218 held by:
"[STUCK] ExecuteThread: '12' for queue: 'weblogic.kernel.Default (self-tuning)'" id=134 idx=0x208 tid=28802 waiting for java/util/concurrent/locks/ReentrantLock$NonfairSync@0xb13cc448 held by:
"[ACTIVE] ExecuteThread: '8' for queue: 'weblogic.kernel.Default (self-tuning)'" id=91 idx=0x158 tid=25559 waiting for java/util/concurrent/locks/ReentrantLock$NonfairSync@0x6e247740 held by:
"[STUCK] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'" id=59 idx=0xdc tid=25408 in chain 2
Chain 8:
"[ACTIVE] ExecuteThread: '7' for queue: 'weblogic.kernel.Default (self-tuning)'" id=90 idx=0x15c tid=25558 waiting for java/util/concurrent/locks/ReentrantLock$NonfairSync@0xa25b8588 held by:
"[STUCK] ExecuteThread: '9' for queue: 'weblogic.kernel.Default (self-tuning)'" id=92 idx=0x160 tid=25566 waiting for java/util/concurrent/locks/ReentrantLock$NonfairSync@0xa25b8738 held by:
"[STUCK] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)'" id=88 idx=0x150 tid=25523 in chain 2
Chain 11:
"[ACTIVE] ExecuteThread: '18' for queue: 'weblogic.kernel.Default (self-tuning)'" id=141 idx=0x224 tid=28906 waiting for java/util/concurrent/locks/ReentrantLock$NonfairSync@0x37247b68 held by:
"[STUCK] ExecuteThread: '13' for queue: 'weblogic.kernel.Default (self-tuning)'" id=135 idx=0x20c tid=28803 waiting for java/util/concurrent/locks/ReentrantLock$NonfairSync@0x37247e90 held by:
"[ACTIVE] ExecuteThread: '8' for queue: 'weblogic.kernel.Default (self-tuning)'" id=91 idx=0x158 tid=25559 in chain 7
Chain 14:
"[ACTIVE] ExecuteThread: '17' for queue: 'weblogic.kernel.Default (self-tuning)'" id=140 idx=0x220 tid=28905 waiting for java/util/concurrent/locks/ReentrantLock$NonfairSync@0xaff95198 held by:
"[ACTIVE] ExecuteThread: '19' for queue: 'weblogic.kernel.Default (self-tuning)'" id=142 idx=0x228 tid=28907 in chain 9
Open lock chains
================
Chain 3:
"ExecuteThread: '1' for queue: 'weblogic.socket.Muxer'" id=25 idx=0x60 tid=25258 waiting for java/lang/String@0x7b64a308 held by:
"ExecuteThread: '0' for queue: 'weblogic.socket.Muxer'" id=24 idx=0x5c tid=25257 (active)
Chain 9:
"[ACTIVE] ExecuteThread: '15' for queue: 'weblogic.kernel.Default (self-tuning)'" id=138 idx=0x218 tid=28902 waiting for java/util/concurrent/locks/ReentrantLock$NonfairSync@0x7e26bdb8 held by:
"[ACTIVE] ExecuteThread: '11' for queue: 'weblogic.kernel.Default (self-tuning)'" id=133 idx=0x204 tid=28801 waiting for oracle/toplink/internal/indirection/BatchValueHolder@0x3c206fe0 held by:
"[ACTIVE] ExecuteThread: '19' for queue: 'weblogic.kernel.Default (self-tuning)'" id=142 idx=0x228 tid=28907 (active)
Chain 16:
"[ACTIVE] ExecuteThread: '20' for queue: 'weblogic.kernel.Default (self-tuning)'" id=143 idx=0x22c tid=28908 waiting for java/util/concurrent/locks/ReentrantLock$NonfairSync@0xb374d300 held by:
"[ACTIVE] ExecuteThread: '22' for queue: 'weblogic.kernel.Default (self-tuning)'" id=145 idx=0x234 tid=28910 waiting for java/util/concurrent/locks/ReentrantLock$NonfairSync@0xb374d4a8 held by:
"[ACTIVE] ExecuteThread: '14' for queue: 'weblogic.kernel.Default (self-tuning)'" id=137 idx=0x214 tid=28878 (waiting on notification)