Integration

Announcement

For appeals, questions and feedback about Oracle Forums, please email oracle-forums-moderators_us@oracle.com. Technical questions should be asked in the appropriate category. Thank you!

Several STUCK threads on app side. Need to restart coherence cluster.

user7584423Jan 24 2017 — edited Jan 27 2017

Hello,

This is on an installation using coherence 12c. A coherence cluster with two nodes and basic configuration, also app cluster with basic config, and our application.

After some days, we see some threads on STUCK status and our app stops giving service. Our solution is to restart the coherence cluster. We see a lot of entries on our app logs like that:

<Error> <WebLogicServer> <BEA-000337> <[STUCK] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "601" seconds working on the request "Workmanager: default, Version: 0, Scheduled=false, Started=true, Started time: 601614 ms

", which is more than the configured time (StuckThreadMaxTime) of "600" seconds in "server-failure-trigger". Stack trace:

sun.misc.Unsafe.park(Native Method)

java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

com.oracle.common.base.Blocking.parkNanos(Blocking.java:150)

com.oracle.common.base.Blocking.park(Blocking.java:134)

com.oracle.common.base.SingleWaiterMultiNotifier.await(SingleWaiterMultiNotifier.java:54)

com.oracle.common.base.SingleWaiterCooperativeNotifier.await(SingleWaiterCooperativeNotifier.java:49)

com.tangosol.coherence.component.net.Poll.waitCompletion(Poll.CDB:6)

com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.poll(Grid.CDB:34)

com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.poll(Grid.CDB:1)

com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache$BinaryMap.sendPartitionedRequest(PartitionedCache.CDB:59)

com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache$BinaryMap.aggregate(PartitionedCache.CDB:17)

com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache$ViewMap.aggregatePart(PartitionedCache.CDB:23)

com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache$ViewMap.aggregateStreaming(PartitionedCache.CDB:61)

com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache$ViewMap.aggregate(PartitionedCache.CDB:65)

com.tangosol.coherence.component.util.SafeNamedCache.aggregate(SafeNamedCache.CDB:1)

com.tangosol.internal.util.stream.ReferencePipeline.lambda$iterator$114(ReferencePipeline.java:234)

com.tangosol.internal.util.stream.ReferencePipeline$$Lambda$12/1217286868.apply(Unknown Source)

com.tangosol.internal.util.stream.ReferencePipeline$PartitionedIterator.nextIterable(ReferencePipeline.java:507)

com.tangosol.net.partition.AbstractPartitionedIterator.advance(AbstractPartitionedIterator.java:85)

com.oracle.common.collections.AbstractStableIterator.hasNext(AbstractStableIterator.java:41)

java.util.Iterator.forEachRemaining(Iterator.java:115)

com.tangosol.internal.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:249)

es.ree.realtime.gemas.coherence.dao.AbstractFacade.listCreator(AbstractFacade.java:106)

es.ree.realtime.gemas.coherence.dao.AbstractFacade.findAll(AbstractFacade.java:63)

sun.reflect.GeneratedMethodAccessor260.invoke(Unknown Source)

sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

java.lang.reflect.Method.invoke(Method.java:498)

com.bea.core.repackaged.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:310)

com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:182)

com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:149)

com.oracle.pitchfork.intercept.MethodInvocationInvocationContext.proceed(MethodInvocationInvocationContext.java:101)

com.oracle.pitchfork.intercept.JeeInterceptorInterceptor.invoke(JeeInterceptorInterceptor.java:101)

com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)

com.oracle.pitchfork.intercept.MethodInvocationInvocationContext.proceed(MethodInvocationInvocationContext.java:101)

org.jboss.weld.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:64)

org.jboss.weld.ejb.SessionBeanInterceptor.aroundInvoke(SessionBeanInterceptor.java:52)

sun.reflect.GeneratedMethodAccessor173.invoke(Unknown Source)

sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

java.lang.reflect.Method.invoke(Method.java:498)

com.oracle.pitchfork.intercept.JeeInterceptorInterceptor.invoke(JeeInterceptorInterceptor.java:94)

com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)

com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)

com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:119)

com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)

com.bea.core.repackaged.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)

com.sun.proxy.$Proxy236.findAll(Unknown Source)

es.ree.realtime.gemas.coherence.dao.RpcAnomalyDataFacade_leyk90_NoIntfViewImpl.__WL_invoke(Unknown Source)

weblogic.ejb.container.internal.SessionLocalMethodInvoker.invoke(SessionLocalMethodInvoker.java:33)

es.ree.realtime.gemas.coherence.dao.RpcAnomalyDataFacade_leyk90_NoIntfViewImpl.findAll(Unknown Source)

es.ree.realtime.gemas.timer.AnomalyUpdatedTimer.refreshAnomalies(AnomalyUpdatedTimer.java:215)

sun.reflect.GeneratedMethodAccessor222.invoke(Unknown Source)

sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

java.lang.reflect.Method.invoke(Method.java:498)

com.oracle.pitchfork.inject.Jsr250Metadata.invokeTimeoutMethodInternal(Jsr250Metadata.java:372)

com.oracle.pitchfork.intercept.LifecycleEventCallbackInvocationContext.proceed(LifecycleEventCallbackInvocationContext.java:126)

org.jboss.weld.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:73)

org.jboss.weld.ejb.SessionBeanInterceptor.aroundInvoke(SessionBeanInterceptor.java:52)

sun.reflect.GeneratedMethodAccessor173.invoke(Unknown Source)

sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

java.lang.reflect.Method.invoke(Method.java:498)

com.oracle.pitchfork.inject.Jsr250Metadata.invokeLifecycleMethod(Jsr250Metadata.java:387)

com.oracle.pitchfork.intercept.LifecycleEventCallbackInvocationContext.proceed(LifecycleEventCallbackInvocationContext.java:121)

com.oracle.pitchfork.intercept.LifecycleEventCallbackInvocationContext.proceed(LifecycleEventCallbackInvocationContext.java:150)

com.oracle.pitchfork.intercept.InterceptionMetadata.invokeTimeoutMethod(InterceptionMetadata.java:610)

weblogic.ejb.container.injection.EjbComponentCreatorImpl.invokeTimer(EjbComponentCreatorImpl.java:71)

weblogic.ejb.container.injection.InjectionBasedEjbComponentCreator.invokeTimer(InjectionBasedEjbComponentCreator.java:156)

weblogic.ejb.container.manager.BaseEJBManager.invokeTimeoutMethod(BaseEJBManager.java:186)

weblogic.ejb.container.timer.TimerImpl.timerExpired(TimerImpl.java:338)

weblogic.timers.internal.TimerImpl.run(TimerImpl.java:290)

weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:643)

weblogic.invocation.ComponentInvocationContextManager._runAs(ComponentInvocationContextManager.java:348)

weblogic.invocation.ComponentInvocationContextManager.runAs(ComponentInvocationContextManager.java:333)

weblogic.work.LivePartitionUtility.doRunWorkUnderContext(LivePartitionUtility.java:54)

weblogic.work.PartitionUtility.runWorkUnderContext(PartitionUtility.java:41)

weblogic.work.SelfTuningWorkManagerImpl.runWorkUnderContext(SelfTuningWorkManagerImpl.java:617)

weblogic.work.ExecuteThread.execute(ExecuteThread.java:397)

weblogic.work.ExecuteThread.run(ExecuteThread.java:346)

<Error> <WebLogicServer> <BEA-000337> <[STUCK] ExecuteThread: '20' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "609" seconds working on the request "Workmanager: default, Version: 0, Scheduled=false, Started=true, Started time: 609865 ms

", which is more than the configured time (StuckThreadMaxTime) of "600" seconds in "server-failure-trigger". Stack trace:

sun.misc.Unsafe.park(Native Method)

java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

com.oracle.common.base.Blocking.parkNanos(Blocking.java:150)

com.oracle.common.base.Blocking.park(Blocking.java:134)

com.oracle.common.base.SingleWaiterMultiNotifier.await(SingleWaiterMultiNotifier.java:54)

com.oracle.common.base.SingleWaiterCooperativeNotifier.await(SingleWaiterCooperativeNotifier.java:49)

com.tangosol.coherence.component.net.Poll.waitCompletion(Poll.CDB:6)

com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.poll(Grid.CDB:34)

com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.poll(Grid.CDB:1)

com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache$BinaryMap.lock(PartitionedCache.CDB:36)

com.tangosol.util.ConverterCollections$ConverterConcurrentMap.lock(ConverterCollections.java:494)

com.tangosol.util.ConverterCollections$ConverterNamedCache.lock(ConverterCollections.java:1031)

com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache$ViewMap.lock(PartitionedCache.CDB:1)

com.tangosol.coherence.component.util.SafeNamedCache.lock(SafeNamedCache.CDB:1)

es.ree.realtime.gemas.jms.InputRawWriteMessageBean.saveToCache(InputRawWriteMessageBean.java:127)

es.ree.realtime.gemas.jms.InputRawWriteMessageBean.write(InputRawWriteMessageBean.java:146)

es.ree.realtime.gemas.jms.InputRawWriteMessageBean.onMessage(InputRawWriteMessageBean.java:109)

sun.reflect.GeneratedMethodAccessor350.invoke(Unknown Source)