Can anyone help me where to troubleshoot for this problem.
In the System test environment, we are having trouble as bam server fails after deadlock. and this is happing again and again.
below is the log. Please guide me for finding root cause of this issue.
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
<Critical> <WebLogicServer> <xxxxx.xxx.com> <bam_server1> <[STANDBY] ExecuteThread: '14' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1307671464267> <BEA-000394> <
DEADLOCK DETECTED:
==================
[deadlocked thread] pool-12-thread-30:
-------------------------------------
Thread 'pool-12-thread-30' is waiting to acquire lock 'oracle.bam.adc.security.authentication.Authenticator@3934351b' that is held by thread '[ACTIVE] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)''
Stack trace:
------------
oracle.bam.adc.security.authentication.Authenticator.synchronizeUser(Authenticator.java:1110)
oracle.bam.adc.security.authentication.Authenticator.setUserContext(Authenticator.java:788)
oracle.bam.adc.kernel.server.DataStoreServer.setUserContext(DataStoreServer.java:480)
oracle.bam.adc.ejb.BamAdcServerBean.interceptor(BamAdcServerBean.java:262)
sun.reflect.GeneratedMethodAccessor338.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:597)
com.bea.core.repackaged.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:310)
com.bea.core.repackaged.springframework.jee.intercept.JeeInterceptorInterceptor.invoke(JeeInterceptorInterceptor.java:69)
com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
com.bea.core.repackaged.springframework.jee.intercept.MethodInvocationInvocationContext.proceed(MethodInvocationInvocationContext.java:104)
oracle.security.jps.ee.ejb.JpsAbsInterceptor$1.run(JpsAbsInterceptor.java:88)
java.security.AccessController.doPrivileged(Native Method)
oracle.security.jps.util.JpsSubject.doAsPrivileged(JpsSubject.java:313)
oracle.security.jps.ee.util.JpsPlatformUtil.runJaasMode(JpsPlatformUtil.java:414)
oracle.security.jps.wls.JpsWeblogicEjbInterceptor.runJaasMode(JpsWeblogicEjbInterceptor.java:61)
oracle.security.jps.ee.ejb.JpsAbsInterceptor.intercept(JpsAbsInterceptor.java:106)
oracle.security.jps.ee.ejb.JpsInterceptor.intercept(JpsInterceptor.java:106)
sun.reflect.GeneratedMethodAccessor337.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:597)
com.bea.core.repackaged.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:310)
com.bea.core.repackaged.springframework.jee.intercept.JeeInterceptorInterceptor.invoke(JeeInterceptorInterceptor.java:69)
com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)
com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:119)
com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
com.bea.core.repackaged.springframework.jee.spi.MethodInvocationVisitorImpl.visit(MethodInvocationVisitorImpl.java:37)
weblogic.ejb.container.injection.EnvironmentInterceptorCallbackImpl.callback(EnvironmentInterceptorCallbackImpl.java:54)
com.bea.core.repackaged.springframework.jee.spi.EnvironmentInterceptor.invoke(EnvironmentInterceptor.java:50)
com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
com.bea.core.repackaged.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:89)
com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)
com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:119)
com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
com.bea.core.repackaged.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
$Proxy180.rollbackTransaction(Unknown Source)
oracle.bam.adc.ejb.BamAdcServerBean_wf34ei_BamAdcServerRemoteImpl.rollbackTransaction(BamAdcServerBean_wf34ei_BamAdcServerRemoteImpl.java:901)
oracle.bam.adc.api.client.BamAdcClient.rollbackTransaction(BamAdcClient.java:1220)
sun.reflect.GeneratedMethodAccessor1020.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:597)
oracle.bam.common.remoting.BamEjbProxy.invoke(BamEjbProxy.java:209)
$Proxy179.rollbackTransaction(Unknown Source)
oracle.bam.adc.api.util.Context.rollbackTransaction(Context.java:492)
oracle.bam.ems.engine.MessageHandler.handle(MessageHandler.java:820)
oracle.bam.ems.engine.JMSListener.onMessage(JMSListener.java:334)
oracle.bam.common.messaging.api.MessageConsumerRegistry.pushMessage(MessageConsumerRegistry.java:1140)
oracle.bam.common.messaging.api.MessageConsumerRegistry$Friend.pushMessage(MessageConsumerRegistry.java:1370)
oracle.bam.common.messaging.util.MessageConsumerRegistryQueuedPusher$Pusher.run(MessageConsumerRegistryQueuedPusher.java:756)
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
java.lang.Thread.run(Thread.java:619)
[deadlocked thread] [ACTIVE] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)':
--------------------------------------------------------------------------------------------------
Thread '[ACTIVE] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'' is waiting to acquire lock 'java.util.HashMap@128a9cf4' that is held by thread 'Viewset Expiration Thread'
Stack trace:
------------
oracle.bam.adc.dse.common.ListenerManager.getListeners(ListenerManager.java:56)
oracle.bam.adc.dse.common.StorageEngine.updateDataSetRow(StorageEngine.java:989)
oracle.bam.adc.dse.oracle.OracleStorageEngine.updateDataSetRow(OracleStorageEngine.java:1237)
oracle.bam.adc.kernel.datasets.DatasetBase.updateRow(DatasetBase.java:723)
oracle.bam.adc.kernel.datasets.DatasetBase.updateRow(DatasetBase.java:695)
oracle.bam.adc.kernel.datasets.SystemDataset.updateRowInternal(SystemDataset.java:200)
oracle.bam.adc.kernel.folders.FolderManager.processUpdateObject(FolderManager.java:289)
oracle.bam.adc.kernel.datasets.SystemDataset.updateRow(SystemDataset.java:189)
oracle.bam.adc.data.SystemObject.update(SystemObject.java:756)
oracle.bam.adc.data.SystemObject.save(SystemObject.java:435)
oracle.bam.adc.security.authentication.Authenticator.synchronizeJpsUser(Authenticator.java:1418)
oracle.bam.adc.security.authentication.Authenticator.synchronizeJps(Authenticator.java:1287)
oracle.bam.adc.security.authentication.Authenticator.synchronizeUser(Authenticator.java:1194)
oracle.bam.adc.security.authentication.Authenticator.setUserContext(Authenticator.java:788)
oracle.bam.adc.kernel.server.DataStoreServer.setUserContext(DataStoreServer.java:480)
oracle.bam.adc.ejb.BamAdcServerBean.interceptor(BamAdcServerBean.java:262)
sun.reflect.GeneratedMethodAccessor338.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:597)
com.bea.core.repackaged.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:310)
com.bea.core.repackaged.springframework.jee.intercept.JeeInterceptorInterceptor.invoke(JeeInterceptorInterceptor.java:69)
com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
com.bea.core.repackaged.springframework.jee.intercept.MethodInvocationInvocationContext.proceed(MethodInvocationInvocationContext.java:104)
oracle.security.jps.ee.ejb.JpsAbsInterceptor$1.run(JpsAbsInterceptor.java:88)
java.security.AccessController.doPrivileged(Native Method)
oracle.security.jps.util.JpsSubject.doAsPrivileged(JpsSubject.java:313)
oracle.security.jps.ee.util.JpsPlatformUtil.runJaasMode(JpsPlatformUtil.java:414)
oracle.security.jps.wls.JpsWeblogicEjbInterceptor.runJaasMode(JpsWeblogicEjbInterceptor.java:61)
oracle.security.jps.ee.ejb.JpsAbsInterceptor.intercept(JpsAbsInterceptor.java:106)
oracle.security.jps.ee.ejb.JpsInterceptor.intercept(JpsInterceptor.java:106)
sun.reflect.GeneratedMethodAccessor337.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:597)
com.bea.core.repackaged.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:310)
com.bea.core.repackaged.springframework.jee.intercept.JeeInterceptorInterceptor.invoke(JeeInterceptorInterceptor.java:69)
com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)
com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:119)
com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
com.bea.core.repackaged.springframework.jee.spi.MethodInvocationVisitorImpl.visit(MethodInvocationVisitorImpl.java:37)
weblogic.ejb.container.injection.EnvironmentInterceptorCallbackImpl.callback(EnvironmentInterceptorCallbackImpl.java:54)
com.bea.core.repackaged.springframework.jee.spi.EnvironmentInterceptor.invoke(EnvironmentInterceptor.java:50)
com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
com.bea.core.repackaged.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:89)
com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)
com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:119)
com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
com.bea.core.repackaged.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
$Proxy180.exists(Unknown Source)
oracle.bam.adc.ejb.BamAdcServerBean_wf34ei_BamAdcServerRemoteImpl.exists(BamAdcServerBean_wf34ei_BamAdcServerRemoteImpl.java:3011)
oracle.bam.adc.api.client.BamAdcClient.exists(BamAdcClient.java:599)
sun.reflect.GeneratedMethodAccessor339.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:597)
oracle.bam.common.remoting.BamEjbProxy.invoke(BamEjbProxy.java:209)
$Proxy179.exists(Unknown Source)
oracle.bam.adc.api.datasets.SystemType.exists(SystemType.java:299)
oracle.bam.adc.api.datasets.SystemType.loadSchema(SystemType.java:203)
oracle.bam.adc.api.datasets.Dataset.<init>(Dataset.java:415)
oracle.bam.adc.api.datasets.SystemType.<init>(SystemType.java:140)
oracle.bam.adc.api.datasets.SystemType.<init>(SystemType.java:113)
oracle.bam.adc.api.rowsets.Objectset.<init>(Objectset.java:65)
oracle.bam.adc.api.datasets.DatasetSchema.loadLookupInformation(DatasetSchema.java:696)
oracle.bam.adc.api.datasets.DatasetSchema.load(DatasetSchema.java:551)
oracle.bam.adc.api.datasets.DatasetSchema.get(DatasetSchema.java:385)
oracle.bam.adc.api.datasets.Dataset.loadSchema(Dataset.java:438)
oracle.bam.adc.api.datasets.Dataset.<init>(Dataset.java:415)
oracle.bam.adc.api.datasets.Dataset.<init>(Dataset.java:364)
oracle.bam.webservices.common.OperationImpl.getDataset(OperationImpl.java:599)
oracle.bam.webservices.common.OperationImpl.getDataset(OperationImpl.java:531)
oracle.bam.webservices.common.OperationImpl.upsert(OperationImpl.java:1135)
oracle.bam.webservices.dataobjectoperationsbyid.DataObjectOperationsByID.upsert(DataObjectOperationsByID.java:307)
sun.reflect.GeneratedMethodAccessor392.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:597)
weblogic.wsee.jaxws.WLSInstanceResolver$WLSInvoker.invoke(WLSInstanceResolver.java:92)
weblogic.wsee.jaxws.WLSInstanceResolver$WLSInvoker.invoke(WLSInstanceResolver.java:74)
com.sun.xml.ws.server.InvokerTube$2.invoke(InvokerTube.java:151)
com.sun.xml.ws.server.sei.EndpointMethodHandlerImpl.invoke(EndpointMethodHandlerImpl.java:265)
com.sun.xml.ws.server.sei.SEIInvokerTube.processRequest(SEIInvokerTube.java:100)
com.sun.xml.ws.api.pipe.Fiber.__doRun(Fiber.java:604)
com.sun.xml.ws.api.pipe.Fiber._doRun(Fiber.java:563)
com.sun.xml.ws.api.pipe.Fiber.doRun(Fiber.java:548)
com.sun.xml.ws.api.pipe.Fiber.runSync(Fiber.java:445)
com.sun.xml.ws.server.WSEndpointImpl$2.process(WSEndpointImpl.java:373)
com.sun.xml.ws.transport.http.HttpAdapter$HttpToolkit.handle(HttpAdapter.java:524)
com.sun.xml.ws.transport.http.HttpAdapter.handle(HttpAdapter.java:255)
com.sun.xml.ws.transport.http.servlet.ServletAdapter.handle(ServletAdapter.java:140)
weblogic.wsee.jaxws.WLSServletAdapter.handle(WLSServletAdapter.java:208)
weblogic.wsee.jaxws.HttpServletAdapter$AuthorizedInvoke.run(HttpServletAdapter.java:310)
weblogic.wsee.jaxws.HttpServletAdapter.post(HttpServletAdapter.java:223)
weblogic.wsee.jaxws.JAXWSServlet.doPost(JAXWSServlet.java:124)
javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
weblogic.wsee.jaxws.JAXWSServlet.service(JAXWSServlet.java:79)
javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:227)
weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:125)
weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:300)
weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:26)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
oracle.security.jps.ee.http.JpsAbsFilter$1.run(JpsAbsFilter.java:94)
java.security.AccessController.doPrivileged(Native Method)
oracle.security.jps.util.JpsSubject.doAsPrivileged(JpsSubject.java:313)
oracle.security.jps.ee.util.JpsPlatformUtil.runJaasMode(JpsPlatformUtil.java:414)
oracle.security.jps.ee.http.JpsAbsFilter.doFilter(JpsAbsFilter.java:138)
oracle.security.jps.ee.http.JpsFilter.doFilter(JpsFilter.java:71)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
oracle.dms.wls.DMSServletFilter.doFilter(DMSServletFilter.java:330)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.doIt(WebAppServletContext.java:3684)
weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3650)
weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
weblogic.security.service.SecurityManager.runAs(SecurityManager.java:121)
weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2268)
weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2174)
weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1446)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:201)
weblogic.work.ExecuteThread.run(ExecuteThread.java:173)
[deadlocked thread] Viewset Expiration Thread:
---------------------------------------------
Thread 'Viewset Expiration Thread' is waiting to acquire lock 'java.util.ArrayList@4a4660ae' that is held by thread '[ACTIVE] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)''
Stack trace:
------------
oracle.bam.adc.dse.common.ListenerManager.removeListener(ListenerManager.java:104)
oracle.bam.adc.dse.common.StorageEngine.unregisterForDataSetChangedEvent(StorageEngine.java:950)
oracle.bam.adc.kernel.viewsets.utilities.externaldata.TemporaryStorageManager.close(TemporaryStorageManager.java:246)
oracle.bam.adc.kernel.viewsets.Viewset.close(Viewset.java:175)
oracle.bam.adc.kernel.viewsets.ActiveViewset.close(ActiveViewset.java:518)
oracle.bam.adc.kernel.viewsets.PushViewset.close(PushViewset.java:405)
oracle.bam.adc.kernel.viewsets.ViewsetManager.closeViewset(ViewsetManager.java:622)
oracle.bam.adc.kernel.viewsets.ViewsetManager.expireViewsets(ViewsetManager.java:203)
oracle.bam.adc.kernel.viewsets.ViewsetManager.run(ViewsetManager.java:122)
java.lang.Thread.run(Thread.java:619)
[deadlocked thread] [ACTIVE] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)':
--------------------------------------------------------------------------------------------------
Thread '[ACTIVE] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)'' is waiting to acquire lock 'java.util.HashMap@128a9cf4' that is held by thread 'Viewset Expiration Thread'
Stack trace:
------------
oracle.bam.adc.dse.common.ListenerManager.getListeners(ListenerManager.java:56)
oracle.bam.adc.dse.common.StorageEngine.insertDataSetRow(StorageEngine.java:791)
oracle.bam.adc.dse.oracle.OracleStorageEngine.insertDataSetRow(OracleStorageEngine.java:1190)
oracle.bam.adc.kernel.viewsets.utilities.externaldata.ExternalDataManager.processDataSetChangedEvent(ExternalDataManager.java:351)
oracle.bam.adc.kernel.viewsets.utilities.externaldata.TemporaryStorageManager.handleDataSetChanged(TemporaryStorageManager.java:238)
sun.reflect.GeneratedMethodAccessor365.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:597)
oracle.bam.system.Delegate$Sink.invoke(Delegate.java:80)
--
---
---
---
weblogic.work.ExecuteThread.run(ExecuteThread.java:173)
>
<Critical> <Health> <xxxxx.xxx.com> <bam_server1> <[ACTIVE] ExecuteThread: '6' for queue: 'weblogic.kernel.Default (self-tuning)'
<<WLS Kernel>> <> <> <1307671464288> <BEA-310006> <Critical Subsystem core has failed. Setting server state to FAILED.
Reason: Thread deadlock detected>
<Critical> <WebLogicServer> <xxxxx.xxx.com> <bam_server1> <[ACTIVE] ExecuteThread: '6' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1307671464292> <BEA-000385> <Server health failed. Reason: health of critical service 'core' failed>
<Notice> <WebLogicServer> <xxxxx.xxx.com> <bam_server1> <[ACTIVE] ExecuteThread: '6' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1307671464295> <BEA-000365> <Server state changed to FAILED>
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------