Hi
From the logs it looks like the extend client did not recover due to some reason, the client runs on websphere.
Following are the logs from the client:
2013-10-22 12:54:55,025 DEBUG [ner : 17] WebEventsController: GetNewEvents : 44483
2013-10-22 12:54:55,147 WARN [thTiFiMg] ThreadTimeoutFilterManager: Delayed Thread ? for WebContainer : 0
.... java.lang.Object.wait(Native Method)
.... com.tangosol.coherence.component.net.extend.message.Request$Status.waitForResponse(Request.CDB:44)
.... com.tangosol.coherence.component.net.extend.Channel.request(Channel.CDB:20)
.... com.tangosol.coherence.component.net.extend.Channel.request(Channel.CDB:1)
.... com.tangosol.coherence.component.net.extend.RemoteNamedCache$BinaryCache.aggregate(RemoteNamedCache.CDB:12)
.... com.tangosol.util.ConverterCollections$ConverterInvocableMap.aggregate(ConverterCollections.java:2274)
.... com.tangosol.util.ConverterCollections$ConverterNamedCache.aggregate(ConverterCollections.java:2781)
.... com.tangosol.coherence.component.net.extend.RemoteNamedCache.aggregate(RemoteNamedCache.CDB:1)
.... com.tangosol.coherence.component.util.SafeNamedCache.aggregate(SafeNamedCache.CDB:1)
.... com.tangosol.net.cache.NearCache.aggregate(NearCache.java:472)
.... testerpack.set.eventgateway.impl.search.ChorusCacheServiceImpl.search(ChorusCacheServiceImpl.java:383)
.... testerpack.set.eventgateway.web.impl.ChorusSessionManagerImpl.searchChorus(ChorusSessionManagerImpl.java:67)
.... testerpack.set.eventgateway.web.controller.ChorusController.loadTrades(ChorusController.java:134)
.... sun.reflect.GeneratedMethodAccessor609.invoke(Unknown Source)
.... sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
.... java.lang.reflect.Method.invoke(Method.java:597)
.... org.springframework.web.method.support.InvocableHandlerMethod.invoke(InvocableHandlerMethod.java:212)
.... org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:126)
.... org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:96)
.... org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:617)
.... org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:578)
.... org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:80)
.... org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:900)
.... org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:827)
.... org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:882)
.... org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:789)
.... javax.servlet.http.HttpServlet.service(HttpServlet.java:738)
.... javax.servlet.http.HttpServlet.service(HttpServlet.java:831)
.... com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1657)
.... com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1597)
.... com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:131)
.... testerpack.servlet.ajax.filter.SecurityTokenFilter.doFilter(SecurityTokenFilter.java:160)
.... com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:188)
.... com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:116)
.... testerpack.servlet.ajax.filter.PostTrailerFilter.doFilter(PostTrailerFilter.java:63)
.... com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:188)
.... com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:116)
.... testerpack.servlet.ajax.filter.ThreadTimeoutFilter.doFilter(ThreadTimeoutFilter.java:49)
.... com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:188)
.... com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:116)
.... com.ibm.ws.webcontainer.filter.WebAppFilterChain._doFilter(WebAppFilterChain.java:77)
.... com.ibm.ws.webcontainer.filter.WebAppFilterManager.doFilter(WebAppFilterManager.java:908)
.... com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:934)
.... com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:502)
.... com.ibm.ws.webcontainer.servlet.ServletWrapperImpl.handleRequest(ServletWrapperImpl.java:181)
.... com.ibm.ws.webcontainer.servlet.CacheServletWrapper.handleRequest(CacheServletWrapper.java:91)
.... com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:864)
.... com.ibm.ws.webcontainer.WSWebContainer.handleRequest(WSWebContainer.java:1592)
.... com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java:186)
.... com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:452)
.... com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewRequest(HttpInboundLink.java:511)
.... com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.processRequest(HttpInboundLink.java:305)
.... com.ibm.ws.http.channel.inbound.impl.HttpICLReadCallback.complete(HttpICLReadCallback.java:83)
.... com.ibm.ws.tcp.channel.impl.WorkQueueManager.requestComplete(WorkQueueManager.java:557)
.... com.ibm.ws.tcp.channel.impl.WorkQueueManager.attemptIO(WorkQueueManager.java:607)
.... com.ibm.ws.tcp.channel.impl.WorkQueueManager.workerRun(WorkQueueManager.java:984)
.... com.ibm.ws.tcp.channel.impl.WorkQueueManager$Worker.run(WorkQueueManager.java:1069)
.... com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1646)
2013-10-22 12:54:55,224 DEBUG [ner : 14] WebEventsController: GetNewEvents : 15153
2013-10-22 12:54:55,310 DEBUG [ner : 27] WebEventsController: GetNewEvents : 28368
2013-10-22 12:54:55,483 DEBUG [ner : 23] WebEventsController: GetNewEvents : 61628
2013-10-22 12:54:55,660 DEBUG [ner : 26] WebEventsController: GetNewEvents : 95366
2013-10-22 12:54:55,684 DEBUG [ner : 22] WebEventsController: GetNewEvents : 10776
2013-10-22 12:54:56,004 DEBUG [iner : 2] WebEventsController: GetNewEvents : 47801
2013-10-22 12:54:56,207 DEBUG [ner : 25] WebEventsController: GetNewEvents : 388
2013-10-22 12:54:56,437 DEBUG [ner : 17] WebEventsController: GetNewEvents : 35509
2013-10-22 12:54:56,502 DEBUG [ner : 14] WebEventsController: GetNewEvents : 48807
2013-10-22 12:54:56,738 DEBUG [ner : 27] WebEventsController: GetNewEvents : 97692
2013-10-22 12:54:56,878 DEBUG [ner : 26] WebEventsController: GetNewEvents : 95317
2013-10-22 12:54:56,909 DEBUG [ner : 22] WebEventsController: GetNewEvents : 40687
2013-10-22 12:54:57,080 DEBUG [iner : 2] WebEventsController: GetNewEvents : 40692
2013-10-22 12:54:57,110 DEBUG [ner : 21] WebEventsController: GetNewEvents : 97487
2013-10-22 12:54:57,530 DEBUG [ner : 17] WebEventsController: GetNewEvents : 42191
2013-10-22 12:54:57,620 DEBUG [ner : 14] WebEventsController: GetNewEvents : 104758
2013-10-22 12:54:58,024 DEBUG [ner : 27] WebEventsController: GetNewEvents : 44483
2013-10-22 12:54:58,231 DEBUG [ner : 23] WebEventsController: GetNewEvents : 15153
2013-10-22 12:54:58,245 WARN [thTiFiMg] ThreadTimeoutFilterManager: Delayed Thread ? for WebContainer : 20
.... java.lang.Object.wait(Native Method)
.... com.tangosol.coherence.component.net.extend.message.Request$Status.waitForResponse(Request.CDB:44)
.... com.tangosol.coherence.component.net.extend.Channel.request(Channel.CDB:20)
.... com.tangosol.coherence.component.net.extend.Channel.request(Channel.CDB:1)
.... com.tangosol.coherence.component.net.extend.RemoteNamedCache$BinaryCache.aggregate(RemoteNamedCache.CDB:12)
.... com.tangosol.util.ConverterCollections$ConverterInvocableMap.aggregate(ConverterCollections.java:2274)
.... com.tangosol.util.ConverterCollections$ConverterNamedCache.aggregate(ConverterCollections.java:2781)
.... com.tangosol.coherence.component.net.extend.RemoteNamedCache.aggregate(RemoteNamedCache.CDB:1)
.... com.tangosol.coherence.component.util.SafeNamedCache.aggregate(SafeNamedCache.CDB:1)
.... com.tangosol.net.cache.NearCache.aggregate(NearCache.java:472)
.... testerpack.set.eventgateway.impl.search.ChorusCacheServiceImpl.search(ChorusCacheServiceImpl.java:383)
.... testerpack.set.eventgateway.web.impl.ChorusSessionManagerImpl.searchChorus(ChorusSessionManagerImpl.java:67)
.... testerpack.set.eventgateway.web.controller.ChorusController.loadTrades(ChorusController.java:134)
.... sun.reflect.GeneratedMethodAccessor609.invoke(Unknown Source)
.... sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
2013-10-22 12:54:58,320 DEBUG [ner : 26] WebEventsController: GetNewEvents : 28368
2013-10-22 12:54:58,463 DEBUG [ner : 22] WebEventsController: GetNewEvents : 61628
.... java.lang.Object.wait(Native Method)
.... com.tangosol.coherence.component.net.extend.message.Request$Status.waitForResponse(Request.CDB:44)
.... com.tangosol.coherence.component.net.extend.Channel.request(Channel.CDB:20)
.... com.tangosol.coherence.component.net.extend.Channel.request(Channel.CDB:1)
.... com.tangosol.coherence.component.net.extend.RemoteNamedCache$BinaryCache.aggregate(RemoteNamedCache.CDB:12)
.... com.tangosol.util.ConverterCollections$ConverterInvocableMap.aggregate(ConverterCollections.java:2274)
.... com.tangosol.util.ConverterCollections$ConverterNamedCache.aggregate(ConverterCollections.java:2781)
.... com.tangosol.coherence.component.net.extend.RemoteNamedCache.aggregate(RemoteNamedCache.CDB:1)
.... com.tangosol.coherence.component.util.SafeNamedCache.aggregate(SafeNamedCache.CDB:1)
.... com.tangosol.net.cache.NearCache.aggregate(NearCache.java:472)
.... testerpack.set.eventgateway.impl.search.ChorusCacheServiceImpl.search(ChorusCacheServiceImpl.java:383)
.... testerpack.set.eventgateway.web.impl.ChorusSessionManagerImpl.searchChorus(ChorusSessionManagerImpl.java:67)
.... testerpack.set.eventgateway.web.controller.ChorusController.loadTrades(ChorusController.java:134)
.... sun.reflect.GeneratedMethodAccessor609.invoke(Unknown Source)
.... sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
.... java.lang.reflect.Method.invoke(Method.java:597)
2013-10-22 12:54:59,226 DEBUG [ner : 27] WebEventsController: GetNewEvents : 388
2013-10-22 12:54:59,459 DEBUG [ner : 23] WebEventsController: GetNewEvents : 35509
2013-10-22 12:54:59,523 DEBUG [ner : 26] WebEventsController: GetNewEvents : 48807
10-22 12:55:58,954 INFO [iner : 0] BlotterEventDispatcher: Dispatching blotter event [TradeId=107085,STATUS=VALIDATED] for user 84558
2013-10-22 12:55:58,954 INFO [iner : 0] BlotterEventDispatcher: Dispatching blotter event [TradeId=107121,STATUS=VALIDATED] for user 84558
2013-10-22 12:55:58,954 INFO [iner : 0] BlotterEventDispatcher: Dispatching blotter event [TradeId=107106,STATUS=STALE_PRICE] for user 84558
2013-10-22 12:55:58,954 INFO [iner : 0] BlotterEventDispatcher: Dispatching blotter event [TradeId=107179,STATUS=VALIDATED] for user 84558
2013-10-22 12:55:58,955 INFO [iner : 0] BlotterEventDispatcher: Dispatching blotter event [TradeId=107218,STATUS=VALIDATED] for user 84558
2013-10-22 12:55:59,194 WARN [thTiFiMg] ThreadTimeoutFilterManager: Delayed Thread ? for WebContainer : 2... java.lang.Object.wait(Native Method)
.... com.tangosol.coherence.component.net.extend.message.Request$Status.waitForResponse(Request.CDB:44)
.... com.tangosol.coherence.component.net.extend.message.Request$Status.waitForResponse(Request.CDB:1)
.... com.tangosol.coherence.component.net.extend.RemoteNamedCache$BinaryCache.query(RemoteNamedCache.CDB:36)
.... com.tangosol.coherence.component.net.extend.RemoteNamedCache$BinaryCache.entrySet(RemoteNamedCache.CDB:6)
.... com.tangosol.util.ConverterCollections$ConverterQueryMap.entrySet(ConverterCollections.java:2508)
.... com.tangosol.util.ConverterCollections$ConverterNamedCache.entrySet(ConverterCollections.java:2800)
.... com.tangosol.coherence.component.net.extend.RemoteNamedCache.entrySet(RemoteNamedCache.CDB:22)
.... com.tangosol.coherence.component.util.SafeNamedCache.entrySet(SafeNamedCache.CDB:1)
.... com.tangosol.net.cache.NearCache.entrySet(NearCache.java:399)
.... testerpack.set.eventgateway.impl.search.ChorusCacheServiceImpl.search(ChorusCacheServiceImpl.java:396)
.... testerpack.set.eventgateway.web.impl.ChorusSessionManagerImpl.searchChorus(ChorusSessionManagerImpl.java:67)
.... testerpack.set.eventgateway.web.controller.ChorusController.loadTrades(ChorusController.java:134)
.... sun.reflect.GeneratedMethodAccessor609.invoke(Unknown Source)
2013-10-22 12:55:59,206 INFO [thTiFiMg] ThreadTimeoutFilterManager: "WebContainer : 28" daemon prio=5 Thread id=1537 BLOCKED
.... WebContainer : 28 testerpack.set.eventgateway.web.event.dispatchers.BlotterEventDispatcher.refresh(BlotterEventDispatcher.java:157)
.... WebContainer : 28 testerpack.set.eventgateway.web.impl.WebSessionManager.refresh(WebSessionManager.java:353)
.... WebContainer : 28 testerpack.set.eventgateway.web.controller.HomeController.handleRequest(HomeController.java:126)
.... WebContainer : 28 sun.reflect.GeneratedMethodAccessor651.invoke(Unknown Source)
.... WebContainer : 28 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
.... WebContainer : 28 java.lang.reflect.Method.invoke(Method.java:597)
2013-10-22 12:55:59,206 INFO [thTiFiMg] ThreadTimeoutFilterManager: "ExtendTcpProxyService:TcpInitiator" daemon prio=10 Thread id=110 BLOCKED
.... ExtendTcpProxyService:TcpInitiator java.lang.Object.wait(Native Method)
.... ExtendTcpProxyService:TcpInitiator com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:18)
.... ExtendTcpProxyService:TcpInitiator com.tangosol.coherence.component.util.daemon.queueProcessor.Service.onWait(Service.CDB:4)
.... ExtendTcpProxyService:TcpInitiator com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:39)
.... ExtendTcpProxyService:TcpInitiator java.lang.Thread.run(Thread.java:662)
2013-10-22 12:55:59,420 DEBUG [ner : 32] WebEventsController: GetNewEvents : 47801
2013-10-22 12:55:59,448 DEBUG [ner : 33] WebEventsController: GetNewEvents : 61628
2013-10-22 12:55:59,556 DEBUG [ner : 34] WebEventsController: GetNewEvents : 48807
2013-10-22 12:55:59,598 DEBUG [ner : 35] WebEventsController: GetNewEvents : 388
2013-10-22 12:56:48,720 WARN [thTiFiMg] ThreadTimeoutFilterManager: Delayed Thread ? for WebContainer : 48
.... sun.misc.Unsafe.park(Native Method)
.... java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
.... java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)
.... java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:941)
.... java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1261)
.... java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:594)
.... testerpack.set.eventgateway.web.impl.WebSessionManager.getActiveDispatchersForSession(WebSessionManager.java:367)
.... testerpack.set.eventgateway.web.impl.MarketWatchSessionManagerImpl.marketWatchSearch(MarketWatchSessionManagerImpl.java:61)
.... testerpack.set.eventgateway.web.controller.MarketWatchController.loadProducts(MarketWatchController.java:240)
.... sun.reflect.GeneratedMethodAccessor619.invoke(Unknown Source)
.... sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
.... java.lang.reflect.Method.invoke(Method.java:597)
2013-10-22 12:56:50,252 INFO [thTiFiMg] ThreadTimeoutFilterManager: "Bus-Remote-__23" daemon prio=5 Thread id=718 BLOCKED
.... Bus-Remote-__23 java.lang.Object.wait(Native Method)
.... Bus-Remote-__23 java.lang.Object.wait(Object.java:485)
.... Bus-Remote-__23 com.hsoft.util.HSKeyFifo.pop(HSKeyFifo.java:162)
.... Bus-Remote-__23 com.hsoft.util.HSKeyFifo.popChunk(HSKeyFifo.java:202)
.... Bus-Remote-__23 com.hsoft.util.TaskForce$Worker.run(TaskForce.java:237)
2013-10-22 12:56:50,261 INFO [thTiFiMg] ThreadTimeoutFilterManager: "TF-session-7" daemon prio=5 Thread id=1330 BLOCKED
.... TF-session-7 java.lang.Object.wait(Native Method)
.... TF-session-7 java.lang.Object.wait(Object.java:485)
.... TF-session-7 com.hsoft.util.HSKeyFifo.pop(HSKeyFifo.java:162)
.... TF-session-7 com.hsoft.util.TaskForce$Worker.run(TaskForce.java:219)
2013-10-22 12:56:50,261 INFO [thTiFiMg] ThreadTimeoutFilterManager: "TF-session-9" daemon prio=5 Thread id=1399 BLOCKED
.... TF-session-9 java.lang.Object.wait(Native Method)
.... TF-session-9 java.lang.Object.wait(Object.java:485)
.... TF-session-9 com.hsoft.util.HSKeyFifo.pop(HSKeyFifo.java:162)
.... TF-session-9 com.hsoft.util.TaskForce$Worker.run(TaskForce.java:219)
2013-10-22 12:56:50,261 INFO [thTiFiMg] ThreadTimeoutFilterManager: "ExtendTcpProxyService:TcpInitiator" daemon prio=10 Thread id=110 BLOCKED
.... ExtendTcpProxyService:TcpInitiator java.lang.Object.wait(Native Method)
.... ExtendTcpProxyService:TcpInitiator com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:18)
.... ExtendTcpProxyService:TcpInitiator com.tangosol.coherence.component.util.daemon.queueProcessor.Service.onWait(Service.CDB:4)
.... ExtendTcpProxyService:TcpInitiator com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:39)
.... ExtendTcpProxyService:TcpInitiator java.lang.Thread.run(Thread.java:662)
2013-10-22 12:56:53,941 WARN [thTiFiMg] ThreadTimeoutFilterManager: Delayed Thread ? for WebContainer : 49
.... sun.misc.Unsafe.park(Native Method)
.... java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
.... java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)
.... java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:941)
.... java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1261)
.... java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:594)
.... testerpack.set.eventgateway.web.impl.WebSessionManager.refresh(WebSessionManager.java:345)
.... testerpack.set.eventgateway.web.controller.HomeController.handleRequest(HomeController.java:126)
.... sun.reflect.GeneratedMethodAccessor651.invoke(Unknown Source)
.... sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
2013-10-22 12:56:54,975 WARN [thTiFiMg] ThreadTimeoutFilterManager: Delayed Thread ? for WebContainer : 50
.... sun.misc.Unsafe.park(Native Method)
.... java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
.... java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)
.... java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:941)
.... java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1261)
.... java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:594)
.... testerpack.set.eventgateway.web.impl.WebSessionManager.getNewEvents(WebSessionManager.java:204)
.... testerpack.set.eventgateway.web.controller.WebEventsController.getNewEvents(WebEventsController.java:62)
.... testerpack.set.eventgateway.web.controller.WebEventsController.getNewEvents(WebEventsController.java:76)
.... sun.reflect.GeneratedMethodAccessor231.invoke(Unknown Source)
2013-10-22 12:56:55,233 WARN [thTiFiMg] ThreadTimeoutFilterManager: Delayed Thread ? for WebContainer : 51
.... sun.misc.Unsafe.park(Native Method)
.... java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
.... java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)
.... java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:941)
.... java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1261)
.... java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:594)
.... testerpack.set.eventgateway.web.impl.WebSessionManager.getNewEvents(WebSessionManager.java:204)
.... testerpack.set.eventgateway.web.controller.WebEventsController.getNewEvents(WebEventsController.java:62)
.... testerpack.set.eventgateway.web.controller.WebEventsController.getNewEvents(WebEventsController.java:76)
.... sun.reflect.GeneratedMethodAccessor231.invoke(Unknown Source)
.... sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
The following are the logs from the cache server:
Around the same time the cache server logs had the following:
DEBUG [2013-10-22 12:54:17,984] b.s.cache.loader.ChorusBlotterLoader - Storing the entity class testerpack.set.ejb.model.blotter.fortis.ChorusBlotterDb with id 7580
DEBUG [2013-10-22 12:54:17,984] b.s.cache.loader.ChorusBlotterLoader - Storing the entity class testerpack.set.ejb.model.blotter.fortis.ChorusBlotterDb with id 7580
DEBUG [2013-10-22 12:54:17,985] b.s.cache.loader.ChorusBlotterLoader - isSkip Store is true hence not storing
DEBUG [2013-10-22 12:54:17,985] b.s.cache.loader.ChorusBlotterLoader - isSkip Store is true hence not storing
INFO [2013-10-22 12:54:19,995] Coherence - 2013-10-22 12:54:19.995/98189.315 Oracle Coherence GE 3.7.1.9 <Info> (thread=DistributedCache, member=2): Extend*TCP has determined that TcpConnection(Id=0x00000141DA2A9EE80A5E50118F07DB0A1503A9B68F5689CECFAB139E1E60D093, Open=true, Member(Id=0, Timestamp=2013-10-20 12:50:05.005, Address=10.185.8.112:0, MachineId=0, Location=site:,machine:parss1112531,process:17241), LocalAddress=10.94.80.17:10495, RemoteAddress=10.185.8.112:63066) is no longer a suspect: The connection has reduced its backlog to 218259 bytes; the target was 2000000 bytes.
INFO [2013-10-22 12:54:58,914] Coherence - 2013-10-22 12:54:58.913/98228.233 Oracle Coherence GE 3.7.1.9 <Info> (thread=DistributedCache, member=2): Extend*TCP has marked TcpConnection(Id=0x00000141DA2A9EE80A5E50118F07DB0A1503A9B68F5689CECFAB139E1E60D093, Open=true, Member(Id=0, Timestamp=2013-10-20 12:50:05.005, Address=10.185.8.112:0, MachineId=0, Location=site:,machine:parss1112531,process:17241), LocalAddress=10.94.80.17:10495, RemoteAddress=10.185.8.112:63066) as suspect: The connection has fallen 10001 messages (338987 bytes) behind; the threshold is 10000 messages or 10000000 bytes.
INFO [2013-10-22 12:55:01,915] Coherence - 2013-10-22 12:55:01.915/98231.235 Oracle Coherence GE 3.7.1.9 <Info> (thread=DistributedCache, member=2): Extend*TCP has determined that TcpConnection(Id=0x00000141DA2A9EE80A5E50118F07DB0A1503A9B68F5689CECFAB139E1E60D093, Open=true, Member(Id=0, Timestamp=2013-10-20 12:50:05.005, Address=10.185.8.112:0, MachineId=0, Location=site:,machine:parss1112531,process:17241), LocalAddress=10.94.80.17:10495, RemoteAddress=10.185.8.112:63066) is no longer a suspect: The connection has reduced its backlog to 438217 bytes; the target was 2000000 bytes.
INFO [2013-10-22 12:55:04,930] Coherence - 2013-10-22 12:55:04.930/98234.250 Oracle Coherence GE 3.7.1.9 <Info> (thread=DistributedCache, member=2): Extend*TCP has marked TcpConnection(Id=0x00000141DA2A9EE80A5E50118F07DB0A1503A9B68F5689CECFAB139E1E60D093, Open=true, Member(Id=0, Timestamp=2013-10-20 12:50:05.005, Address=10.185.8.112:0, MachineId=0, Location=site:,machine:parss1112531,process:17241), LocalAddress=10.94.80.17:10495, RemoteAddress=10.185.8.112:63066) as suspect: The connection has fallen 15811 messages (534273 bytes) behind; the threshold is 10000 messages or 10000000 bytes.
INFO [2013-10-22 12:55:07,931] Coherence - 2013-10-22 12:55:07.931/98237.251 Oracle Coherence GE 3.7.1.9 <Info> (thread=DistributedCache, member=2): Extend*TCP has determined that TcpConnection(Id=0x00000141DA2A9EE80A5E50118F07DB0A1503A9B68F5689CECFAB139E1E60D093, Open=true, Member(Id=0, Timestamp=2013-10-20 12:50:05.005, Address=10.185.8.112:0, MachineId=0, Location=site:,machine:parss1112531,process:17241), LocalAddress=10.94.80.17:10495, RemoteAddress=10.185.8.112:63066) is no longer a suspect: The connection has reduced its backlog to 617128 bytes; the target was 2000000 bytes.
INFO [2013-10-22 12:55:10,937] Coherence - 2013-10-22 12:55:10.937/98240.257 Oracle Coherence GE 3.7.1.9 <Info> (thread=DistributedCache, member=2): Extend*TCP has marked TcpConnection(Id=0x00000141DA2A9EE80A5E50118F07DB0A1503A9B68F5689CECFAB139E1E60D093, Open=true, Member(Id=0, Timestamp=2013-10-20 12:50:05.005, Address=10.185.8.112:0, MachineId=0, Location=site:,machine:parss1112531,process:17241), LocalAddress=10.94.80.17:10495, RemoteAddress=10.185.8.112:63066) as suspect: The connection has fallen 20956 messages (706640 bytes) behind; the threshold is 10000 messages or 10000000 bytes.
INFO [2013-10-22 12:55:13,939] Coherence - 2013-10-22 12:55:13.939/98243.259 Oracle Coherence GE 3.7.1.9 <Info> (thread=DistributedCache, member=2): Extend*TCP has determined that TcpConnection(Id=0x00000141DA2A9EE80A5E50118F07DB0A1503A9B68F5689CECFAB139E1E60D093, Open=true, Member(Id=0, Timestamp=2013-10-20 12:50:05.005, Address=10.185.8.112:0, MachineId=0, Location=site:,machine:parss1112531,process:17241), LocalAddress=10.94.80.17:10495, RemoteAddress=10.185.8.112:63066) is no longer a suspect: The connection has reduced its backlog to 795235 bytes; the target was 2000000 bytes.
INFO [2013-10-22 12:55:16,940] Coherence - 2013-10-22 12:55:16.940/98246.260 Oracle Coherence GE 3.7.1.9 <Info> (thread=DistributedCache, member=2): Extend*TCP has marked TcpConnection(Id=0x00000141DA2A9EE80A5E50118F07DB0A1503A9B68F5689CECFAB139E1E60D093, Open=true, Member(Id=0, Timestamp=2013-10-20 12:50:05.005, Address=10.185.8.112:0, MachineId=0, Location=site:,machine:parss1112531,process:17241), LocalAddress=10.94.80.17:10495, RemoteAddress=10.185.8.112:63066) as suspect: The connection has fallen 13366 messages (445830 bytes) behind; the threshold is 10000 messages or 10000000 bytes.
INFO [2013-10-22 12:55:19,941] Coherence - 2013-10-22 12:55:19.941/98249.261 Oracle Coherence GE 3.7.1.9 <Info> (thread=DistributedCache, member=2): Extend*TCP has determined that TcpConnection(Id=0x00000141DA2A9EE80A5E50118F07DB0A1503A9B68F5689CECFAB139E1E60D093, Open=true, Member(Id=0, Timestamp=2013-10-20 12:50:05.005, Address=10.185.8.112:0, MachineId=0, Location=site:,machine:parss1112531,process:17241), LocalAddress=10.94.80.17:10495, RemoteAddress=10.185.8.112:63066) is no longer a suspect: The connection has reduced its backlog to 59964 bytes; the target was 2000000 bytes.
INFO [2013-10-22 12:55:28,569] Coherence - 2013-10-22 12:55:28.569/98257.889 Oracle Coherence GE 3.7.1.9 <Info> (thread=DistributedCache, member=2): Extend*TCP has marked TcpConnection(Id=0x00000141DA2A9EE80A5E50118F07DB0A1503A9B68F5689CECFAB139E1E60D093, Open=true, Member(Id=0, Timestamp=2013-10-20 12:50:05.005, Address=10.185.8.112:0, MachineId=0, Location=site:,machine:parss1112531,process:17241), LocalAddress=10.94.80.17:10495, RemoteAddress=10.185.8.112:63066) as suspect: The connection has fallen 10001 messages (336280 bytes) behind; the threshold is 10000 messages or 10000000 bytes.
INFO [2013-10-22 12:55:31,570] Coherence - 2013-10-22 12:55:31.570/98260.890 Oracle Coherence GE 3.7.1.9 <Info> (thread=DistributedCache, member=2): Extend*TCP has determined that TcpConnection(Id=0x00000141DA2A9EE80A5E50118F07DB0A1503A9B68F5689CECFAB139E1E60D093, Open=true, Member(Id=0, Timestamp=2013-10-20 12:50:05.005, Address=10.185.8.112:0, MachineId=0, Location=site:,machine:parss1112531,process:17241), LocalAddress=10.94.80.17:10495, RemoteAddress=10.185.8.112:63066) is no longer a suspect: The connection has reduced its backlog to 441154 bytes; the target was 2000000 bytes.
DEBUG [2013-10-22 12:57:39,427] b.s.c.l.ProductTradingHourLoader - Loading the TradingHours for 48453
DEBUG [2013-10-22 12:57:39,427] b.s.c.l.ProductTradingHourLoader - Loading the TradingHours for 48453
Around the same time the cache server slave had the following:
DEBUG [2013-10-22 12:54:18,210] b.s.cache.loader.ChorusBlotterLoader - isSkip Store is true hence not storing
DEBUG [2013-10-22 12:54:18,210] b.s.cache.loader.ChorusBlotterLoader - isSkip Store is true hence not storing
WARN [2013-10-22 12:56:12,247] Coherence - 2013-10-22 12:56:12.247/98300.867 Oracle Coherence GE 3.7.1.9 <Warning> (thread=PacketPublisher, member=1): Experienced a 1693 ms communication delay (probable remote GC) with Member(Id=2, Timestamp=2013-10-21 09:38:40.957, Address=127.0.0.1:8090, MachineId=60314, Location=site:,machine:localhost,process:13982); 37 packets rescheduled, PauseRate=4.0E-4, Threshold=1020
DEBUG [2013-10-22 12:57:30,844] b.s.cache.loader.ChorusBlotterLoader - Storing the entity class testerpack.set.ejb.model.blotter.fortis.ChorusBlotterDb with id 7582
DEBUG [2013-10-22 12:57:30,844] b.s.cache.loader.ChorusBlotterLoader - Storing the entity class testerpack.set.ejb.model.blotter.fortis.ChorusBlotterDb with id 7582
DEBUG [2013-10-22 12:57:30,844] b.s.cache.loader.ChorusBlotterLoader - isSkip Store is true hence not storing
Coherence version: 3.7.1.9
Coherence client request-timeout: 20 seconds
Coherence client heartbeat-interval:2m