Hi,
After much load testing , we pushed full prod load to coherence based app today and had to fall back on non coherence version after 2 days.
I am seeing following exception in weblogic stdout around the time when users started complaining.
Application makes heavy use of EntryProcessors, lock/unlock , near cache. Each of two weblogics run 6k app threads . we run 4 storage enabled coherence nodes.
We saw that app workiong okay at 2/3rd of the load when it started to fail . weblogic thread count doesnt change much with load.
I did not come across this anytime so far.
2010-12-13 12:18:22.923/38307.644 Oracle Coherence GE 3.5.3/465p7 <Warning> (th
read=EncoreWaitTimeNotifier Thread For Queue ID : 390003, member=6): The event
queue appears to be stuck.
2010-12-13 12:18:22.923/38307.645 Oracle Coherence GE 3.5.3/465p7 <Warning> (th
read=[ENCORE_QUEUE_MONITOR]-2304, member=6): The event queue appears to be stuc
k.
2010-12-13 12:18:25.918/38310.639 Oracle Coherence GE 3.5.3/465p7 <Error> (thre
ad=[AIMS-userid-EXT-PRODUCTION-SERVER-2-103599-1057392-26001], member=6): Full
Thread Dump
Thread[[AIMS-userid-EXT-PRODUCTION-SERVER-2-103599-1057392-26001],5,Thread Grou
p for Queue: 'weblogic.kernel.System']
java.lang.Thread.dumpThreads(Native Method)
java.lang.Thread.getAllStackTraces(Thread.java:1487)
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.ja
va:39)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccesso
rImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:597)
com.tangosol.net.GuardSupport.logStackTraces(GuardSupport.java:886)
com.tangosol.coherence.component.util.daemon.queueProcessor.Service$Eve
ntDispatcher.drainOverflow(Service.CDB:45)
com.tangosol.coherence.component.util.daemon.queueProcessor.service.Gri
d$EventDispatcher.drainOverflow(Grid.CDB:9)
com.tangosol.coherence.component.util.daemon.queueProcessor.service.Gri
d.post(Grid.CDB:17)
com.tangosol.coherence.component.util.daemon.queueProcessor.service.Gri
d.send(Grid.CDB:1)
com.tangosol.coherence.component.util.daemon.queueProcessor.service.Gri
d.poll(Grid.CDB:13)
<distributed-scheme>
<scheme-name>distributed</scheme-name>
<service-name>DistributedCache</service-name>
<thread-count system-property="aims.distributed.threads">8</thread-count>
<partition-count system-property="aims.distributed.partitions">257</partition-count>
<backup-count system-property="aims.distributed.backups">1</backup-count>
<backing-map-scheme>
<local-scheme>
<scheme-ref system-property="aims.local.scheme">{backing-map-selection binary-backing-map}</scheme-ref>
</local-scheme>
</backing-map-scheme>
<serializer>
<class-name>com.tangosol.io.pof.ConfigurablePofContext</class-name>
</serializer>
<autostart>true</autostart>
</distributed-scheme>
I will appreciate any help on this
Thanks,
Tarun
Edited by: Tarun Jindal on Dec 13, 2010 2:28 PM