entrySet performance is lower than I expect
Hi all,
I'm using Coherence 3.5. I have 8 servers, each with 1 storage-disabled WebLogic server and 2 cache nodes.
I have a case where I have a near cache containing employee objects keyed by employee id (String.) I do an entrySet(filter, comparator) on it to search the employee by lanId. I expect to get 0 or 1 employees back. There is not a JMX metric for this that I know of, but I have Wily tracking entrySet, keySet, get, put, etc across all caches. From browsing Wily traces specific to services using this employee cache, I think entrySet regularly takes tens if not hundreds of milliseconds. The cache isn't big (1000 entries per node) and the CPU load isn't high (<50%.) I do have an index (ReflectionExtractor on getLanId.) In fact I use the exact same EqualsFilter/ReflectionExtractor object when querying.
My original guess was that because employee is serializable and not externalizable, I was paying a big cost there. But if there is an index, the query should be able to quickly scan a list of small strings to find the hit. If there is a hit, then the cache ships 1 entry over the wire and deserializes it. I think 1 entry is not more than 4k, so I can't see why this would take so long. I've tried serializing and deserializing a sample employee object on my laptop and got 100 iterations in only 31ms and 1000 in 244ms. Compared to what I see in Wily, I suspect the index isn't being used.
What could be going on? Is the index not working? Am I doing it wrong? I've noticed cache performance correlates with CPU load or cache activity. Higher load --> clearly worse performance.
FWIW: these are the response times in Wily for the different cache methods:
get: 2ms
keySet: 35ms
entrySet: 45ms
containsKey: 3ms
putAll: 162ms
put: 14ms
getAll: 25ms
Simple DB operations (read 1 record from 1 table) are in the 1-2ms range.
thanks
john