I have 3 JVMs running in a cluster. I got this error on all 3 of them and all the cache entries disappeared. What causes this and how can I prevent it? If I just stop using the DB cache store I can be sure this won't happen again, right?
Thanks!
-Andrew
2009-10-28 14:29:25.027/56359.315 Oracle Coherence GE 3.5.2/463 <Error> (thread=DistributedCache, member=1): Attempting recovery (due to soft timeout) of Daemon{Thread="Thread[WriteBehindThread:CacheStoreWrapper(oms.grid.PositionCacheStore),5,WriteBehindThread:CacheStoreWrapper(oms.grid.PositionCacheStore)]", State=Running}
2009-10-28 14:29:28.528/56362.816 Oracle Coherence GE 3.5.2/463 <Error> (thread=DistributedCache, member=1): Terminating guarded execution (due to hard timeout) of Daemon{Thread="Thread[WriteBehindThread:CacheStoreWrapper(oms.grid.PositionCacheStore),5,WriteBehindThread:CacheStoreWrapper(oms.grid.PositionCacheStore)]", State=Running}
2009-10-28 14:29:28.528/56362.816 Oracle Coherence GE 3.5.2/463 <Error> (thread=Termination Thread, member=1): Write-behind thread timed out; stopping the cache service
2009-10-28 14:29:28.529/56362.817 Oracle Coherence GE 3.5.2/463 <D5> (thread=DistributedCache, member=1): Service DistributedCache left the cluster
2009-10-28 14:29:30.573/56364.861 Oracle Coherence GE 3.5.2/463 <Info> (thread=main, member=1): Restarting Service: DistributedCache
2009-10-28 14:29:30.609/56364.897 Oracle Coherence GE 3.5.2/463 <D5> (thread=DistributedCache, member=1): Service DistributedCache joined the cluster with senior service member 2
2009-10-28 14:29:30.613/56364.901 Oracle Coherence GE 3.5.2/463 <D5> (thread=DistributedCache, member=1): Service DistributedCache: received ServiceConfigSync containing 266 entries
2009-10-28 14:29:30.671/56364.959 Oracle Coherence GE 3.5.2/463 <D5> (thread=Cluster, member=1): Member 3 left service DistributedCache with senior member 2
2009-10-28 14:29:32.620/56366.908 Oracle Coherence GE 3.5.2/463 <D5> (thread=Cluster, member=1): Member 2 left service DistributedCache with senior member 4
2009-10-28 14:29:32.756/56367.044 Oracle Coherence GE 3.5.2/463 <Warning> (thread=DistributedCache, member=1): Assigned 257 orphaned primary partitions
2009-10-28 14:29:32.756/56367.044 Oracle Coherence GE 3.5.2/463 <D4> (thread=DistributedCache, member=1): 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 13
6, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207, 208, 209, 210, 211, 212, 213, 214, 215, 216, 217, 218, 219, 220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 239, 240, 241, 242, 243, 244, 245, 246, 247, 248, 249, 250, 251, 252, 253, 254, 255, 256,
2009-10-28 14:29:32.759/56367.047 Oracle Coherence GE 3.5.2/463 <D5> (thread=DistributedCache, member=1): Service DistributedCache: received ServiceConfigSync containing 266 entries
2009-10-28 14:29:32.759/56367.047 Oracle Coherence GE 3.5.2/463 <D4> (thread=DistributedCache, member=1): Obsolete partition info; restoring primary owner for partition 0 from member 2 to 1
2009-10-28 14:29:32.759/56367.047 Oracle Coherence GE 3.5.2/463 <D4> (thread=DistributedCache, member=1): Obsolete partition info; restoring primary owner for partition 1 from member 2 to 1
2009-10-28 14:29:32.759/56367.047 Oracle Coherence GE 3.5.2/463 <D4> (thread=DistributedCache, member=1): Obsolete partition info; restoring primary owner for partition 2 from member 2 to 1
2009-10-28 14:29:32.759/56367.047 Oracle Coherence GE 3.5.2/463 <D4> (thread=DistributedCache, member=1): Obsolete partition info; restoring primary owner for partition 3 from member 2 to 1
2009-10-28 14:29:32.759/56367.047 Oracle Coherence GE 3.5.2/463 <D4> (thread=DistributedCache, member=1): Obsolete partition info; restoring primary owner for partition 4 from member 2 to 1
2009-10-28 14:29:32.759/56367.047 Oracle Coherence GE 3.5.2/463 <D4> (thread=DistributedCache, member=1): Obsolete partition info; restoring primary owner for partition 5 from member 2 to 1
2009-10-28 14:29:32.759/56367.047 Oracle Coherence GE 3.5.2/463 <D4> (thread=DistributedCache, member=1): Obsolete partition info; restoring primary owner for partition 6 from member 2 to 1
2009-10-28 14:29:32.759/56367.047 Oracle Coherence GE 3.5.2/463 <D4> (thread=DistributedCache, member=1): Obsolete partition info; restoring primary owner for partition 7 from member 2 to 1
...
2009-10-28 14:29:32.772/56367.060 Oracle Coherence GE 3.5.2/463 <D4> (thread=DistributedCache, member=1): Obsolete partition info; restoring primary owner for partition 245 from member 2 to 1
2009-10-28 14:29:32.772/56367.060 Oracle Coherence GE 3.5.2/463 <D4> (thread=DistributedCache, member=1): Obsolete partition info; restoring primary owner for partition 242 from member 2 to 1
2009-10-28 14:29:32.772/56367.060 Oracle Coherence GE 3.5.2/463 <D4> (thread=DistributedCache, member=1): Obsolete partition info; restoring primary owner for partition 243 from member 2 to 1
2009-10-28 14:29:32.772/56367.060 Oracle Coherence GE 3.5.2/463 <D4> (thread=DistributedCache, member=1): Obsolete partition info; restoring primary owner for partition 240 from member 2 to 1
2009-10-28 14:29:32.772/56367.060 Oracle Coherence GE 3.5.2/463 <D4> (thread=DistributedCache, member=1): Obsolete partition info; restoring primary owner for partition 241 from member 2 to 1
2009-10-28 14:29:32.773/56367.061 Oracle Coherence GE 3.5.2/463 <D4> (thread=DistributedCache, member=1): Obsolete partition info; restoring primary owner for partition 256 from member 2 to 1
2009-10-28 14:29:32.773/56367.061 Oracle Coherence GE 3.5.2/463 <D5> (thread=DistributedCache, member=1): Deferring the distribution due to 255 pending configuration updates
2009-10-28 14:29:32.801/56367.089 Oracle Coherence GE 3.5.2/463 <D5> (thread=Cluster, member=1): Member 3 joined Service DistributedCache with senior member 4
2009-10-28 14:29:32.833/56367.121 Oracle Coherence GE 3.5.2/463 <D5> (thread=Cluster, member=1): Member 2 joined Service DistributedCache with senior member 4
2009-10-28 14:29:32.995/56367.283 Oracle Coherence GE 3.5.2/463 <D5> (thread=DistributedCache, member=1): 3> Transferring 86 out of 257 primary partitions to member 2 requesting 86
2009-10-28 14:29:35.011/56369.299 Oracle Coherence GE 3.5.2/463 <D5> (thread=DistributedCache, member=1): 3> Transferring 85 out of 171 primary partitions to member 3 requesting 85
2009-10-28 14:29:35.098/56369.386 Oracle Coherence GE 3.5.2/463 <D4> (thread=DistributedCache, member=1): 1> Transferring 86 out of 86 partitions to a node-safe backup 1 at member 2 (under 86)
2009-10-28 14:29:35.104/56369.392 Oracle Coherence GE 3.5.2/463 <D5> (thread=DistributedCache, member=1): Transferring 0KB of backup[1] for PartitionSet{171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207, 208, 209, 210, 211, 212, 213, 214, 215, 216, 217, 218, 219, 220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 239, 240, 241, 242, 243, 244, 245, 246, 247, 248, 249, 250, 251, 252, 253, 254, 255, 256} to member 2
sSql=INSERT INTO executions (exec_id,exchange, exchange_connection_instance, serial_session_id, account, symbol, _52_week_range, dslow, kslow, order_id, order_source, price, quantity, side, status, transact_time, type ) VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,? ) ON DUPLICATE KEY UPDATE price=?, quantity=?, status=?, transact_time=?
2009-10-28 14:40:12.197/57006.485 Oracle Coherence GE 3.5.2/463 <D5> (thread=Cluster, member=1): Member(Id=9, Timestamp=2009-10-28 14:40:12.011, Address=192.168.10.167:8096, MachineId=25255, Location=machine:box2,process:6828,member:andrew, Role=OrderbookviewerViewer) joined Cluster with senior member 1
2009-10-28 14:40:12.346/57006.634 Oracle Coherence GE 3.5.2/463 <D5> (thread=Cluster, member=1): Member 9 joined Service Management with senior member 1
2009-10-28 14:40:13.470/57007.758 Oracle Coherence GE 3.5.2/463 <D5> (thread=Cluster, member=1): Member 9 joined Service DistributedCache with senior member 4
2009-10-28 14:41:35.726/57090.014 Oracle Coherence GE 3.5.2/463 <D5> (thread=Cluster, member=1): MemberLeft notification for Member 9 received from Member(Id=3, Timestamp=2009-10-27 22:50:28.728, Address=192.168.10.197:8090, MachineId=38410, Location=site:bxxxxxllc.com,machine:mail,process:18812, Role=CoherenceServer)
2009-10-28 14:41:35.726/57090.014 Oracle Coherence GE 3.5.2/463 <D5> (thread=Cluster, member=1): Member 9 left service Management with senior member 1
2009-10-28 14:41:35.726/57090.014 Oracle Coherence GE 3.5.2/463 <D5> (thread=Cluster, member=1): Member 9 left service DistributedCache with senior member 4
2009-10-28 14:41:35.726/57090.014 Oracle Coherence GE 3.5.2/463 <D5> (thread=Cluster, member=1): Member(Id=9, Timestamp=2009-10-28 14:41:35.726, Address=192.168.10.167:8096, MachineId=25255, Location=machine:box2,process:6828,member:andrew, Role=OrderbookviewerViewer) left Cluster with senior member 1
2009-10-28 14:41:56.151/57110.439 Oracle Coherence GE 3.5.2/463 <D5> (thread=Cluster, member=1): Member(Id=10, Timestamp=2009-10-28 14:41:55.965, Address=192.168.10.167:8092, MachineId=25255, Location=machine:box2,process:7964,member:andrew, Role=OrderbookviewerViewer) joined Cluster with senior member 1
2009-10-28 14:41:56.430/57110.718 Oracle Coherence GE 3.5.2/463 <D5> (thread=Cluster, member=1): Member 10 joined Service Management with senior member 1
2009-10-28 14:41:57.633/57111.921 Oracle Coherence GE 3.5.2/463 <D5> (thread=Cluster, member=1): Member 10 joined Service DistributedCache with senior member 4
2009-10-28 14:42:10.742/57125.030 Oracle Coherence GE 3.5.2/463 <D5> (thread=Cluster, member=1): MemberLeft notification for Member 10 received from Member(Id=3, Timestamp=2009-10-27 22:50:28.728, Address=192.168.10.197:8090, MachineId=38410, Location=site:bxxxxxllc.com,machine:mail,process:18812, Role=CoherenceServer)
2009-10-28 14:42:10.742/57125.030 Oracle Coherence GE 3.5.2/463 <D5> (thread=Cluster, member=1): Member 10 left service Management with senior member 1
2009-10-28 14:42:10.742/57125.030 Oracle Coherence GE 3.5.2/463 <D5> (thread=Cluster, member=1): Member 10 left service DistributedCache with senior member 4
2009-10-28 14:42:10.742/57125.030 Oracle Coherence GE 3.5.2/463 <D5> (thread=Cluster, member=1): Member(Id=10, Timestamp=2009-10-28 14:42:10.742, Address=192.168.10.167:8092, MachineId=25255, Location=machine:box2,process:7964,member:andrew, Role=OrderbookviewerViewer) left Cluster with senior member 1
2009-10-28 14:42:20.319/57134.607 Oracle Coherence GE 3.5.2/463 <D5> (thread=Cluster, member=1): Member(Id=11, Timestamp=2009-10-28 14:42:20.135, Address=192.168.10.167:8092, MachineId=25255, Location=machine:box2,process:5516,member:andrew, Role=OrderbookviewerViewer) joined Cluster with senior member 1
2009-10-28 14:42:20.485/57134.773 Oracle Coherence GE 3.5.2/463 <D5> (thread=Cluster, member=1): Member 11 joined Service Management with senior member 1
2009-10-28 14:42:21.469/57135.757 Oracle Coherence GE 3.5.2/463 <D5> (thread=Cluster, member=1): Member 11 joined Service DistributedCache with senior member 4
2009-10-28 14:43:26.747/57201.035 Oracle Coherence GE 3.5.2/463 <D5> (thread=Cluster, member=1): TcpRing: disconnected from member 29 due to a kill request
2009-10-28 14:43:26.748/57201.036 Oracle Coherence GE 3.5.2/463 <D5> (thread=Cluster, member=1): Member 29 left service Management with senior member 1
2009-10-28 14:43:26.748/57201.036 Oracle Coherence GE 3.5.2/463 <D5> (thread=Cluster, member=1): Member 29 left service DistributedCache with senior member 4
2009-10-28 14:43:26.748/57201.036 Oracle Coherence GE 3.5.2/463 <D5> (thread=Cluster, member=1): Member(Id=29, Timestamp=2009-10-28 14:43:26.748, Address=192.168.10.40:8090, MachineId=25128, Location=machine:kf,process:608,member:kxxx, Role=OrderbookviewerViewer) left Cluster with senior member 1
2009-10-28 14:43:51.603/57225.891 Oracle Coherence GE 3.5.2/463 <D5> (thread=Cluster, member=1): MemberLeft notification for Member 4 received from Member(Id=16, Timestamp=2009-10-28 07:27:38.682, Address=192.168.10.187:8088, MachineId=25275, Location=machine:machine6,process:1344,member:Kent, Role=OrderbookviewerViewer)
2009-10-28 14:43:51.603/57225.891 Oracle Coherence GE 3.5.2/463 <D5> (thread=Cluster, member=1): Member 4 left service Management with senior member 1
2009-10-28 14:43:51.603/57225.891 Oracle Coherence GE 3.5.2/463 <D5> (thread=Cluster, member=1): Member 4 left service DistributedCache with senior member 8
2009-10-28 14:43:51.603/57225.891 Oracle Coherence GE 3.5.2/463 <D5> (thread=Cluster, member=1): Member(Id=4, Timestamp=2009-10-28 14:43:51.603, Address=192.168.10.197:8091, MachineId=38410, Location=site:bxxxxxllc.com,machine:mail,process:13980,member:Administrator, Role=CoupGsecGsecCoup) left Cluster with senior member 1
2009-10-28 14:43:51.612/57225.900 Oracle Coherence GE 3.5.2/463 <D5> (thread=DistributedCache, member=1): Service DistributedCache: received ServiceConfigSync containing 266 entries
2009-10-28 14:43:51.784/57226.072 Oracle Coherence GE 3.5.2/463 <D5> (thread=Cluster, member=1): MemberLeft notification for Member 11 received from Member(Id=3, Timestamp=2009-10-27 22:50:28.728, Address=192.168.10.197:8090, MachineId=38410, Location=site:bxxxxxllc.com,machine:mail,process:18812, Role=CoherenceServer)
2009-10-28 14:43:51.784/57226.072 Oracle Coherence GE 3.5.2/463 <D5> (thread=Cluster, member=1): Member 11 left service Management with senior member 1
2009-10-28 14:43:51.784/57226.072 Oracle Coherence GE 3.5.2/463 <D5> (thread=Cluster, member=1): Member 11 left service DistributedCache with senior member 8
2009-10-28 14:43:51.784/57226.072 Oracle Coherence GE 3.5.2/463 <D5> (thread=Cluster, member=1): Member(Id=11, Timestamp=2009-10-28 14:43:51.784, Address=192.168.10.167:8092, MachineId=25255, Location=machine:box2,process:5516,member:andrew, Role=OrderbookviewerViewer) left Cluster with senior member 1
2009-10-28 14:44:26.219/57260.507 Oracle Coherence GE 3.5.2/463 <D5> (thread=Cluster, member=1): Member(Id=9, Timestamp=2009-10-28 14:44:26.047, Address=192.168.10.197:8091, MachineId=38410, Location=site:bxxxxxllc.com,machine:mail,process:7160,member:Administrator, Role=CoupGsecGsecCoup) joined Cluster with senior member 1
2009-10-28 14:44:26.294/57260.582 Oracle Coherence GE 3.5.2/463 <D5> (thread=Cluster, member=1): Member 9 joined Service Management with senior member 1
2009-10-28 14:44:26.685/57260.973 Oracle Coherence GE 3.5.2/463 <D5> (thread=Cluster, member=1): Member 9 joined Service DistributedCache with senior member 8
2009-10-28 14:44:39.535/57273.823 Oracle Coherence GE 3.5.2/463 <D5> (thread=Cluster, member=1): Member(Id=10, Timestamp=2009-10-28 14:44:39.347, Address=192.168.10.167:8092, MachineId=25255, Location=machine:box2,process:3456, Role=CoherenceConsole) joined Cluster with senior member 1
2009-10-28 14:44:39.631/57273.919 Oracle Coherence GE 3.5.2/463 <D5> (thread=Cluster, member=1): Member 10 joined Service Management with senior member 1
2009-10-28 14:44:45.552/57279.840 Oracle Coherence GE 3.5.2/463 <D5> (thread=Cluster, member=1): Member 10 joined Service DistributedCache with senior member 8
2009-10-28 14:45:51.937/57346.225 Oracle Coherence GE 3.5.2/463 <D5> (thread=Cluster, member=1): MemberLeft notification for Member 27 received from Member(Id=8, Timestamp=2009-10-28 05:59:34.904, Address=192.168.10.131:8089, MachineId=27138, Location=machine:amd2,process:3628,member:user, Role=PositioneditorQuoteProcessor)
2009-10-28 14:45:51.937/57346.225 Oracle Coherence GE 3.5.2/463 <D5> (thread=Cluster, member=1): Member 27 left service Management with senior member 1
2009-10-28 14:45:51.937/57346.225 Oracle Coherence GE 3.5.2/463 <D5> (thread=Cluster, member=1): Member 27 left service DistributedCache with senior member 8
2009-10-28 14:45:51.937/57346.225 Oracle Coherence GE 3.5.2/463 <D5> (thread=Cluster, member=1): Member(Id=27, Timestamp=2009-10-28 14:45:51.937, Address=192.168.10.40:8089, MachineId=25128, Location=machine:kf,process:1016,member:kxxx, Role=Execution_viewerExecutionViewer) left Cluster with senior member 1
2009-10-28 14:45:52.613/57346.901 Oracle Coherence GE 3.5.2/463 <D5> (thread=Cluster, member=1): MemberLeft notification for Member 28 received from Member(Id=43, Timestamp=2009-10-28 08:34:33.222, Address=192.168.10.210:8088, MachineId=25298, Location=machine:fishbed,process:3272,member:joed, Role=OrderbookviewerViewer)
2009-10-28 14:45:52.613/57346.901 Oracle Coherence GE 3.5.2/463 <D5> (thread=Cluster, member=1): Member 28 left service Management with senior member 1
2009-10-28 14:45:52.613/57346.901 Oracle Coherence GE 3.5.2/463 <D5> (thread=Cluster, member=1): Member 28 left service ReplicatedCache.Optimistic with senior member 1
2009-10-28 14:45:52.613/57346.901 Oracle Coherence GE 3.5.2/463 <D5> (thread=Cluster, member=1): Member 28 left service DistributedCache with senior member 8
2009-10-28 14:45:52.613/57346.901 Oracle Coherence GE 3.5.2/463 <D5> (thread=Cluster, member=1): Member(Id=28, Timestamp=2009-10-28 14:45:52.613, Address=192.168.10.40:8091, MachineId=25128, Location=machine:kf,process:4064,member:kxxx, Role=Order_entry_window) left Cluster with senior member 1
2009-10-28 14:45:53.937/57348.225 Oracle Coherence GE 3.5.2/463 <D5> (thread=Cluster, member=1): MemberLeft notification for Member 26 received from Member(Id=8, Timestamp=2009-10-28 05:59:34.904, Address=192.168.10.131:8089, MachineId=27138, Location=machine:amd2,process:3628,member:user, Role=PositioneditorQuoteProcessor)
2009-10-28 14:45:53.937/57348.225 Oracle Coherence GE 3.5.2/463 <D5> (thread=Cluster, member=1): Member 26 left service Management with senior member 1
2009-10-28 14:45:53.937/57348.225 Oracle Coherence GE 3.5.2/463 <D5> (thread=Cluster, member=1): Member 26 left service ReplicatedCache.Optimistic with senior member 1
2009-10-28 14:45:53.939/57348.227 Oracle Coherence GE 3.5.2/463 <D5> (thread=Cluster, member=1): Member 26 left service DistributedCache with senior member 8
2009-10-28 14:45:53.939/57348.227 Oracle Coherence GE 3.5.2/463 <D5> (thread=Cluster, member=1): Member(Id=26, Timestamp=2009-10-28 14:45:53.939, Address=192.168.10.40:8088, MachineId=25128, Location=machine:kf,process:496,member:kxxx, Role=Strategies) left Cluster with senior member 1
2009-10-28 14:45:57.039/57351.327 Oracle Coherence GE 3.5.2/463 <D5> (thread=Cluster, member=1): MemberLeft notification for Member 31 received from Member(Id=36, Timestamp=2009-10-28 08:15:39.217, Address=192.168.10.8:8092, MachineId=25096, Location=process:2092,member:Administrator, Role=Order_entry_window)
2009-10-28 14:45:57.039/57351.327 Oracle Coherence GE 3.5.2/463 <D5> (thread=Cluster, member=1): Member 31 left service Management with senior member 1
2009-10-28 14:45:57.039/57351.327 Oracle Coherence GE 3.5.2/463 <D5> (thread=Cluster, member=1): Member 31 left service ReplicatedCache.Optimistic with senior member 1
2009-10-28 14:45:57.039/57351.327 Oracle Coherence GE 3.5.2/463 <D5> (thread=Cluster, member=1): Member 31 left service DistributedCache with senior member 8
2009-10-28 14:45:57.039/57351.327 Oracle Coherence GE 3.5.2/463 <D5> (thread=Cluster, member=1): Member(Id=31, Timestamp=2009-10-28 14:45:57.039, Address=192.168.10.40:8092, MachineId=25128, Location=machine:kf,process:736,member:kxxx, Role=Positioneditor) left Cluster with senior member 1
2009-10-28 14:47:58.716/57473.004 Oracle Coherence GE 3.5.2/463 <D5> (thread=Cluster, member=1): Member(Id=4, Timestamp=2009-10-28 14:47:58.537, Address=192.168.10.167:8096, MachineId=25255, Location=machine:box2,process:6004,member:andrew, Role=Execution_viewerExecutionViewer) joined Cluster with senior member 1
2009-10-28 14:47:58.848/57473.136 Oracle Coherence GE 3.5.2/463 <D5> (thread=Cluster, member=1): Member 4 joined Service Management with senior member 1
2009-10-28 14:48:00.058/57474.346 Oracle Coherence GE 3.5.2/463 <D5> (thread=Cluster, member=1): Member 4 joined Service DistributedCache with senior member 8
2009-10-28 14:48:27.541/57501.829 Oracle Coherence GE 3.5.2/463 <D5> (thread=Cluster, member=1): MemberLeft notification for Member 4 received from Member(Id=9, Timestamp=2009-10-28 14:44:26.047, Address=192.168.10.197:8091, MachineId=38410, Location=site:bxxxxxllc.com,machine:mail,process:7160,member:Administrator, Role=CoupGsecGsecCoup)
2009-10-28 14:48:27.541/57501.829 Oracle Coherence GE 3.5.2/463 <D5> (thread=Cluster, member=1): Member 4 left service Management with senior member 1
2009-10-28 14:48:27.541/57501.829 Oracle Coherence GE 3.5.2/463 <D5> (thread=Cluster, member=1): Member 4 left service DistributedCache with senior member 8
2009-10-28 14:48:27.541/57501.829 Oracle Coherence GE 3.5.2/463 <D5> (thread=Cluster, member=1): Member(Id=4, Timestamp=2009-10-28 14:48:27.541, Address=192.168.10.167:8096, MachineId=25255, Location=machine:box2,process:6004,member:andrew, Role=Execution_viewerExecutionViewer) left Cluster with senior member 1