Skip to Main Content
Forums

Integration

Announcement

For appeals, questions and feedback about Oracle Forums, please email oracle-forums-moderators_us@oracle.com. Technical questions should be asked in the appropriate category. Thank you!

<Error> Manual intervention is required to stop the members...

snidely_whiplashDec 15 2010
I'm seeing a couple errors when disconnected members try to rejoin.

I've read what's here about "validate polls":
http://coherence.oracle.com/display/COH35UG/Partitioned+Cache+Service+Log+Messages

I think the relevant stuff is this: 
2010-12-15 09:04:25.193/17363.862 Oracle Coherence GE 3.6.1.0 <Error> (thread=Cluster, member=n/a): validatePolls: This service timed-out due to unanswered handshake request. Manual intervention is required to stop the members that have not responded to this Poll
  {
  PollId=2, active
  InitTimeMillis=1292425166002
  Service=Cluster (0)
  RespondedMemberSet=[1,2,3,4,5,6,7,8,9,10,11,12,17,19,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,40,41,42]
  LeftMemberSet=[20,21,39]
  RemainingMemberSet=[18]
  }
2010-12-15 09:04:25.197/17363.866 Oracle Coherence GE 3.6.1.0 <Error> (thread=qReader, member=n/a): Error while starting cluster: com.tangosol.net.RequestTimeoutException: Timeout during service start: ServiceInfo(Id=0, Name=Cluster, Type=Cluster
  MemberSet=ServiceMemberSet...
I have a number of these apps running simultaneously between two machines. At about the same time all the instances of the app running on one machine experienced the same issue. It looks like those members received cluster heartbeats which did not include the memeber so they left the cluster. They all then tried to rejoin but did not get a timely handshake back from a few members (20,21,39) so they just gave up. Is this correct behavior? Is there something I can do so it continues to reconnect? At a minimum can I somehow listen for this situation and do a System.exit(-1) so the apps' startup scripts know to restart them?

Thanks,
Andrew

The (almost) full log including the bit posted above is: 
2010-12-15 04:15:18.478/17.147 Oracle Coherence 3.6.1.0 <Info> (thread=qReader, member=n/a): Loaded operational configuration from "jar:file:/Z:/coherence/lib/coherence.jar!/tangosol-coherence.xml"
2010-12-15 04:15:18.498/17.167 Oracle Coherence 3.6.1.0 <Info> (thread=qReader, member=n/a): Loaded operational overrides from "jar:file:/Z:/coherence/lib/coherence.jar!/tangosol-coherence-override-dev.xml"

Oracle Coherence Version 3.6.1.0 Build 19636
 Grid Edition: Development mode
Copyright (c) 2000, 2010, Oracle and/or its affiliates. All rights reserved.

2010-12-15 04:15:19.122/17.791 Oracle Coherence GE 3.6.1.0 <Info> (thread=qReader, member=n/a): Loaded cache configuration from "file:/Z:/coherence/cache-config.xml"
2010-12-15 04:15:24.853/23.522 Oracle Coherence GE 3.6.1.0 <Info> (thread=Cluster, member=n/a): Failed to satisfy the variance: allowed=16, actual=52
2010-12-15 04:15:24.853/23.522 Oracle Coherence GE 3.6.1.0 <Info> (thread=Cluster, member=n/a): Increasing allowable variance to 20
2010-12-15 04:15:25.884/24.553 Oracle Coherence GE 3.6.1.0 <Info> (thread=Cluster, member=n/a): This Member(Id=16, Timestamp=2010-12-15 04:15:29.001, Address=192.168.3.6:8092, MachineId=27140, Location=machine:mothra,process:5888,member:six, Role=RediquoteclientRediQuoteClient, Edition=Grid Edition, Mode=Development, CpuCount=32, SocketCount=32) joined cluster "dev" with senior Member(Id=1, Timestamp=2010-12-14 20:01:47.154, Address=192.168.3.20:8088, MachineId=27412, Location=machine:amd4,process:4292,member:Administrator, Role=CoherenceServer, Edition=Grid Edition, Mode=Development, CpuCount=8, SocketCount=2)
2010-12-15 04:15:26.214/24.883 Oracle Coherence GE 3.6.1.0 <Info> (thread=qReader, member=n/a): Started cluster Name=dev

Group{Address=225.0.0.1, Port=54321, TTL=4}

MasterMemberSet
  (
  ThisMember=Member(Id=16, Timestamp=2010-12-15 04:15:29.001, Address=192.168.3.6:8092, MachineId=27140, Location=machine:mothra,process:5888,member:six, Role=RediquoteclientRediQuoteClient)
  OldestMember=Member(Id=1, Timestamp=2010-12-14 20:01:47.154, Address=192.168.3.20:8088, MachineId=27412, Location=machine:amd4,process:4292,member:Administrator, Role=CoherenceServer)
  ActualMemberSet=MemberSet(Size=15, BitSetCount=2
    Member(Id=1, Timestamp=2010-12-14 20:01:47.154, Address=192.168.3.20:8088, MachineId=27412, Location=machine:amd4,process:4292,member:Administrator, Role=CoherenceServer)
    Member(Id=2, Timestamp=2010-12-14 20:01:50.669, Address=192.168.3.20:8090, MachineId=27412, Location=machine:amd4,process:6048,member:Administrator, Role=CoherenceServer)
    Member(Id=3, Timestamp=2010-12-14 20:01:51.794, Address=192.168.3.20:8092, MachineId=27412, Location=machine:amd4,process:6980,member:Administrator, Role=CoherenceServer)
    Member(Id=4, Timestamp=2010-12-15 02:33:41.403, Address=192.168.3.5:8088, MachineId=27139, Location=machine:blackjack,process:12420,member:Administrator, Role=DJclientStartMe)
    Member(Id=5, Timestamp=2010-12-15 02:56:49.727, Address=192.168.3.5:8090, MachineId=27139, Location=machine:blackjack,process:7520,member:Administrator, Role=DJclientBboClient)
    Member(Id=7, Timestamp=2010-12-15 04:11:43.521, Address=192.168.3.5:8092, MachineId=27139, Location=machine:blackjack,process:7456,member:JD, Role=PE)
    Member(Id=8, Timestamp=2010-12-15 04:15:07.121, Address=192.168.3.7:8088, MachineId=27399, Location=machine:amd2,process:8036,member:ten, Role=RediquoteclientRediQuoteClient)
    Member(Id=9, Timestamp=2010-12-15 04:15:07.27, Address=192.168.3.7:8090, MachineId=27399, Location=machine:amd2,process:6328,member:twelve, Role=RediquoteclientRediQuoteClient)
    Member(Id=10, Timestamp=2010-12-15 04:15:07.295, Address=192.168.3.7:8092, MachineId=27399, Location=machine:amd2,process:2028,member:three, Role=RediquoteclientRediQuoteClient)
    Member(Id=11, Timestamp=2010-12-15 04:15:07.413, Address=192.168.3.7:8094, MachineId=27399, Location=machine:amd2,process:2744,member:two, Role=RediquoteclientRediQuoteClient)
    Member(Id=12, Timestamp=2010-12-15 04:15:11.511, Address=192.168.3.7:8096, MachineId=27399, Location=machine:amd2,process:2704,member:one, Role=RediquoteclientRediQuoteClient)
    Member(Id=13, Timestamp=2010-12-15 04:15:23.773, Address=192.168.3.6:8088, MachineId=27140, Location=machine:mothra,process:8424,member:five, Role=RediquoteclientRediQuoteClient)
    Member(Id=14, Timestamp=2010-12-15 04:15:24.957, Address=192.168.3.6:8090, MachineId=27140, Location=machine:mothra,process:8480,member:four, Role=RediquoteclientRediQuoteClient)
    Member(Id=15, Timestamp=2010-12-15 04:15:28.992, Address=192.168.3.6:8094, MachineId=27140, Location=machine:mothra,process:5152,member:seven, Role=RediquoteclientRediQuoteClient)
    Member(Id=16, Timestamp=2010-12-15 04:15:29.001, Address=192.168.3.6:8092, MachineId=27140, Location=machine:mothra,process:5888,member:six, Role=RediquoteclientRediQuoteClient)
    )
  RecycleMillis=1200000
  RecycleSet=MemberSet(Size=0,BitSetCount=0
    )
  )

TcpRing{Connections=[15]}
IpMonitor{AddressListSize=3}

2010-12-15 07:12:32.467/10651.136 Oracle Coherence GE 3.6.1.0 <Warning> (thread=PacketPublisher, member=16): Experienced a 86107 ms communication delay (probable remote GC) with Member(Id=21, Timestamp=2010-12-15 05:56:59.648, Address=192.168.3.20:8096, MachineId=27412, Location=machine:amd4,process:3844,member:Administrator, Role=TvuMainTVU); 447 packets rescheduled, PauseRate=0.0189, Threshold=1878
2010-12-15 07:12:42.215/10660.884 Oracle Coherence GE 3.6.1.0 <Warning> (thread=PacketPublisher, member=16): Experienced a 49852 ms communication delay (probable remote GC) with Member(Id=20, Timestamp=2010-12-15 05:56:02.412, Address=192.168.3.20:8098, MachineId=27412, Location=machine:amd4,process:4704,member:Administrator, Role=StatsStatsLoader2); 266 packets rescheduled, PauseRate=0.0107, Threshold=1878
2010-12-15 07:16:32.750/10891.419 Oracle Coherence GE 3.6.1.0 <Warning> (thread=PacketPublisher, member=16): Experienced a 3372 ms communication delay (probable remote GC) with Member(Id=18, Timestamp=2010-12-15 05:40:07.904, Address=192.168.3.20:8094, MachineId=27412, Location=machine:amd4,process:5764,member:Administrator, Role=A1bookBinaryMainA1Multicast); 33 packets rescheduled, PauseRate=5.0E-4, Threshold=1612
2010-12-15 07:26:31.208/11489.878 Oracle Coherence GE 3.6.1.0 <Warning> (thread=PacketPublisher, member=16): Experienced a 4819 ms communication delay (probable remote GC) with Member(Id=18, Timestamp=2010-12-15 05:40:07.904, Address=192.168.3.20:8094, MachineId=27412, Location=machine:amd4,process:5764,member:Administrator, Role=A1bookBinaryMainA1Multicast); 41 packets rescheduled, PauseRate=0.0012, Threshold=1456
2010-12-15 07:47:21.567/12740.237 Oracle Coherence GE 3.6.1.0 <Warning> (thread=PacketPublisher, member=16): Experienced a 7115 ms communication delay (probable remote GC) with Member(Id=20, Timestamp=2010-12-15 05:56:02.412, Address=192.168.3.20:8098, MachineId=27412, Location=machine:amd4,process:4704,member:Administrator, Role=StatsStatsLoader2); 52 packets rescheduled, PauseRate=0.0085, Threshold=1612
2010-12-15 07:47:49.067/12767.736 Oracle Coherence GE 3.6.1.0 <Warning> (thread=PacketPublisher, member=16): Experienced a 8614 ms communication delay (probable remote GC) with Member(Id=21, Timestamp=2010-12-15 05:56:59.648, Address=192.168.3.20:8096, MachineId=27412, Location=machine:amd4,process:3844,member:Administrator, Role=TvuMainTVU); 60 packets rescheduled, PauseRate=0.0142, Threshold=1696
2010-12-15 07:56:32.988/13291.657 Oracle Coherence GE 3.6.1.0 <Warning> (thread=PacketPublisher, member=16): Experienced a 11513 ms communication delay (probable remote GC) with Member(Id=21, Timestamp=2010-12-15 05:56:59.648, Address=192.168.3.20:8096, MachineId=27412, Location=machine:amd4,process:3844,member:Administrator, Role=TvuMainTVU); 74 packets rescheduled, PauseRate=0.0147, Threshold=1612
2010-12-15 08:51:33.890/16592.559 Oracle Coherence GE 3.6.1.0 <Warning> (thread=PacketPublisher, member=16): Experienced a 33650 ms communication delay (probable remote GC) with Member(Id=20, Timestamp=2010-12-15 05:56:02.412, Address=192.168.3.20:8098, MachineId=27412, Location=machine:amd4,process:4704,member:Administrator, Role=StatsStatsLoader2); 184 packets rescheduled, PauseRate=0.0085, Threshold=1456
2010-12-15 08:52:22.802/16641.471 Oracle Coherence GE 3.6.1.0 <Warning> (thread=PacketPublisher, member=16): Experienced a 10437 ms communication delay (probable remote GC) with Member(Id=18, Timestamp=2010-12-15 05:40:07.904, Address=192.168.3.20:8094, MachineId=27412, Location=machine:amd4,process:5764,member:Administrator, Role=A1bookBinaryMainA1Multicast); 68 packets rescheduled, PauseRate=0.0016, Threshold=1129
2010-12-15 08:53:23.232/16701.901 Oracle Coherence GE 3.6.1.0 <Warning> (thread=PacketPublisher, member=16): Experienced a 5682 ms communication delay (probable remote GC) with Member(Id=18, Timestamp=2010-12-15 05:40:07.904, Address=192.168.3.20:8094, MachineId=27412, Location=machine:amd4,process:5764,member:Administrator, Role=A1bookBinaryMainA1Multicast); 45 packets rescheduled, PauseRate=0.0020, Threshold=1073
2010-12-15 08:53:50.802/16729.471 Oracle Coherence GE 3.6.1.0 <Warning> (thread=PacketPublisher, member=16): Experienced a 15112 ms communication delay (probable remote GC) with Member(Id=39, Timestamp=2010-12-15 07:57:53.415, Address=192.168.3.20:8102, MachineId=27412, Location=machine:amd4,process:4596,member:Administrator, Role=ex_viewer); 92 packets rescheduled, PauseRate=0.0044, Threshold=1785
2010-12-15 08:53:50.814/16729.483 Oracle Coherence GE 3.6.1.0 <Warning> (thread=PacketPublisher, member=16): Experienced a 1263 ms communication delay (probable remote GC) with Member(Id=21, Timestamp=2010-12-15 05:56:59.648, Address=192.168.3.20:8096, MachineId=27412, Location=machine:amd4,process:3844,member:Administrator, Role=TvuMainTVU); 23 packets rescheduled, PauseRate=0.0101, Threshold=1532
2010-12-15 08:54:36.660/16775.329 Oracle Coherence GE 3.6.1.0 <Warning> (thread=PacketPublisher, member=16): Experienced a 76110 ms communication delay (probable remote GC) with Member(Id=20, Timestamp=2010-12-15 05:56:02.412, Address=192.168.3.20:8098, MachineId=27412, Location=machine:amd4,process:4704,member:Administrator, Role=StatsStatsLoader2); 396 packets rescheduled, PauseRate=0.0155, Threshold=1384
2010-12-15 08:54:36.915/16775.584 Oracle Coherence GE 3.6.1.0 <Warning> (thread=PacketPublisher, member=16): Experienced a 5302 ms communication delay (probable remote GC) with Member(Id=21, Timestamp=2010-12-15 05:56:59.648, Address=192.168.3.20:8096, MachineId=27412, Location=machine:amd4,process:3844,member:Administrator, Role=TvuMainTVU); 43 packets rescheduled, PauseRate=0.0105, Threshold=1456
2010-12-15 08:55:28.984/16827.653 Oracle Coherence GE 3.6.1.0 <Warning> (thread=PacketPublisher, member=16): Experienced a 4293 ms communication delay (probable remote GC) with Member(Id=21, Timestamp=2010-12-15 05:56:59.648, Address=192.168.3.20:8096, MachineId=27412, Location=machine:amd4,process:3844,member:Administrator, Role=TvuMainTVU); 38 packets rescheduled, PauseRate=0.0109, Threshold=1384
2010-12-15 08:56:07.114/16865.783 Oracle Coherence GE 3.6.1.0 <Warning> (thread=PacketPublisher, member=16): Experienced a 3385 ms communication delay (probable remote GC) with Member(Id=18, Timestamp=2010-12-15 05:40:07.904, Address=192.168.3.20:8094, MachineId=27412, Location=machine:amd4,process:5764,member:Administrator, Role=A1bookBinaryMainA1Multicast); 33 packets rescheduled, PauseRate=0.0023, Threshold=1020
2010-12-15 08:56:55.091/16913.760 Oracle Coherence GE 3.6.1.0 <Warning> (thread=PacketPublisher, member=16): Experienced a 1398 ms communication delay (probable remote GC) with Member(Id=21, Timestamp=2010-12-15 05:56:59.648, Address=192.168.3.20:8096, MachineId=27412, Location=machine:amd4,process:3844,member:Administrator, Role=TvuMainTVU); 23 packets rescheduled, PauseRate=0.0109, Threshold=1315
2010-12-15 08:57:36.620/16955.289 Oracle Coherence GE 3.6.1.0 <Warning> (thread=PacketPublisher, member=16): Experienced a 9830 ms communication delay (probable remote GC) with Member(Id=18, Timestamp=2010-12-15 05:40:07.904, Address=192.168.3.20:8094, MachineId=27412, Location=machine:amd4,process:5764,member:Administrator, Role=A1bookBinaryMainA1Multicast); 66 packets rescheduled, PauseRate=0.0031, Threshold=969
2010-12-15 08:57:36.825/16955.494 Oracle Coherence GE 3.6.1.0 <Warning> (thread=PacketPublisher, member=16): Experienced a 127134 ms communication delay (probable remote GC) with Member(Id=39, Timestamp=2010-12-15 07:57:53.415, Address=192.168.3.20:8102, MachineId=27412, Location=machine:amd4,process:4596,member:Administrator, Role=ex_viewer); 650 packets rescheduled, PauseRate=0.0394, Threshold=1696
2010-12-15 08:59:22.500/17061.169 Oracle Coherence GE 3.6.1.0 <Error> (thread=Cluster, member=16): Received cluster heartbeat from the senior Member(Id=1, Timestamp=2010-12-14 20:01:47.154, Address=192.168.3.20:8088, MachineId=27412, Location=machine:amd4,process:4292,member:Administrator, Role=CoherenceServer) that does not contain this Member(Id=16, Timestamp=2010-12-15 04:15:29.001, Address=192.168.3.6:8092, MachineId=27140, Location=machine:mothra,process:5888,member:six, Role=RediquoteclientRediQuoteClient); stopping cluster service.
2010-12-15 08:59:22.504/17061.173 Oracle Coherence GE 3.6.1.0 <Error> (thread=Cluster, member=16): Full Thread Dump

Thread[Thread-3,5,main]
	java.lang.Object.wait(Native Method)
	rediquoteclient.QuoteServerLink.pause(QuoteServerLink.java:158)
	rediquoteclient.QuoteServerLink$1.run(QuoteServerLink.java:54)
	java.lang.Thread.run(Thread.java:619)

Thread[Cluster|Member(Id=16, Timestamp=2010-12-15 04:15:29.001, Address=192.168.3.6:8092, MachineId=27140, Location=machine:mothra,process:5888,member:six, Role=RediquoteclientRediQuoteClient),5,Cluster]
	java.lang.Thread.dumpThreads(Native Method)
	java.lang.Thread.getAllStackTraces(Thread.java:1487)
	com.tangosol.net.GuardSupport.logStackTraces(GuardSupport.java:810)
	com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.ClusterService$SeniorMemberHeartbeat.onReceived(ClusterService.CDB:33)
	com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.onMessage(Grid.CDB:11)
	com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.onNotify(Grid.CDB:33)
	com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.ClusterService.onNotify(ClusterService.CDB:3)
	com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:42)
	java.lang.Thread.run(Thread.java:619)
...


Thread[Finalizer,8,system]
	java.lang.Object.wait(Native Method)
	java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
	java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
	java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)

2010-12-15 08:59:23.722/17062.391 Oracle Coherence GE 3.6.1.0 <Info> (thread=qReader, member=16): Restarting NamedCache: quotes.REDI
2010-12-15 08:59:23.723/17062.392 Oracle Coherence GE 3.6.1.0 <Info> (thread=qReader, member=16): Restarting Service: DistributedQuotesCacheService
2010-12-15 08:59:23.723/17062.392 Oracle Coherence GE 3.6.1.0 <Info> (thread=qReader, member=n/a): Restarting cluster
2010-12-15 08:59:25.987/17064.656 Oracle Coherence GE 3.6.1.0 <Info> (thread=Cluster, member=n/a): This Member(Id=43, Timestamp=2010-12-15 08:59:44.971, Address=192.168.3.6:8092, MachineId=27140, Location=machine:mothra,process:5888,member:six, Role=RediquoteclientRediQuoteClient, Edition=Grid Edition, Mode=Development, CpuCount=32, SocketCount=32) joined cluster "dev" with senior Member(Id=1, Timestamp=2010-12-14 20:01:47.154, Address=192.168.3.20:8088, MachineId=27412, Location=machine:amd4,process:4292,member:Administrator, Role=CoherenceServer, Edition=Grid Edition, Mode=Development, CpuCount=8, SocketCount=2)
2010-12-15 09:03:59.364/17338.033 Oracle Coherence GE 3.6.1.0 <Error> (thread=Cluster, member=n/a): Detected soft timeout) of {WrapperGuardable Guard{Daemon=IpMonitor} Service=ClusterService{Name=Cluster, State=(SERVICE_STARTED, STATE_JOINED), Id=0, Version=3.6, OldestMemberId=1}}
2010-12-15 09:03:59.370/17338.039 Oracle Coherence GE 3.6.1.0 <Warning> (thread=Recovery Thread, member=n/a): Attempting recovery of Guard{Daemon=IpMonitor}
2010-12-15 09:04:25.193/17363.862 Oracle Coherence GE 3.6.1.0 <Error> (thread=Cluster, member=n/a): validatePolls: This service timed-out due to unanswered handshake request. Manual intervention is required to stop the members that have not responded to this Poll
  {
  PollId=2, active
  InitTimeMillis=1292425166002
  Service=Cluster (0)
  RespondedMemberSet=[1,2,3,4,5,6,7,8,9,10,11,12,17,19,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,40,41,42]
  LeftMemberSet=[20,21,39]
  RemainingMemberSet=[18]
  }
2010-12-15 09:04:25.197/17363.866 Oracle Coherence GE 3.6.1.0 <Error> (thread=qReader, member=n/a): Error while starting cluster: com.tangosol.net.RequestTimeoutException: Timeout during service start: ServiceInfo(Id=0, Name=Cluster, Type=Cluster
  MemberSet=ServiceMemberSet(
    OldestMember=n/a
    ActualMemberSet=MemberSet(Size=33, BitSetCount=2
      Member(Id=1, Timestamp=2010-12-14 20:01:47.154, Address=192.168.3.20:8088, MachineId=27412, Location=machine:amd4,process:4292,member:Administrator, Role=CoherenceServer)
      Member(Id=2, Timestamp=2010-12-14 20:01:50.669, Address=192.168.3.20:8090, MachineId=27412, Location=machine:amd4,process:6048,member:Administrator, Role=CoherenceServer)
      Member(Id=3, Timestamp=2010-12-14 20:01:51.794, Address=192.168.3.20:8092, MachineId=27412, Location=machine:amd4,process:6980,member:Administrator, Role=CoherenceServer)
      Member(Id=5, Timestamp=2010-12-15 02:56:49.727, Address=192.168.3.5:8090, MachineId=27139, Location=machine:blackjack,process:7520,member:Administrator, Role=DJclientBboClient)
      Member(Id=7, Timestamp=2010-12-15 04:11:43.521, Address=192.168.3.5:8092, MachineId=27139, Location=machine:blackjack,process:7456,member:JD, Role=PE)
      Member(Id=8, Timestamp=2010-12-15 04:15:07.121, Address=192.168.3.7:8088, MachineId=27399, Location=machine:amd2,process:8036,member:ten, Role=RediquoteclientRediQuoteClient)
      Member(Id=9, Timestamp=2010-12-15 04:15:07.27, Address=192.168.3.7:8090, MachineId=27399, Location=machine:amd2,process:6328,member:twelve, Role=RediquoteclientRediQuoteClient)
      Member(Id=10, Timestamp=2010-12-15 04:15:07.295, Address=192.168.3.7:8092, MachineId=27399, Location=machine:amd2,process:2028,member:three, Role=RediquoteclientRediQuoteClient)
      Member(Id=11, Timestamp=2010-12-15 04:15:07.413, Address=192.168.3.7:8094, MachineId=27399, Location=machine:amd2,process:2744,member:two, Role=RediquoteclientRediQuoteClient)
      Member(Id=12, Timestamp=2010-12-15 04:15:11.511, Address=192.168.3.7:8096, MachineId=27399, Location=machine:amd2,process:2704,member:one, Role=RediquoteclientRediQuoteClient)
      Member(Id=17, Timestamp=2010-12-15 05:24:58.776, Address=192.168.3.5:8096, MachineId=27139, Location=machine:blackjack,process:9836,member:Administrator, Role=DJclientRestarter)
      Member(Id=18, Timestamp=2010-12-15 05:40:07.904, Address=192.168.3.20:8094, MachineId=27412, Location=machine:amd4,process:5764,member:Administrator, Role=A1bookBinaryMainA1Multicast)
      Member(Id=22, Timestamp=2010-12-15 06:00:07.414, Address=192.168.3.25:8088, MachineId=27417, Location=machine:amd6,process:316,member:Administrator, Role=OmsOms)
      Member(Id=23, Timestamp=2010-12-15 06:00:09.872, Address=192.168.3.7:8098, MachineId=27399, Location=machine:amd2,process:2984,member:user, Role=PEQuoteProcessor)
      Member(Id=24, Timestamp=2010-12-15 06:00:51.201, Address=192.168.3.24:8088, MachineId=27416, Location=machine:Amd5,process:3096,member:St, Role=OmsOms)
      Member(Id=25, Timestamp=2010-12-15 06:52:44.897, Address=192.168.3.32:8088, MachineId=27424, Location=machine:themysticx,process:3804,member:Tony, Role=OmsOms)
      Member(Id=26, Timestamp=2010-12-15 07:13:05.082, Address=192.168.3.26:8088, MachineId=27418, Location=machine:dabulls,process:3256,member:St, Role=BboBBOClientMain)
      Member(Id=27, Timestamp=2010-12-15 07:27:00.105, Address=192.168.3.120:8090, MachineId=27512, Location=machine:TEST-1234,process:2936,member:TEST, Role=EXV)
      Member(Id=28, Timestamp=2010-12-15 07:27:00.183, Address=192.168.3.120:8088, MachineId=27512, Location=machine:TEST-1234,process:3116,member:TEST, Role=oew)
      Member(Id=29, Timestamp=2010-12-15 07:27:00.251, Address=192.168.3.120:8092, MachineId=27512, Location=machine:TEST-1234,process:272,member:TEST, Role=OV)
      Member(Id=30, Timestamp=2010-12-15 07:51:41.145, Address=192.168.3.116:8088, MachineId=27508, Location=machine:BE,process:2144,member:user, Role=EXV)
      Member(Id=32, Timestamp=2010-12-15 07:34:03.385, Address=192.168.3.5:8098, MachineId=27139, Location=machine:blackjack,process:5836,member:WayneN, Role=PE)
      Member(Id=35, Timestamp=2010-12-15 08:22:49.77, Address=192.168.3.121:8092, MachineId=27513, Location=machine:Hd,process:4052,member:Hd, Role=oew)
      Member(Id=36, Timestamp=2010-12-15 07:51:41.529, Address=192.168.3.116:8090, MachineId=27508, Location=machine:BE,process:2292,member:user, Role=OV)
      Member(Id=37, Timestamp=2010-12-15 07:51:41.556, Address=192.168.3.116:8091, MachineId=27508, Location=machine:BE,process:2064,member:user, Role=oew)
      Member(Id=38, Timestamp=2010-12-15 07:51:41.863, Address=192.168.3.116:8094, MachineId=27508, Location=machine:BE,process:2108,member:user, Role=PE)
      Member(Id=40, Timestamp=2010-12-15 08:06:00.705, Address=192.168.3.5:8100, MachineId=27139, Location=machine:blackjack,process:5484,member:ChrisL, Role=PE)
      Member(Id=41, Timestamp=2010-12-15 08:22:59.722, Address=192.168.3.121:8094, MachineId=27513, Location=machine:Hd,process:2972,member:Hd, Role=PE)
      Member(Id=43, Timestamp=2010-12-15 08:59:44.971, Address=192.168.3.6:8092, MachineId=27140, Location=machine:mothra,process:5888,member:six, Role=RediquoteclientRediQuoteClient)
      Member(Id=44, Timestamp=2010-12-15 09:01:17.056, Address=192.168.3.121:8090, MachineId=27513, Location=machine:Hd,process:860,member:Hd)
      Member(Id=45, Timestamp=2010-12-15 09:01:35.962, Address=192.168.3.5:8094, MachineId=27139, Location=machine:blackjack,process:4852,member:Administrator)
      Member(Id=46, Timestamp=2010-12-15 09:01:41.282, Address=192.168.3.5:8088, MachineId=27139, Location=machine:blackjack,process:12420,member:Administrator)
      Member(Id=47, Timestamp=2010-12-15 09:02:44.877, Address=192.168.3.121:8088, MachineId=27513, Location=machine:Hd,process:2804,member:Hd)
      )
    MemberId/ServiceVersion/ServiceJoined/MemberState
      1/3.6/Tue Dec 14 20:01:47 CST 2010/JOINED,
      2/3.6/Tue Dec 14 20:01:50 CST 2010/JOINED,
      3/3.6/Tue Dec 14 20:01:51 CST 2010/JOINED,
      5/3.6/Wed Dec 15 02:56:49 CST 2010/JOINED,
      7/3.6/Wed Dec 15 04:11:43 CST 2010/JOINED,
      8/3.6/Wed Dec 15 04:15:07 CST 2010/JOINED,
      9/3.6/Wed Dec 15 04:15:07 CST 2010/JOINED,
      10/3.6/Wed Dec 15 04:15:07 CST 2010/JOINED,
      11/3.6/Wed Dec 15 04:15:07 CST 2010/JOINED,
      12/3.6/Wed Dec 15 04:15:11 CST 2010/JOINED,
      17/3.6/Wed Dec 15 05:24:58 CST 2010/JOINED,
      18/3.6/Wed Dec 15 05:40:07 CST 2010/JOINING,
      22/3.6/Wed Dec 15 06:00:07 CST 2010/JOINED,
      23/3.6/Wed Dec 15 06:00:09 CST 2010/JOINED,
      24/3.6/Wed Dec 15 06:00:51 CST 2010/JOINED,
      25/3.6/Wed Dec 15 06:52:44 CST 2010/JOINED,
      26/3.6/Wed Dec 15 07:13:05 CST 2010/JOINED,
      27/3.6/Wed Dec 15 07:27:00 CST 2010/JOINED,
      28/3.6/Wed Dec 15 07:27:00 CST 2010/JOINED,
      29/3.6/Wed Dec 15 07:27:00 CST 2010/JOINED,
      30/3.6/Wed Dec 15 07:51:41 CST 2010/JOINED,
      32/3.6/Wed Dec 15 07:34:03 CST 2010/JOINED,
      35/3.6/Wed Dec 15 08:22:49 CST 2010/JOINED,
      36/3.6/Wed Dec 15 07:51:41 CST 2010/JOINED,
      37/3.6/Wed Dec 15 07:51:41 CST 2010/JOINED,
      38/3.6/Wed Dec 15 07:51:41 CST 2010/JOINED,
      40/3.6/Wed Dec 15 08:06:00 CST 2010/JOINED,
      41/3.6/Wed Dec 15 08:22:59 CST 2010/JOINED,
      43/3.6/Wed Dec 15 08:59:44 CST 2010/JOINED,
      44/3.6/Wed Dec 15 09:01:17 CST 2010/JOINED,
      45/3.6/Wed Dec 15 09:01:35 CST 2010/JOINED,
      46/3.6/Wed Dec 15 09:01:41 CST 2010/JOINED,
      47/3.6/Wed Dec 15 09:02:44 CST 2010/JOINED
    )
)
	at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.onStartupTimeout(Grid.CDB:6)
	at com.tangosol.coherence.component.util.daemon.queueProcessor.Service.start(Service.CDB:28)
	at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.start(Grid.CDB:6)
	at com.tangosol.coherence.component.net.Cluster.onStart(Cluster.CDB:636)
	at com.tangosol.coherence.component.net.Cluster.start(Cluster.CDB:11)
	at com.tangosol.coherence.component.util.SafeCluster.startCluster(SafeCluster.CDB:3)
	at com.tangosol.coherence.component.util.SafeCluster.restartCluster(SafeCluster.CDB:7)
	at com.tangosol.coherence.component.util.SafeCluster.ensureRunningCluster(SafeCluster.CDB:26)
	at com.tangosol.coherence.component.util.SafeService.restartService(SafeService.CDB:22)
	at com.tangosol.coherence.component.util.SafeService.ensureRunningService(SafeService.CDB:39)
	at com.tangosol.coherence.component.util.safeService.SafeCacheService.ensureRunningCacheService(SafeCacheService.CDB:3)
	at com.tangosol.coherence.component.util.SafeNamedCache$CacheAction.run(SafeNamedCache.CDB:3)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:337)
	at com.tangosol.coherence.component.util.SafeNamedCache.restartNamedCache(SafeNamedCache.CDB:8)
	at com.tangosol.coherence.component.util.SafeNamedCache.ensureRunningNamedCache(SafeNamedCache.CDB:33)
	at com.tangosol.coherence.component.util.SafeNamedCache.getRunningNamedCache(SafeNamedCache.CDB:1)
	at com.tangosol.coherence.component.util.SafeNamedCache.putAll(SafeNamedCache.CDB:1)
	at rediquoteclient.CoherencePublisher$1.run(CoherencePublisher.java:105)
	at java.lang.Thread.run(Thread.java:619)
Comments
Locked Post
New comments cannot be posted to this locked post.
Post Details
Locked on Jan 12 2011
Added on Dec 15 2010
#coherence, #coherence-support
0 comments
970 views