Integration

Announcement

For appeals, questions and feedback about Oracle Forums, please email oracle-forums-moderators_us@oracle.com. Technical questions should be asked in the appropriate category. Thank you!

coherence node prompt Aborting connection, network error with Exalogic system

User621063-OracleMay 14 2015 — edited May 14 2015

Hi,

I need some help to resolve a customer specific Coherence issue which we can not reproduce internally.

Thanks in advance.

Regards,

Jasper

There are two nodes in cluster with JVM 1.7.0.75. The two nodes run on the same Exalogic box.

The 1st node is storage enabled node, it crashed after started, with strange log message which is not part of our code.

[Fri May 8 09:59:16 2015][error][bus ] [0x7f5c2c1ce5a0][0x7f5c2c763570] Aborting connection, network error.

The 2nd node is storage disabled node, it crashed at time starting up the cache service and seems to be able join with the 1st node briefly.

The following are the logs by the customer.

====1st node log====

2015-05-08 09:59:07.120 CLT INFO - - - - Oracle Coherence 12.1.2.0.1 (thread=main, member=1):

Services

(

ClusterService{Name=Cluster, State=(SERVICE_STARTED, STATE_JOINED), Id=0, Version=12.1.2, OldestMemberId=1}

InvocationService{Name=Management, State=(SERVICE_STARTED), Id=1, Version=12.1.2, OldestMemberId=1}

PartitionedCache{Name=BRMDistributedCache, State=(SERVICE_STARTED), LocalStorage=enabled, PartitionCount=4093, BackupCount=1, AssignedPartitions=4093, BackupPartitions=0}

ReplicatedCache{Name=ReplicatedCache, State=(SERVICE_STARTED), Id=3, Version=12.1.2, OldestMemberId=1}

InvocationService{Name=InvocationService, State=(SERVICE_STARTED), Id=4, Version=12.1.2, OldestMemberId=1}

PartitionedCache{Name=XRefDistributedCache, State=(SERVICE_STARTED), LocalStorage=enabled, PartitionCount=257, BackupCount=1, AssignedPartitions=257, BackupPartitions=0}

PartitionedCache{Name=PoidIdDistributedCache, State=(SERVICE_STARTED), LocalStorage=enabled, PartitionCount=257, BackupCount=1, AssignedPartitions=257, BackupPartitions=0}

)

Started DefaultCacheServer...

[Fri May 8 09:59:16 2015][error][bus ] [0x7f5c2c1ce5a0][0x7f5c2c763570] Aborting connection, network error.

====2nd node log====

2015-05-08 09:59:42.467 CLT ERROR - - - - Oracle Coherence 12.1.2.0.1 (thread=main, member=2): Error while starting service "BRMDistributedCache": com.tangosol.net.RequestTimeoutException: Timeout during service start: ServiceInfo(Id=2, Name=BRMDistributedCache, Type=DistributedCache

MemberSet=ServiceMemberSet(

OldestMember=Member(Id=1, Timestamp=2015-05-08 09:58:54.856, Address=171.16.0.17:18088, MachineId=62285, Location=site:,machine:ipoib-virt-admin-171-16-0-17,process:26483,member:ecs1, Role=OracleChargingLauncher)

ActualMemberSet=MemberSet(Size=2

Member(Id=1, Timestamp=2015-05-08 09:58:54.856, Address=171.16.0.17:18088, MachineId=62285, Location=site:,machine:ipoib-virt-admin-171-16-0-17,process:26483,member:ecs1, Role=OracleChargingLauncher)

Member(Id=2, Timestamp=2015-05-08 09:59:08.727, Address=171.16.0.17:18090, MachineId=62285, Location=site:,machine:ipoib-virt-admin-171-16-0-17,process:27078,member:configLoader, Role=OracleConfigLoaderLauncher)

)

MemberId|ServiceVersion|ServiceJoined|MemberState|EndPoint

1|12.1.2|2015-05-08 09:59:01.244|JOINED|imb://171.16.0.17:31006,

2|12.1.2|2015-05-08 09:59:12.344|JOINING|imb://171.16.0.17:31009

)

at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.onStartupTimeout(Grid.CDB:3)

at com.tangosol.coherence.component.util.daemon.queueProcessor.Service.start(Service.CDB:28)

at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.start(Grid.CDB:6)

at com.tangosol.coherence.component.util.SafeService.startService(SafeService.CDB:53)

at com.tangosol.coherence.component.util.safeService.SafeCacheService.startService(SafeCacheService.CDB:5)

at com.tangosol.coherence.component.util.SafeService.ensureRunningService(SafeService.CDB:27)

at com.tangosol.coherence.component.util.SafeService.start(SafeService.CDB:14)

at com.tangosol.net.DefaultConfigurableCacheFactory.startService(DefaultConfigurableCacheFactory.java:1218)

at com.tangosol.net.DefaultConfigurableCacheFactory.ensureServiceInternal(DefaultConfigurableCacheFactory.java:1204)

at com.tangosol.net.DefaultConfigurableCacheFactory.ensureService(DefaultConfigurableCacheFactory.java:1035)

at com.tangosol.net.DefaultConfigurableCacheFactory.ensureCache(DefaultConfigurableCacheFactory.java:1017)

at com.tangosol.net.DefaultConfigurableCacheFactory.configureCache(DefaultConfigurableCacheFactory.java:1411)

at com.tangosol.net.DefaultConfigurableCacheFactory.ensureCache(DefaultConfigurableCacheFactory.java:355)

at com.tangosol.net.CacheFactory.getCache(CacheFactory.java:205)

at com.tangosol.net.CacheFactory.getCache(CacheFactory.java:182)

at oracle.communication.brm.charging.util.coherence.internal.CoherenceTemplateImpl.getCache(CoherenceTemplateImpl.java:452)

at oracle.communication.brm.charging.util.coherence.internal.CoherenceTemplateImpl.registerMapListener(CoherenceTemplateImpl.java:814)

at oracle.communication.brm.charging.appconfiguration.internal.coherence.CoherenceConfigurationBeanRepository.<init>(CoherenceConfigurationBeanRepository.java:85)

at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)

at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)

at java.lang.reflect.Constructor.newInstance(Constructor.java:526)

at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:147)

at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:125)

at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:270)

at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1133)

at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1036)

at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:505)

at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:476)

at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:302)

at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:229)

at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:298)

at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:193)

at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:328)

at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:108)

at org.springframework.beans.factory.support.ConstructorResolver.resolveConstructorArguments(ConstructorResolver.java:650)

at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:446)

at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1113)

at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1008)

at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:505)

at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:476)

at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:302)

at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:229)

at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:298)

at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:193)

at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:725)

at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:757)

at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:480)

at org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:139)

at org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:93)

at oracle.communication.brm.charging.migration.config.ConfigLoaderLauncher.main(ConfigLoaderLauncher.java:60)

2015-05-08 09:59:42.467 CLT ERROR - - - - Oracle Coherence 12.1.2.0.1 (thread=DistributedCache:BRMDistributedCache, member=2): validatePolls: This service timed-out due to unanswered handshake request. Manual intervention is required to stop the members that have not responded to this Poll

{

PollId=1, active

InitTimeMillis=2015-05-08 09:59:12.457

Service=BRMDistributedCache (2)

Message=MemberWelcomeRequest

RespondedMemberSet=[]

LeftMemberSet=[]

RemainingMemberSet=[1]

}

Request=Message "MemberWelcomeRequest"

{

FromMember=Member(Id=2, Timestamp=2015-05-08 09:59:08.727, Address=171.16.0.17:18090, MachineId=62285, Location=site:,machine:ipoib-virt-admin-171-16-0-17,process:27078,member:configLoader, Role=OracleConfigLoaderLauncher)

FromMessageId=0

MessagePartCount=0

PendingCount=0

Delivered

MessageType=-1

ToPollId=0

Poll=null

Packets

{

}

Service=PartitionedCache{Name=BRMDistributedCache, State=(SERVICE_STOPPED), Not initialized}

ToMemberSet=MemberSet(Size=1

Member(Id=1, Timestamp=2015-05-08 09:58:54.856, Address=171.16.0.17:18088, MachineId=62285, Location=site:,machine:ipoib-virt-admin-171-16-0-17,process:26483,member:ecs1, Role=OracleChargingLauncher)

)

NotifySent=false

}

Remaining members info:

{

Connection {Peer=imb://171.16.0.17:31006, Service=BRMDistributedCache, Member=Member(Id=1, Timestamp=2015-05-08 09:58:54.856, Address=171.16.0.17:18088, MachineId=62285, Location=site:,machine:ipoib-virt-admin-171-16-0-17,process:26483,member:ecs1, Role=OracleChargingLauncher), Not established, State=DISCONNECTED(java.lang.Exception: Failed network operation)}

}

2015-05-08 09:59:42.469 CLT WARN - - - - Exception encountered during context initialization - cancelling refresh attempt

org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'oracle.communication.brm.charging.balance.internal.coherence.CoherenceBalanceSerializer#4' defined in class path resource [META-INF/spring/balance-context.xml]: Cannot resolve reference to bean 'appConfigurationRepository' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'appConfigurationRepository' defined in class path resource [META-INF/spring/appconfiguration-context.xml]: Instantiation of bean failed; nested exception is org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [oracle.communication.brm.charging.appconfiguration.internal.coherence.CoherenceConfigurationBeanRepository]: Constructor threw exception; nested exception is com.tangosol.net.RequestTimeoutException: Timeout during service start: ServiceInfo(Id=2, Name=BRMDistributedCache, Type=DistributedCache

MemberSet=ServiceMemberSet(

ActualMemberSet=MemberSet(Size=2

Member(Id=1, Timestamp=2015-05-08 09:58:54.856, Address=171.16.0.17:18088, MachineId=62285, Location=site:,machine:ipoib-virt-admin-171-16-0-17,process:26483,member:ecs1, Role=OracleChargingLauncher)

)

MemberId|ServiceVersion|ServiceJoined|MemberState|EndPoint

1|12.1.2|2015-05-08 09:59:01.244|JOINED|imb://171.16.0.17:31006,

2|12.1.2|2015-05-08 09:59:12.344|JOINING|imb://171.16.0.17:31009

)

at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:336)