Hi,
I have 2 node 11g R2 Cluster on RHEL5.WHen I am trying to start the instance, i am gettiing the below error:
[root@rac1 bin]# ./srvctl start instance -i dev1 -d dev
PRCR-1013 : Failed to start resource ora.dev.db
PRCR-1064 : Failed to start resource ora.dev.db on node rac1
ORA-00600: internal error code, arguments: [kcratr_scan_lastbwr], [], [], [], [], [], [], [], [], [], [], []
CRS-2674: Start of 'ora.dev.db' on 'rac1' failed
[root@rac1 bin]# ./srvctl status scan_listener
SCAN Listener LISTENER_SCAN1 is enabled
SCAN listener LISTENER_SCAN1 is running on node rac1
SCAN Listener LISTENER_SCAN2 is enabled
SCAN listener LISTENER_SCAN2 is running on node rac2
SCAN Listener LISTENER_SCAN3 is enabled
SCAN listener LISTENER_SCAN3 is running on node rac1
[root@rac1 bin]# ./srvctl config nodeapps -a
VIP exists.:rac1
VIP exists.: /rac1-vip/10.17.57.114/255.255.255.0/eth0
VIP exists.:rac2
VIP exists.: /rac2-vip/10.17.57.115/255.255.255.0/eth0
[root@rac1 bin]# ./srvctl config nodeapps
VIP exists.:rac1
VIP exists.: /rac1-vip/10.17.57.114/255.255.255.0/eth0
VIP exists.:rac2
VIP exists.: /rac2-vip/10.17.57.115/255.255.255.0/eth0
GSD exists.
ONS daemon exists. Local port 6100, remote port 6200
eONS daemon exists. Multicast port 17412, multicast IP address 234.62.213.92, listening port 2016
[root@rac2 bin]# ./crs_stop -all
CRS-2500: Cannot stop resource 'ora.gsd' as it is not running
CRS-2500: Cannot stop resource 'ora.oc4j' as it is not running
Attempting to stop `ora.eons` on member `rac2`
Attempting to stop `ora.LISTENER.lsnr` on member `rac1`
Attempting to stop `ora.eons` on member `rac1`
Attempting to stop `ora.LISTENER.lsnr` on member `rac2`
CRS-2789: Cannot stop resource 'ora.gsd' as it is not running on server 'rac2'
Attempting to stop `ora.CRS.dg` on member `rac2`
Attempting to stop `ora.DATA.dg` on member `rac2`
Attempting to stop `ora.FRA.dg` on member `rac2`
Attempting to stop `ora.registry.acfs` on member `rac2`
Attempting to stop `ora.ons` on member `rac2`
Attempting to stop `ora.CRS.dg` on member `rac1`
Attempting to stop `ora.DATA.dg` on member `rac1`
Attempting to stop `ora.FRA.dg` on member `rac1`
Attempting to stop `ora.registry.acfs` on member `rac1`
CRS-2789: Cannot stop resource 'ora.gsd' as it is not running on server 'rac1'
Stop of `ora.LISTENER.lsnr` on member `rac2` succeeded.
Stop of `ora.LISTENER.lsnr` on member `rac1` succeeded.
Attempting to stop `ora.ons` on member `rac1`
Attempting to stop `ora.rac1.vip` on member `rac1`
Attempting to stop `ora.LISTENER_SCAN1.lsnr` on member `rac1`
Stop of `ora.rac1.vip` on member `rac1` succeeded.
Stop of `ora.ons` on member `rac2` succeeded.
Attempting to stop `ora.rac2.vip` on member `rac2`
Attempting to stop `ora.LISTENER_SCAN2.lsnr` on member `rac2`
Attempting to stop `ora.LISTENER_SCAN3.lsnr` on member `rac2`
Stop of `ora.rac2.vip` on member `rac2` succeeded.
Stop of `ora.LISTENER_SCAN1.lsnr` on member `rac1` succeeded.
Attempting to stop `ora.scan1.vip` on member `rac1`
Stop of `ora.scan1.vip` on member `rac1` succeeded.
Stop of `ora.ons` on member `rac1` succeeded.
CRS-2789: Cannot stop resource 'ora.ons' as it is not running on server 'rac1'
Attempting to stop `ora.net1.network` on member `rac1`
Stop of `ora.net1.network` on member `rac1` succeeded.
Stop of `ora.LISTENER_SCAN2.lsnr` on member `rac2` succeeded.
Attempting to stop `ora.scan2.vip` on member `rac2`
Stop of `ora.scan2.vip` on member `rac2` succeeded.
Stop of `ora.LISTENER_SCAN3.lsnr` on member `rac2` succeeded.
Attempting to stop `ora.scan3.vip` on member `rac2`
Stop of `ora.scan3.vip` on member `rac2` succeeded.
Attempting to stop `ora.net1.network` on member `rac2`
Stop of `ora.net1.network` on member `rac2` succeeded.
Stop of `ora.registry.acfs` on member `rac1` succeeded.
Stop of `ora.registry.acfs` on member `rac2` succeeded.
Stop of `ora.FRA.dg` on member `rac2` succeeded.
Stop of `ora.eons` on member `rac2` succeeded.
Stop of `ora.CRS.dg` on member `rac1` succeeded.
Stop of `ora.eons` on member `rac1` succeeded.
Stop of `ora.DATA.dg` on member `rac2` succeeded.
Stop of `ora.CRS.dg` on member `rac2` succeeded.
Attempting to stop `ora.asm` on member `rac2`
Stop of `ora.DATA.dg` on member `rac1` succeeded.
Stop of `ora.FRA.dg` on member `rac1` succeeded.
Attempting to stop `ora.asm` on member `rac1`
CRS-2673: Attempting to stop 'ora.asm' on 'rac1'
ORA-15097: cannot SHUTDOWN ASM instance with connected client
CRS-2675: Stop of 'ora.asm' on 'rac1' failed
CRS-2673: Attempting to stop 'ora.asm' on 'rac2'
ORA-15097: cannot SHUTDOWN ASM instance with connected client
CRS-2675: Stop of 'ora.asm' on 'rac2' failed
CRS-0216: Could not stop resource 'ora.asm rac1 1'.
CRS-0216: Could not stop resource 'ora.asm rac2 1'.
CRS-0216: Could not stop resource 'ora.gsd'.
CRS-0216: Could not stop resource 'ora.gsd rac1 1'.
CRS-0216: Could not stop resource 'ora.gsd rac2 1'.
CRS-0216: Could not stop resource 'ora.oc4j'.
CRS-0216: Could not stop resource 'ora.ons rac1 1'.
[root@rac2 bin]# ./crs_stat -t
Name Type Target State Host
------------------------------------------------------------
ora.CRS.dg ora....up.type OFFLINE OFFLINE
ora.DATA.dg ora....up.type OFFLINE OFFLINE
ora.FRA.dg ora....up.type OFFLINE ONLINE rac2
ora....ER.lsnr ora....er.type OFFLINE OFFLINE
ora....N1.lsnr ora....er.type OFFLINE OFFLINE
ora....N2.lsnr ora....er.type OFFLINE OFFLINE
ora....N3.lsnr ora....er.type OFFLINE OFFLINE
ora.asm ora.asm.type OFFLINE ONLINE rac2
ora.dev.db ora....se.type OFFLINE OFFLINE
ora.eons ora.eons.type OFFLINE OFFLINE
ora.gsd ora.gsd.type OFFLINE OFFLINE
ora....network ora....rk.type OFFLINE OFFLINE
ora.oc4j ora.oc4j.type OFFLINE OFFLINE
ora.ons ora.ons.type OFFLINE OFFLINE
ora....SM1.asm application OFFLINE ONLINE rac1
ora....C1.lsnr application OFFLINE OFFLINE
ora.rac1.gsd application OFFLINE OFFLINE
ora.rac1.ons application OFFLINE OFFLINE
ora.rac1.vip ora....t1.type OFFLINE OFFLINE
ora....SM2.asm application OFFLINE ONLINE rac2
ora....C2.lsnr application OFFLINE OFFLINE
ora.rac2.gsd application OFFLINE OFFLINE
ora.rac2.ons application OFFLINE OFFLINE
ora.rac2.vip ora....t1.type OFFLINE OFFLINE
ora....ry.acfs ora....fs.type OFFLINE OFFLINE
ora.scan1.vip ora....ip.type OFFLINE OFFLINE
ora.scan2.vip ora....ip.type OFFLINE OFFLINE
ora.scan3.vip ora....ip.type OFFLINE OFFLINE
[root@rac2 bin]# ./crs_start -all
CRS-2501: Resource 'ora.gsd' is disabled
CRS-2501: Resource 'ora.oc4j' is disabled
CRS-2501: Resource 'ora.gsd' is disabled
CRS-2501: Resource 'ora.gsd' is disabled
Attempting to start `ora.registry.acfs` on member `rac2`
CRS-2664: Resource 'ora.asm' is already running on 'rac2'
Attempting to start `ora.registry.acfs` on member `rac1`
Attempting to start `ora.eons` on member `rac1`
Attempting to start `ora.eons` on member `rac2`
Attempting to start `ora.DATA.dg` on member `rac1`
Attempting to start `ora.FRA.dg` on member `rac1`
Attempting to start `ora.net1.network` on member `rac1`
Attempting to start `ora.net1.network` on member `rac2`
Attempting to start `ora.CRS.dg` on member `rac2`
Attempting to start `ora.CRS.dg` on member `rac1`
Start of `ora.CRS.dg` on member `rac2` succeeded.
Start of `ora.net1.network` on member `rac2` succeeded.
Attempting to start `ora.rac1.vip` on member `rac2`
Attempting to start `ora.scan2.vip` on member `rac2`
Attempting to start `ora.scan3.vip` on member `rac2`
Start of `ora.net1.network` on member `rac1` succeeded.
Attempting to start `ora.scan1.vip` on member `rac1`
Start of `ora.registry.acfs` on member `rac1` succeeded.
Start of `ora.registry.acfs` on member `rac2` succeeded.
Start of `ora.FRA.dg` on member `rac1` succeeded.
Start of `ora.scan3.vip` on member `rac2` succeeded.
Start of `ora.scan2.vip` on member `rac2` succeeded.
Attempting to start `ora.LISTENER_SCAN2.lsnr` on member `rac2`
Start of `ora.rac1.vip` on member `rac2` succeeded.
Start of `ora.scan1.vip` on member `rac1` succeeded.
Attempting to start `ora.LISTENER_SCAN1.lsnr` on member `rac1`
Attempting to start `ora.LISTENER_SCAN3.lsnr` on member `rac2`
Attempting to stop `ora.scan2.vip` on member `rac2`
Attempting to stop `ora.scan3.vip` on member `rac2`
Attempting to start `ora.ons` on member `rac1`
Attempting to start `ora.ons` on member `rac2`
Stop of `ora.scan2.vip` on member `rac2` succeeded.
Attempting to start `ora.scan2.vip` on member `rac1`
Stop of `ora.scan3.vip` on member `rac2` succeeded.
Attempting to start `ora.scan3.vip` on member `rac1`
Start of `ora.LISTENER_SCAN1.lsnr` on member `rac1` succeeded.
Attempting to stop `ora.rac1.vip` on member `rac2`
Stop of `ora.rac1.vip` on member `rac2` succeeded.
Attempting to start `ora.rac1.vip` on member `rac1`
Start of `ora.LISTENER_SCAN2.lsnr` on member `rac2` succeeded.
Start of `ora.LISTENER_SCAN3.lsnr` on member `rac2` succeeded.
Attempting to stop `ora.scan1.vip` on member `rac1`
Stop of `ora.scan1.vip` on member `rac1` succeeded.
Attempting to start `ora.scan1.vip` on member `rac2`
Start of `ora.ons` on member `rac1` succeeded.
Start of `ora.ons` on member `rac2` succeeded.
Start of `ora.rac1.vip` on member `rac1` succeeded.
Start of `ora.scan2.vip` on member `rac1` succeeded.
Attempting to start `ora.LISTENER.lsnr` on member `rac1`
Start of `ora.scan3.vip` on member `rac1` succeeded.
Attempting to stop `ora.LISTENER_SCAN2.lsnr` on member `rac2`
Attempting to stop `ora.LISTENER_SCAN3.lsnr` on member `rac2`
Stop of `ora.LISTENER_SCAN2.lsnr` on member `rac2` succeeded.
Attempting to start `ora.LISTENER_SCAN2.lsnr` on member `rac1`
Stop of `ora.LISTENER_SCAN3.lsnr` on member `rac2` succeeded.
Attempting to start `ora.LISTENER_SCAN3.lsnr` on member `rac1`
Start of `ora.LISTENER.lsnr` on member `rac1` succeeded.
Start of `ora.scan1.vip` on member `rac2` succeeded.
Start of `ora.LISTENER_SCAN2.lsnr` on member `rac1` succeeded.
Start of `ora.LISTENER_SCAN3.lsnr` on member `rac1` succeeded.
Attempting to stop `ora.LISTENER_SCAN1.lsnr` on member `rac1`
Stop of `ora.LISTENER_SCAN1.lsnr` on member `rac1` succeeded.
Attempting to start `ora.LISTENER_SCAN1.lsnr` on member `rac2`
Start of `ora.LISTENER_SCAN1.lsnr` on member `rac2` succeeded.
Start of `ora.CRS.dg` on member `rac1` succeeded.
Start of `ora.DATA.dg` on member `rac1` succeeded.
Start of `ora.eons` on member `rac2` succeeded.
Start of `ora.eons` on member `rac1` succeeded.
Attempting to start `ora.DATA.dg` on member `rac2`
Attempting to start `ora.rac2.vip` on member `rac2`
Attempting to start `ora.dev.db` on member `rac1`
Start of `ora.DATA.dg` on member `rac2` succeeded.
Start of `ora.rac2.vip` on member `rac2` succeeded.
Attempting to start `ora.LISTENER.lsnr` on member `rac2`
Start of `ora.LISTENER.lsnr` on member `rac2` succeeded.
Attempting to start `ora.dev.db` on member `rac2`
ORA-00600: internal error code, arguments: [kcratr_scan_lastbwr], [], [], [], [], [], [], [], [], [], [], []
Start of `ora.dev.db` on member `rac1` failed.
Attempting to stop `ora.dev.db` on member `rac1`
Stop of `ora.dev.db` on member `rac1` succeeded.
CRS-2632: There are no more servers to try to place resource 'ora.dev.db' on that would satisfy its placement policy
ORA-00600: internal error code, arguments: [kcratr_scan_lastbwr], [], [], [], [], [], [], [], [], [], [], []
Start of `ora.dev.db` on member `rac2` failed.
Attempting to stop `ora.dev.db` on member `rac2`
Stop of `ora.dev.db` on member `rac2` succeeded.
Attempting to stop `ora.LISTENER.lsnr` on member `rac2`
Stop of `ora.LISTENER.lsnr` on member `rac2` succeeded.
Attempting to stop `ora.rac2.vip` on member `rac2`
Stop of `ora.rac2.vip` on member `rac2` succeeded.
Attempting to stop `ora.DATA.dg` on member `rac2`
Stop of `ora.DATA.dg` on member `rac2` succeeded.
CRS-0223: Resource 'ora.asm rac2 1' has placement error.
CRS-0223: Resource 'ora.dev.db 1 1' has placement error.
CRS-0215: Could not start resource 'ora.dev.db 2 1'.
CRS-2660: Resource 'ora.gsd' or all of its instances are disabled
CRS-2660: Resource 'ora.oc4j' or all of its instances are disabled
CRS-2660: Resource 'ora.rac1.gsd' or all of its instances are disabled
CRS-2660: Resource 'ora.rac2.gsd' or all of its instances are disabled
[root@rac2 bin]#
alert log of rac2
[root@rac2 rac2]# tail -n 100 alertrac2.log
2013-01-16 13:56:10.825
[ohasd(4816)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
2013-01-16 13:56:10.889
[gpnpd(5269)]CRS-2328:GPNPD started on node rac2.
2013-01-16 13:56:15.858
[cssd(5325)]CRS-1713:CSSD daemon is started in clustered mode
[client(5480)]CRS-10001:ACFS-9327: Verifying ADVM/ACFS devices.
[client(5502)]CRS-10001:ACFS-9322: done.
2013-01-16 13:56:25.065
[cssd(5325)]CRS-1707:Lease acquisition for node rac2 number 2 completed
2013-01-16 13:56:25.098
[cssd(5325)]CRS-1605:CSSD voting file is online: ORCL:VOTES1; details in /raczone/grid_home/log/rac2/cssd/ocssd.log.
2013-01-16 13:56:40.502
[cssd(5325)]CRS-1601:CSSD Reconfiguration complete. Active nodes are rac1 rac2 .
2013-01-16 13:56:42.677
[ctssd(5547)]CRS-2407:The new Cluster Time Synchronization Service reference node is host rac1.
2013-01-16 13:56:42.703
[ctssd(5547)]CRS-2408:The clock on host rac2 has been updated by the Cluster Time Synchronization Service to be synchronous with the mean cluster time.
2013-01-16 13:56:42.980
[ctssd(5547)]CRS-2401:The Cluster Time Synchronization Service started on host rac2.
2013-01-16 13:56:43.012
[/raczone/grid_home/bin/oraagent.bin(5201)]CRS-5011:Check of resource "+ASM" failed: details at "(:CLSN00006:)" in "/raczone/grid_home/log/rac2/agent/ohasd/oraagent_grid/oraagent_grid.log"
2013-01-16 13:57:03.995
[/raczone/grid_home/bin/oraagent.bin(5201)]CRS-5011:Check of resource "+ASM" failed: details at "(:CLSN00006:)" in "/raczone/grid_home/log/rac2/agent/ohasd/oraagent_grid/oraagent_grid.log"
2013-01-16 13:57:15.713
[ctssd(5547)]CRS-2408:The clock on host rac2 has been updated by the Cluster Time Synchronization Service to be synchronous with the mean cluster time.
2013-01-16 13:57:31.719
[ctssd(5547)]CRS-2408:The clock on host rac2 has been updated by the Cluster Time Synchronization Service to be synchronous with the mean cluster time.
2013-01-16 13:57:50.328
[crsd(5747)]CRS-1012:The OCR service started on node rac2.
2013-01-16 13:57:50.435
[evmd(5562)]CRS-1401:EVMD started on node rac2.
2013-01-16 13:57:53.652
[crsd(5747)]CRS-1201:CRSD started on node rac2.
2013-01-16 13:58:01.718
[crsd(5747)]CRS-2772:Server 'rac2' has been assigned to pool 'Generic'.
2013-01-16 13:58:01.719
[crsd(5747)]CRS-2772:Server 'rac2' has been assigned to pool 'ora.dev'.
2013-01-16 14:00:44.597
[ctssd(5547)]CRS-2408:The clock on host rac2 has been updated by the Cluster Time Synchronization Service to be synchronous with the mean cluster time.
2013-01-16 14:01:00.604
[ctssd(5547)]CRS-2408:The clock on host rac2 has been updated by the Cluster Time Synchronization Service to be synchronous with the mean cluster time.
2013-01-16 14:01:21.547
[crsd(5747)]CRS-2772:Server 'rac1' has been assigned to pool 'Generic'.
2013-01-16 14:01:21.551
[crsd(5747)]CRS-2772:Server 'rac1' has been assigned to pool 'ora.dev'.
2013-01-16 14:03:25.490
[ctssd(5547)]CRS-2408:The clock on host rac2 has been updated by the Cluster Time Synchronization Service to be synchronous with the mean cluster time.
2013-01-16 14:03:41.496
[ctssd(5547)]CRS-2408:The clock on host rac2 has been updated by the Cluster Time Synchronization Service to be synchronous with the mean cluster time.
2013-01-16 14:11:42.321
[ctssd(5547)]CRS-2407:The new Cluster Time Synchronization Service reference node is host rac2.
2013-01-16 14:12:02.182
[cssd(5325)]CRS-1612:Network communication with node rac1 (1) missing for 50% of timeout interval. Removal of this node from cluster in 14.290 seconds
2013-01-16 14:12:09.198
[cssd(5325)]CRS-1611:Network communication with node rac1 (1) missing for 75% of timeout interval. Removal of this node from cluster in 7.280 seconds
2013-01-16 14:12:14.209
[cssd(5325)]CRS-1610:Network communication with node rac1 (1) missing for 90% of timeout interval. Removal of this node from cluster in 2.270 seconds
2013-01-16 14:12:16.488
[cssd(5325)]CRS-1632:Node rac1 is being removed from the cluster in cluster incarnation 253469533
2013-01-16 14:12:16.603
[cssd(5325)]CRS-1601:CSSD Reconfiguration complete. Active nodes are rac2 .
2013-01-16 14:12:16.696
[crsd(5747)]CRS-5504:Node down event reported for node 'rac1'.
2013-01-16 14:12:30.571
[crsd(5747)]CRS-2773:Server 'rac1' has been removed from pool 'Generic'.
2013-01-16 14:12:30.571
[crsd(5747)]CRS-2773:Server 'rac1' has been removed from pool 'ora.dev'.
2013-01-16 14:14:55.076
[cssd(5325)]CRS-1601:CSSD Reconfiguration complete. Active nodes are rac1 rac2 .
2013-01-16 14:15:28.630
[crsd(5747)]CRS-2772:Server 'rac1' has been assigned to pool 'Generic'.
2013-01-16 14:15:28.632
[crsd(5747)]CRS-2772:Server 'rac1' has been assigned to pool 'ora.dev'.
2013-01-16 14:25:22.629
[cssd(5325)]CRS-1625:Node rac1, number 1, was manually shut down
2013-01-16 14:25:22.998
[cssd(5325)]CRS-1601:CSSD Reconfiguration complete. Active nodes are rac2 .
2013-01-16 14:25:23.005
[crsd(5747)]CRS-5504:Node down event reported for node 'rac1'.
2013-01-16 14:25:23.006
[crsd(5747)]CRS-2773:Server 'rac1' has been removed from pool 'Generic'.
2013-01-16 14:25:23.006
[crsd(5747)]CRS-2773:Server 'rac1' has been removed from pool 'ora.dev'.
2013-01-16 14:26:25.301
[cssd(5325)]CRS-1601:CSSD Reconfiguration complete. Active nodes are rac1 rac2 .
2013-01-16 14:27:02.746
[crsd(5747)]CRS-2772:Server 'rac1' has been assigned to pool 'Generic'.
2013-01-16 14:27:02.746
[crsd(5747)]CRS-2772:Server 'rac1' has been assigned to pool 'ora.dev'.
2013-01-16 21:03:45.088
[crsd(5747)]CRS-2765:Resource 'ora.net1.network' has failed on server 'rac2'.
2013-01-16 21:03:54.732
[crsd(5747)]CRS-2765:Resource 'ora.net1.network' has failed on server 'rac1'.
2013-01-17 10:05:25.527
[crsd(5747)]CRS-2765:Resource 'ora.net1.network' has failed on server 'rac1'.
2013-01-17 10:05:25.970
[crsd(5747)]CRS-2765:Resource 'ora.net1.network' has failed on server 'rac2'.
2013-01-17 10:05:40.587
[cssd(5325)]CRS-1612:Network communication with node rac1 (1) missing for 50% of timeout interval. Removal of this node from cluster in 14.730 seconds
[root@rac2 rac2]#
alert log of rac1
[root@rac1 rac1]# tail -n 100 alertrac1.log
2013-01-16 14:24:58.057
[/raczone/grid_home/bin/orarootagent.bin(5457)]CRS-5822:Agent '/raczone/grid_home/bin/orarootagent_root' disconnected from server. Details at (:CRSAGF00117:) in /raczone/grid_home/log/rac1/agent/crsd/orarootagent_root/orarootagent_root.log.
2013-01-16 14:25:01.292
[ctssd(5171)]CRS-2405:The Cluster Time Synchronization Service on host rac1 is shutdown by user
[client(7353)]CRS-10001:Waiting for ASM to shutdown.
2013-01-16 14:25:22.627
[cssd(5124)]CRS-1603:CSSD on node rac1 shutdown by user.
2013-01-16 14:25:48.831
[ohasd(7461)]CRS-2112:The OLR service started on node rac1.
2013-01-16 14:25:51.112
[ohasd(7461)]CRS-8011:reboot advisory message from host: rac1, component: ag121731, with time stamp: L-2013-01-14-14:57:16.901
[ohasd(7461)]CRS-8013:reboot advisory message text: Rebooting after limit 28080 exceeded; disk timeout 28080, network timeout 23640, last heartbeat from CSSD at epoch seconds 1358155608.710, 28170 milliseconds ago based on invariant clock value of 7675224
2013-01-16 14:25:51.113
[ohasd(7461)]CRS-8017:location: /etc/oracle/lastgasp has 20 reboot advisory log files, 1 were announced and 0 errors occurred
2013-01-16 14:25:53.007
[ohasd(7461)]CRS-2772:Server 'rac1' has been assigned to pool 'Free'.
2013-01-16 14:26:00.147
[ohasd(7461)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
2013-01-16 14:26:03.350
[cssd(7684)]CRS-1713:CSSD daemon is started in clustered mode
[client(7794)]CRS-10001:ACFS-9327: Verifying ADVM/ACFS devices.
[client(7877)]CRS-10001:ACFS-9322: done.
2013-01-16 14:26:13.842
[cssd(7684)]CRS-1707:Lease acquisition for node rac1 number 1 completed
2013-01-16 14:26:13.885
[cssd(7684)]CRS-1605:CSSD voting file is online: ORCL:VOTES1; details in /raczone/grid_home/log/rac1/cssd/ocssd.log.
2013-01-16 14:26:25.258
[cssd(7684)]CRS-1601:CSSD Reconfiguration complete. Active nodes are rac1 rac2 .
2013-01-16 14:26:27.458
[ctssd(7919)]CRS-2407:The new Cluster Time Synchronization Service reference node is host rac2.
2013-01-16 14:26:28.277
[ctssd(7919)]CRS-2401:The Cluster Time Synchronization Service started on host rac1.
2013-01-16 14:26:51.860
[crsd(8072)]CRS-1012:The OCR service started on node rac1.
2013-01-16 14:26:55.017
[crsd(8072)]CRS-1201:CRSD started on node rac1.
2013-01-16 14:27:00.522
[ctssd(7919)]CRS-2408:The clock on host rac1 has been updated by the Cluster Time Synchronization Service to be synchronous with the mean cluster time.
2013-01-16 14:27:16.527
[ctssd(7919)]CRS-2408:The clock on host rac1 has been updated by the Cluster Time Synchronization Service to be synchronous with the mean cluster time.
2013-01-16 14:43:40.942
[ctssd(7919)]CRS-2408:The clock on host rac1 has been updated by the Cluster Time Synchronization Service to be synchronous with the mean cluster time.
2013-01-16 14:44:04.951
[ctssd(7919)]CRS-2408:The clock on host rac1 has been updated by the Cluster Time Synchronization Service to be synchronous with the mean cluster time.
2013-01-16 14:56:13.621
[ctssd(7919)]CRS-2408:The clock on host rac1 has been updated by the Cluster Time Synchronization Service to be synchronous with the mean cluster time.
2013-01-16 14:56:29.627
[ctssd(7919)]CRS-2408:The clock on host rac1 has been updated by the Cluster Time Synchronization Service to be synchronous with the mean cluster time.
2013-01-17 10:05:37.600
[cssd(7684)]CRS-1612:Network communication with node rac2 (2) missing for 50% of timeout interval. Removal of this node from cluster in 14.390 seconds
2013-01-17 10:05:37.844
[cssd(7684)]CRS-1615:No I/O has completed after 50% of the maximum interval. Voting file ORCL:VOTES1 will be considered not functional in 10020 milliseconds
2013-01-17 10:05:41.828
[cssd(7684)]CRS-1614:No I/O has completed after 75% of the maximum interval. Voting file ORCL:VOTES1 will be considered not functional in 6030 milliseconds
2013-01-17 10:05:44.713
[cssd(7684)]CRS-1611:Network communication with node rac2 (2) missing for 75% of timeout interval. Removal of this node from cluster in 7.280 seconds
2013-01-17 10:06:27.587
[ctssd(7919)]CRS-2408:The clock on host rac1 has been updated by the Cluster Time Synchronization Service to be synchronous with the mean cluster time.
2013-01-17 10:06:43.591
[ctssd(7919)]CRS-2408:The clock on host rac1 has been updated by the Cluster Time Synchronization Service to be synchronous with the mean cluster time.
2013-01-17 10:22:51.994
[ctssd(7919)]CRS-2408:The clock on host rac1 has been updated by the Cluster Time Synchronization Service to be synchronous with the mean cluster time.
2013-01-17 10:45:56.525
[ctssd(7919)]CRS-2408:The clock on host rac1 has been updated by the Cluster Time Synchronization Service to be synchronous with the mean cluster time.
2013-01-17 10:52:28.656
[ctssd(7919)]CRS-2408:The clock on host rac1 has been updated by the Cluster Time Synchronization Service to be synchronous with the mean cluster time.
2013-01-17 10:52:44.663
[ctssd(7919)]CRS-2408:The clock on host rac1 has been updated by the Cluster Time Synchronization Service to be synchronous with the mean cluster time.
2013-01-17 10:56:04.766
[ctssd(7919)]CRS-2408:The clock on host rac1 has been updated by the Cluster Time Synchronization Service to be synchronous with the mean cluster time.
2013-01-17 11:00:30.001
[ctssd(7919)]CRS-2408:The clock on host rac1 has been updated by the Cluster Time Synchronization Service to be synchronous with the mean cluster time.
2013-01-17 11:00:46.024
[ctssd(7919)]CRS-2408:The clock on host rac1 has been updated by the Cluster Time Synchronization Service to be synchronous with the mean cluster time.
2013-01-17 11:25:29.077
[ctssd(7919)]CRS-2408:The clock on host rac1 has been updated by the Cluster Time Synchronization Service to be synchronous with the mean cluster time.
2013-01-17 11:28:25.248
[ctssd(7919)]CRS-2408:The clock on host rac1 has been updated by the Cluster Time Synchronization Service to be synchronous with the mean cluster time.
2013-01-17 11:28:41.253
[ctssd(7919)]CRS-2408:The clock on host rac1 has been updated by the Cluster Time Synchronization Service to be synchronous with the mean cluster time.
2013-01-17 11:36:41.512
[ctssd(7919)]CRS-2408:The clock on host rac1 has been updated by the Cluster Time Synchronization Service to be synchronous with the mean cluster time.
2013-01-17 11:36:57.519
[ctssd(7919)]CRS-2408:The clock on host rac1 has been updated by the Cluster Time Synchronization Service to be synchronous with the mean cluster time.
2013-01-17 11:40:33.660
[ctssd(7919)]CRS-2408:The clock on host rac1 has been updated by the Cluster Time Synchronization Service to be synchronous with the mean cluster time.
2013-01-17 11:40:49.670
[ctssd(7919)]CRS-2408:The clock on host rac1 has been updated by the Cluster Time Synchronization Service to be synchronous with the mean cluster time.
2013-01-17 11:47:38.670
[ctssd(7919)]CRS-2408:The clock on host rac1 has been updated by the Cluster Time Synchronization Service to be synchronous with the mean cluster time.
2013-01-17 11:47:54.676
[ctssd(7919)]CRS-2408:The clock on host rac1 has been updated by the Cluster Time Synchronization Service to be synchronous with the mean cluster time.
2013-01-17 11:48:10.768
[ctssd(7919)]CRS-2408:The clock on host rac1 has been updated by the Cluster Time Synchronization Service to be synchronous with the mean cluster time.
2013-01-17 11:49:15.539
[ctssd(7919)]CRS-2408:The clock on host rac1 has been updated by the Cluster Time Synchronization Service to be synchronous with the mean cluster time.
2013-01-17 11:50:35.707
[ctssd(7919)]CRS-2408:The clock on host rac1 has been updated by the Cluster Time Synchronization Service to be synchronous with the mean cluster time.
2013-01-17 11:50:51.713
[ctssd(7919)]CRS-2408:The clock on host rac1 has been updated by the Cluster Time Synchronization Service to be synchronous with the mean cluster time.
[root@rac1 rac1]#
It seems from the alert log that due to heatbeat/diskbeat, CSSD has evicted the node from cluster.
Can anynody suggest me for the same....i