We completed database build in Aug 2010
We complete PSU patching in Jan ending
Feb 4th the database crashed
We cannot start ASM on node1
ASM starts good on node2 and node3 but node1 cannot join
If ASM is down on node2, node3 then we can start ASM node1
Reconfiguration started (old inc 0, new inc 6)
ASM instance
List of nodes:
0 1 2
Global Resource Directory frozen
* allocate domain 0, invalid = TRUE
Communication channels reestablished
* allocate domain 1, invalid = TRUE
* allocate domain 2, invalid = TRUE
Mon Mar 01 16:53:00 2010
Trace dumping is performing id=[cdmp_20100301165301]
Mon Mar 01 16:53:55 2010
ERROR: LMD0 (ospid: 274638) detects an idle connection to instance 2
Mon Mar 01 16:54:44 2010
Errors in file /oradb/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_lmon_860280.trc (incident=116865):
ORA-29740: evicted by member 1, group incarnation 8
Incident details in: /oradb/oracle/diag/asm/+asm/+ASM1/incident/incdir_116865/+ASM1_lmon_860280_i116865.trc
Errors in file /oradb/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_lmon_860280.trc:
ORA-29740: evicted by member 1, group incarnation 8
LMON (ospid: 860280): terminating the instance due to error 29740
Mon Mar 01 16:54:46 2010
System state dump is made for local instance
Errors in file /oradb/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_diag_614488.trc (incident=116833):
ORA-29740: evicted by member , group incarnation
Incident details in: /oradb/oracle/diag/asm/+asm/+ASM1/incident/incdir_116833/+ASM1_diag_614488_i116833.trc
Mon Mar 01 16:54:46 2010
ORA-1092 : opitsk aborting process
Errors in file /oradb/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_diag_614488.trc:
ORA-29740: evicted by member , group incarnation
Trace dumping is performing id=[cdmp_20100301165446]
Instance terminated by LMON, pid = 860280
Another thing we found that when we start ASM on node1, the cluster interconnect hangs when we try to ping
We did modify the cluster_interconnect parameter to try to start using public interface but the issued remained the same and we were not able to ping public interface
The crs is fine
$ crs_stat -t
Name Type Target State Host
------------------------------------------------------------
ora....p1.inst application ONLINE OFFLINE
ora....p2.inst application ONLINE ONLINE noden2
ora....p3.inst application ONLINE ONLINE noden3
ora....1p2.srv application ONLINE ONLINE noden2
ora....1p3.srv application ONLINE ONLINE noden3
ora.....net.cs application ONLINE ONLINE noden1
ora.appl.db application ONLINE ONLINE noden1
ora....SM1.asm application ONLINE OFFLINE
ora....N1.lsnr application ONLINE ONLINE noden1
ora....8n1.gsd application ONLINE ONLINE noden1
ora....8n1.ons application ONLINE ONLINE noden1
ora....8n1.vip application ONLINE ONLINE noden1
ora....SM2.asm application ONLINE ONLINE noden2
ora....N2.lsnr application ONLINE ONLINE noden2
ora....8n2.gsd application ONLINE ONLINE noden2
ora....8n2.ons application ONLINE ONLINE noden2
ora....8n2.vip application ONLINE ONLINE noden2
ora....SM3.asm application ONLINE ONLINE noden3
ora....N3.lsnr application ONLINE ONLINE noden3
ora....8n3.gsd application ONLINE ONLINE noden3
ora....8n3.ons application ONLINE ONLINE noden3
ora....8n3.vip application ONLINE ONLINE noden3
Any inpts can help