Hello Guru's
We encountered a weird closing of DB instance in our RAC. And when we checked the alert logs, it gave us Global Resource directory frozen.
can anyone help us in identifying the possible root cause of this.
Thanks in advance
Oracle RAC 11g 11.0.2.3 patch set 0.7
alertNode1.log
3.268.826611443
Thu Mar 03 08:34:39 2016
Archived Log entry 87141 added for thread 1 sequence 19439 ID 0x50bc329e dest 1:
Thu Mar 03 09:09:13 2016
Thread 1 advanced to log sequence 19441 (LGWR switch)
Current log# 1 seq# 19441 mem# 0: +FLASHDG/orcl/onlinelog/group_1.304.826611435
Current log# 1 seq# 19441 mem# 1: +DATADG/orcl/onlinelog/group_1.261.826611437
Thu Mar 03 09:09:16 2016
Archived Log entry 87142 added for thread 1 sequence 19440 ID 0x50bc329e dest 1:
Thu Mar 03 09:14:45 2016
Reconfiguration started (old inc 50, new inc 52)
List of instances:
1 (myinst: 1)
Global Resource Directory frozen
* dead instance detected - domain 0 invalid = TRUE
Communication channels reestablished
Master broadcasted resource hash value bitmaps
Non-local Process blocks cleaned out
Thu Mar 03 09:14:45 2016
LMS 1: 5 GCS shadows cancelled, 4 closed, 0 Xw survived
Thu Mar 03 09:14:45 2016
LMS 0: 3 GCS shadows cancelled, 0 closed, 0 Xw survived
Set master node info
Submitted all remote-enqueue requests
Dwn-cvts replayed, VALBLKs dubious
All grantable enqueues granted
Post SMON to start 1st pass IR
Thu Mar 03 09:14:45 2016
Instance recovery: looking for dead threads
Instance recovery: lock domain invalid but no dead threads
Submitted all GCS remote-cache requests
Post SMON to start 1st pass IR
Fix write in gcs resources
Reconfiguration complete
Thu Mar 03 09:14:47 2016
minact-scn: Master considers inst:2 dead
Thu Mar 03 09:15:01 2016
Reconfiguration started (old inc 52, new inc 54)
List of instances:
1 2 (myinst: 1)
Global Resource Directory frozen
Communication channels reestablished
Master broadcasted resource hash value bitmaps
Non-local Process blocks cleaned out
Thu Mar 03 09:15:01 2016
LMS 1: 0 GCS shadows cancelled, 0 closed, 0 Xw survived
Thu Mar 03 09:15:01 2016
LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived
Set master node info
Submitted all remote-enqueue requests
Dwn-cvts replayed, VALBLKs dubious
All grantable enqueues granted
Thu Mar 03 09:15:02 2016
minact-scn: Master returning as live inst:2 has inc# mismatch instinc:0 cur:54 errcnt:0
Submitted all GCS remote-cache requests
Fix write in gcs resources
Reconfiguration started (old inc 54, new inc 56)
List of instances:
1 2 (myinst: 1)
Nested reconfiguration detected.
Global Resource Directory frozen
Communication channels reestablished
Master broadcasted resource hash value bitmaps
Non-local Process blocks cleaned out
LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived
LMS 1: 0 GCS shadows cancelled, 0 closed, 0 Xw survived
Set master node info
Submitted all remote-enqueue requests
Dwn-cvts replayed, VALBLKs dubious
All grantable enqueues granted
Thu Mar 03 09:15:13 2016
Submitted all GCS remote-cache requests
Fix write in gcs resources
Reconfiguration complete
Thu Mar 03 09:16:16 2016
System state dump requested by (instance=1, osid=3846 (CKPT)), summary=[abnormal instance termination].
Thu Mar 03 09:16:16 2016
CKPT (ospid: 3846): terminating the instance
Thu Mar 03 09:16:16 2016
opiodr aborting process unknown ospid (29792) as a result of ORA-1092
Thu Mar 03 09:16:16 2016
opiodr aborting process unknown ospid (29854) as a result of ORA-1092
Thu Mar 03 09:16:16 2016
ORA-1092 : opitsk aborting process
Thu Mar 03 09:16:16 2016
opiodr aborting process unknown ospid (29164) as a result of ORA-1092
Thu Mar 03 09:16:16 2016
ORA-1092 : opitsk aborting process
System State dumped to trace file /u01/app/oracle/diag/rdbms/orcl/orcl1/trace/orcl1_diag_3812.trc
Thu Mar 03 09:16:16 2016
opiodr aborting process unknown ospid (3921) as a result of ORA-1092
Thu Mar 03 09:16:17 2016
ORA-1092 : opitsk aborting process
Thu Mar 03 09:16:17 2016
ORA-1092 : opitsk aborting process
Thu Mar 03 09:16:17 2016
License high water mark = 1317
Instance terminated by CKPT, pid = 3846
USER (ospid: 762): terminating the instance
Instance terminated by USER, pid = 762
Thu Mar 03 09:16:25 2016
Starting ORACLE instance (normal)
LICENSE_MAX_SESSION = 0
LICENSE_SESSIONS_WARNING = 0
Private Interface 'lan901:801' configured from GPnP for use as a private interconnect.
[name='lan901:801', type=1, ip=169.254.139.159, mac=b4-b5-2f-00-93-f2, net=169.254.0.0/16, mask=255.255.0.0, use=haip:cluster_interconnect/62]
Public Interface 'lan900' configured from GPnP for use as a public interface.
[name='lan900', type=1, ip=10.244.2.11, mac=b4-b5-2f-00-93-e6, net=10.244.2.0/24, mask=255.255.255.0, use=public/1]
Public Interface 'lan900:801' configured from GPnP for use as a public interface.
[name='lan900:801', type=1, ip=10.244.2.13, mac=b4-b5-2f-00-93-e6, net=10.244.2.0/24, mask=255.255.255.0, use=public/1]
Public Interface 'lan900:805' configured from GPnP for use as a public interface.
[name='lan900:805', type=1, ip=10.244.2.15, mac=b4-b5-2f-00-93-e6, net=10.244.2.0/24, mask=255.255.255.0, use=public/1]
Public Interface 'lan900:804' configured from GPnP for use as a public interface.
[name='lan900:804', type=1, ip=10.244.2.16, mac=b4-b5-2f-00-93-e6, net=10.244.2.0/24, mask=255.255.255.0, use=public/1]
Picked latch-free SCN scheme 3
Autotune of undo retention is turned on.
LICENSE_MAX_USERS = 0
SYS auditing is enabled
Starting up: