Oracle Database Discussions

Dear all,

Our company use rac 10g (10.1.0.4) on linux ES3,we have two nodes,recently node one
always reboot,I guest maybe dead lock
,can anyone help me how to solve this problem,thank you~

This is Error message "KTSMG_UPDATE_MQL() MMNL absent for 4294967293 secs; Foregrounds taking over"


This is node one error,

============================================
alter log
============================================
ARCH: Connecting to console port...
Committing creation of archivelog '/u01/arch/1_28179_597862062.dbf'
Thu Oct 9 10:00:44 2008
ARC1: Completed archiving thread 1 sequence 28179
(3737983174-3738502275) (yoydb1)
ARCH: Connecting to console port...
Thu Oct 9 10:28:53 2008
KTSMG_UPDATE_MQL(): MMNL absent for 4294967293 secs; Foregrounds
taking over
Thu Oct 9 10:32:00 2008
KTSMG_UPDATE_MQL(): MMNL absent for 4294967290 secs; Foregrounds
taking over
Thu Oct 9 10:41:22 2008
Error: KGXGN aborts the instance (6)
Thu Oct 9 10:41:22 2008
Errors in file /oracle/admin/yoydb/bdump/yoydb1_lmon_2460.trc:
ORA-29702: error occurred in Cluster Group Service operation
LMON: terminating instance due to error 29702
Thu Oct 9 10:41:22 2008
Errors in file /oracle/admin/yoydb/bdump/yoydb1_lms0_2464.trc:
ORA-29702: error occurred in Cluster Group Service operation
Thu Oct 9 10:41:22 2008
Errors in file /oracle/admin/yoydb/bdump/yoydb1_lms1_2466.trc:
ORA-29702: error occurred in Cluster Group Service operation
Thu Oct 9 10:41:22 2008
Errors in file /oracle/admin/yoydb/bdump/yoydb1_lmd0_2462.trc:
ORA-29702: error occurred in Cluster Group Service operation
Thu Oct 9 10:41:22 2008
System state dump is made for local instance
Thu Oct 9 10:41:24 2008
Trace dumping is performing id=[cdmp_20081009104122]
Thu Oct 9 11:18:11 2008

====================================================================
ocssd1.log
====================================================================

2008-10-09 10:34:49.317 [77296256] >TRACE: clssgmDeleteClientListener: cleanup for proc(0x200000000892c9d0) con(0x200000000892a7b0) pid()
2008-10-09 10:37:08.102 [77296256] >TRACE: clssgmClientConnectMsg: Connect from con(0x200000000892a7b0), proc(0x200000000892c9d0) pid()
2008-10-09 10:37:31.424 [77296256] >TRACE: clsc_receive: (0x200000000892a2a0) Connection failed, transport error (507, 0, 0)

2008-10-09 10:37:31.424 [77296256] >TRACE: clscreceive: (0x200000000892a7b0) Physical connection (0x200000000892a2a0) not active, rc 11

2008-10-09 10:37:31.424 [77296256] >TRACE: clssgmDeleteClientListener: cleanup for proc(0x200000000892c9d0) con(0x200000000892a7b0) pid()
2008-10-09 10:40:09.661 [77296256] >TRACE: clssgmClientConnectMsg: Connect from con(0x200000000892a8e0), proc(0x200000000892cb10) pid()
2008-10-09 10:41:12.338 [142766720] >ERROR: clssnmDiskPingMonitorThread: voting device access hanging (-100663195 miliseconds)
2008-10-09 10:41:12.552 [142766720] >TRACE: clssscctx: dump of 0x0x6000000000024270, len 4288
2008-10-09 10:41:12.554 [142766720] >TRACE: 0x0x6000000000024270 b0 5f 0c 00 00 00 00 60 - 50 cb 05 00 00 00 00 60 ._.....`P......`
2008-10-09 10:41:12.554 [142766720] >TRACE: 0x0x6000000000024280 80 ab 0c 02 00 00 00 20 - 10 40 02 00 00 00 00 60 ....... .@.....`
2008-10-09 10:41:12.554 [142766720] >TRACE: 0x0x6000000000024290 50 c6 02 00 00 00 00 60 - f0 3f 06 00 00 00 00 60 P......`.?.....`
2008-10-09 10:41:12.554 [142766720] >TRACE: 0x0x60000000000242a0 00 00 70 00 00 00 00 00 - 70 42 02 00 00 00 00 60 ..p.....pB.....`
2008-10-09 10:41:12.554 [142766720] >TRACE: 0x0x60000000000242b0 00 00 00 00 00 00 00 00 - 00 00 00 00 00 00 00 00 ................
2008-10-09 10:41:12.554 [142766720] >TRACE: 0x0x60000000000242c0 00 00 00 00 00 00 00 00 - 4d 61 69 6e 00 00 00 00 ........Main....
2008-10-09 10:41:12.554 [142766720] >TRACE: 0x0x60000000000242d0 00 00 00 00 00 00 00 00 - 00 00 00 00 00 00 00 00 ................
.........................
.........................
.........................
.........................
.........................
--- DUMP GROCK STATE DB ---
----------
type 2, Id 6, Name = (DBYOYDB)
flags: 0x0
grant: count=0, type 0, wait 0
Member Count =2, master 1
. . . . .
memberNo =1, seq 1
flags = 0x0, granted 0
refCnt = 0
nodeNum = 2, nodeBirth 1
privateDataSize = 96
publicDataSize = 12
. . . . .
memberNo =0, seq 9
flags = 0x0, granted 0
refCnt = 13
nodeNum = 1, nodeBirth 9
privateDataSize = 96
publicDataSize = 12
----------
----------
type 2, Id 5, Name = (DGYOYDB)
flags: 0x0
grant: count=0, type 0, wait 0
Member Count =2, master 1
. . . . .
memberNo =1, seq 1
flags = 0x0, granted 0
refCnt = 0
nodeNum = 2, nodeBirth 1
privateDataSize = 64
publicDataSize = 11
. . . . .
memberNo =0, seq 9
flags = 0x0, granted 0
refCnt = 1
nodeNum = 1, nodeBirth 9
privateDataSize = 64
publicDataSize = 11
----------
----------
type 2, Id 7, Name = (DAALL_DB)
flags: 0x0
grant: count=0, type 0, wait 0
Member Count =2, master 1
. . . . .
memberNo =1, seq 1
flags = 0x100, granted 0
refCnt = 0
nodeNum = 2, nodeBirth 1
privateDataSize = 0
publicDataSize = 15
. . . . .
memberNo =0, seq 9
flags = 0x100, granted 0
refCnt = 1
nodeNum = 1, nodeBirth 9
privateDataSize = 0
publicDataSize = 15
----------
----------
type 2, Id 1, Name = (ocr_crs)
flags: 0x0
grant: count=0, type 0, wait 0
Member Count =2, master 2
. . . . .
memberNo =2, seq 29
flags = 0x0, granted 0
refCnt = 0
nodeNum = 2, nodeBirth 1
privateDataSize = 24
publicDataSize = 24
. . . . .
memberNo =1, seq 35
flags = 0x0, granted 0
refCnt = 1
nodeNum = 1, nodeBirth 9
privateDataSize = 24
publicDataSize = 24
----------
----------
type 2, Id 8, Name = (IGYOYDBALL)
flags: 0x0
grant: count=0, type 0, wait 0
Member Count =2, master 2
. . . . .
memberNo =2, seq 1
flags = 0x0, granted 0
refCnt = 0
nodeNum = 2, nodeBirth 1
privateDataSize = 0
publicDataSize = 0
. . . . .
memberNo =1, seq 9
flags = 0x0, granted 0
refCnt = 1
nodeNum = 1, nodeBirth 9
privateDataSize = 0
publicDataSize = 0
----------
--- END OF GROCK STATE DUMP ---
------- End Dump -------
2008-10-09 10:53:32.934 >USER: Oracle Database 10g CSS Release 10.1.0.3.0 Production Copyright 1996, 2004 Oracle. All rights reserved.
2008-10-09 10:53:32.934 >USER: CSS daemon log for node yoyodb01, number 1, in cluster crs
2008-10-09 10:53:32.942 [41369600] >TRACE: clssscmain: local-only set to false
2008-10-09 10:53:32.948 [41369600] >TRACE: clssnmReadNodeInfo: added node 1 (yoyodb01) to cluster
2008-10-09 10:53:32.951 [41369600] >TRACE: clssnmReadNodeInfo: added node 2 (yoyodb02) to cluster
2008-10-09 10:53:32.953 [41369600] >TRACE: clssnmVotingDevInit: quorum disk configured to be (/u01/vote/voting.dbf)
2008-10-09 10:53:32.993 [41369600] >TRACE: clssscFatalInit: fatal mode enabled
2008-10-09 10:53:32.993 [41369600] >TRACE: clssnm_skgxnonline: Using vacuous skgxn monitor
2008-10-09 10:53:32.994 [65680000] >TRACE: clsc_listen: (0x6000000000108670) Listening on (ADDRESS=(PROTOCOL=tcp)(HOST=inter-yoyodb01)(PORT=49895))

2008-10-09 10:53:32.996 [77296256] >TRACE: clsc_listen: (0x600000000010d610) Listening on (ADDRESS=(PROTOCOL=ipc)(KEY=Oracle_CSS_LclLstnr_crs_1))

2008-10-09 10:53:32.996 [99693184] >TRACE: clsc_listen: (0x6000000000147430) Listening on (ADDRESS=(PROTOCOL=tcp)(HOST=inter-yoyodb01))

2008-10-09 10:53:34.741 [65680000] >TRACE: clssnmHandleSync: Acknowledging sync: src[2] seq[37] sync[11]
2008-10-09 10:54:09.867 [65680000] >USER: clssnmHandleUpdate: SYNC(11) from node(2) completed
2008-10-09 10:54:09.867 [65680000] >USER: clssnmHandleUpdate: NODE(1) IS ACTIVE MEMBER OF CLUSTER
2008-10-09 10:54:09.867 [65680000] >USER: clssnmHandleUpdate: NODE(2) IS ACTIVE MEMBER OF CLUSTER
2008-10-09 10:54:09.883 [65680000] >TRACE: clssnmvReadFatal: fatal mode assumed from no op
2008-10-09 10:54:09.965 [41369600] >USER: NMEVENT_SUSPEND [00][00][00][00]
2008-10-09 10:54:10.966 [153252480] >USER: NMEVENT_RECONFIG [00][00][00][06]
CLSS-3000: reconfiguration successful, incarnation 11 with 2 nodes

CLSS-3001: local node number 1, master node number 2

2008-10-09 10:54:11.187 [77296256] >TRACE: clssgmClientConnectMsg: Connect from con(0x60000000001508c0), proc(0x6000000000155290) pid()
2008-10-09 10:54:11.187 [77296256] >TRACE: clssgmClientConnectMsg: Connect from con(0x6000000000153110), proc(0x60000000001554c0) pid()
2008-10-09 10:59:27.345 [111309440] >WARNING: clssnmPollingThread: node(2) missed(4) checkin(s)
2008-10-09 10:59:32.350 [111309440] >WARNING: clssnmPollingThread: node(2) missed(4) checkin(s)
2008-10-09 10:59:33.351 [111309440] >WARNING: clssnmPollingThread: node(2) missed(5) checkin(s)
2008-10-09 10:59:34.352 [111309440] >WARNING: clssnmPollingThread: node(2) missed(6) checkin(s)
2008-10-09 11:14:58.498 [77296256] >TRACE: clssgmClientConnectMsg: Connect from con(0x6000000000158580), proc(0x600000000015a780) pid()
2008-10-09 11:18:11.449 [77296256] >TRACE: clssgmClientConnectMsg: Connect from con(0x600000000015af90), proc(0x600000000015d110) pid()
2008-10-09 11:18:11.450 [77296256] >TRACE: clsc_receive: (0x600000000015a920) Remote disconnect

2008-10-09 11:18:11.450 [77296256] >TRACE: clssgmDeleteClientListener: cleanup for proc(0x600000000015d110) con(0x600000000015af90) pid()
2008-10-09 11:18:11.462 [77296256] >TRACE: clssgmClientConnectMsg: Connect from con(0x2000000009300ff0), proc(0x2000000009302930) pid()
2008-10-09 11:18:11.463 [77296256] >TRACE: clsc_receive: (0x2000000009300920) Remote disconnect

2008-10-09 11:18:39.018 [77296256] >TRACE: clssgmDeleteClientListener: cleanup for proc(0x200000000930b950) con(0x2000000009309720) pid()
2008-10-09 11:18:43.862 [77296256] >TRACE: clsc_receive: (0x6000000000157c80) Connection failed, transport error (507, 0, 0)

2008-10-09 11:18:43.862 [77296256] >TRACE: clscreceive: (0x6000000000158580) Physical connection (0x6000000000157c80) not active, rc 11

2008-10-09 11:18:43.862 [77296256] >TRACE: clssgmDeleteClientListener: cleanup for proc(0x600000000015a780) con(0x6000000000158580) pid()
2008-10-09 11:18:48.716 [77296256] >TRACE: clssgmClientConnectMsg: Connect from con(0x200000000931a890), proc(0x200000000930a2e0) pid()
2008-10-09 11:18:55.187 [77296256] >TRACE: clssgmClientConnectMsg: Connect from con(0x200000000931bd10), proc(0x200000000931b890) pid()
2008-10-09 11:18:56.083 [77296256] >TRACE: clsc_receive: (0x200000000930b130) Connection failed, transport error (507, 0, 0)

2008-10-09 11:18:56.083 [77296256] >TRACE: clscreceive: (0x200000000931bd10) Physical connection (0x200000000930b130) not active, rc 11

2008-10-09 11:18:56.084 [77296256] >TRACE: clssgmDeleteClientListener: cleanup for proc(0x200000000931b890) con(0x200000000931bd10) pid()
2008-10-09 11:18:57.084 [77296256] >TRACE: clsc_receive: (0x2000000009309210) Connection failed, transport error (507, 0, 0)