Database Software

Announcement

For appeals, questions and feedback about Oracle Forums, please email oracle-forums-moderators_us@oracle.com. Technical questions should be asked in the appropriate category. Thank you!

Instance rebooted - ORA-29702

MarceloDalaMay 28 2008 — edited May 30 2008

Hi, we have RAC env with 4 nodes and we got instance rebooted on node #2 (part of logs below). Could you guys help us ?

ALERT.LOG
Wed May 28 09:04:50 2008
Error: unexpected error (6) from the Cluster Service (LCK0)
Wed May 28 09:04:50 2008
Errors in file /oracle/RP1/saptrace/background/rp1_002_lck0_19564.trc:
ORA-29702: error occurred in Cluster Group Service operation
LCK0: terminating instance due to error 29702
Wed May 28 09:04:50 2008
Errors in file /oracle/RP1/saptrace/background/rp1_002_lmon_19263.trc:
ORA-29702: error occurred in Cluster Group Service operation
Wed May 28 09:04:50 2008
Errors in file /oracle/RP1/saptrace/background/rp1_002_lms1_19297.trc:
ORA-29702: error occurred in Cluster Group Service operation
Wed May 28 09:04:50 2008
Errors in file /oracle/RP1/saptrace/background/rp1_002_lms0_19293.trc:
ORA-29702: error occurred in Cluster Group Service operation
Wed May 28 09:04:50 2008
Errors in file /oracle/RP1/saptrace/background/rp1_002_lmd0_19265.trc:
ORA-29702: error occurred in Cluster Group Service operation
Wed May 28 09:04:50 2008
System state dump is made for local instance
System State dumped to trace file /oracle/RP1/saptrace/background/rp1_002_diag_19259.trc
Wed May 28 09:04:54 2008
Instance terminated by LCK0, pid = 19564

CSSD.LOG
[ CSSD]2008-05-28 09:04:56.631 >USER: Oracle Database 10g CSS Release 10.2.0.2.0 Production Copyright 1996, 2004 Oracle. All rights reserved.
[ CSSD]2008-05-28 09:04:56.631 >USER: CSS daemon log for node ashb016d02pr, number 2, in cluster ashb016dpr
[ CSSD]2008-05-28 09:04:56.646 [2538232576] >TRACE: clssscmain: local-only set to false
[ clsdmt]Listening to (ADDRESS=(PROTOCOL=ipc)(KEY=ashb016d02prDBG_CSSD))
[ CSSD]2008-05-28 09:04:56.654 [2538232576] >TRACE: clssnmReadNodeInfo: added node 1 (ashb016d01pr) to cluster
[ CSSD]2008-05-28 09:04:56.666 [2538232576] >TRACE: clssnmReadNodeInfo: added node 2 (ashb016d02pr) to cluster
[ CSSD]2008-05-28 09:04:56.697 [2538232576] >TRACE: clssnmReadNodeInfo: added node 3 (ashb016d03pr) to cluster
[ CSSD]2008-05-28 09:04:56.703 [2538232576] >TRACE: clssnmReadNodeInfo: added node 4 (ashb016d04pr) to cluster
[ CSSD]2008-05-28 09:04:56.707 [1115699552] >TRACE: clssnm_skgxnmon: skgxn init failed
[ CSSD]2008-05-28 09:04:56.707 [2538232576] >TRACE: clssnm_skgxnonline: Using vacuous skgxn monitor
[ CSSD]2008-05-28 09:04:56.721 [2538232576] >TRACE: clssnmNMInitialize: misscount set to (200), impending reconfig threshold set to (196000)
[ CSSD]2008-05-28 09:04:56.722 [2538232576] >TRACE: clssnmNMInitialize: diskShortTimeout set to (197000)ms
[ CSSD]2008-05-28 09:04:56.733 [2538232576] >TRACE: clssnmNMInitialize: diskLongTimeout set to (200000)ms
[ CSSD]2008-05-28 09:04:56.736 [2538232576] >TRACE: clssnmDiskStateChange: state from 1 to 2 disk (0//oracle/102_64/oraocr/vote.crs)
[ CSSD]2008-05-28 09:04:56.736 [1115699552] >TRACE: clssnmvDPT: spawned for disk 0 (/oracle/102_64/oraocr/vote.crs)
[ CSSD]2008-05-28 09:04:56.744 [2538232576] >TRACE: clssnmDiskStateChange: state from 1 to 2 disk (1//oracle/102_64/oraocr2/vote2.crs)
[ CSSD]2008-05-28 09:04:56.745 [1126189408] >TRACE: clssnmvDPT: spawned for disk 1 (/oracle/102_64/oraocr2/vote2.crs)
[ CSSD]2008-05-28 09:04:56.747 [2538232576] >TRACE: clssnmDiskStateChange: state from 1 to 2 disk (2//oracle/102_64/crs/oraocr3/vote3.crs)
[ CSSD]2008-05-28 09:04:56.747 [1136679264] >TRACE: clssnmvDPT: spawned for disk 2 (/oracle/102_64/crs/oraocr3/vote3.crs)
[ CSSD]2008-05-28 09:04:58.744 [1115699552] >TRACE: clssnmDiskStateChange: state from 2 to 4 disk (0//oracle/102_64/oraocr/vote.crs)
[ CSSD]2008-05-28 09:04:58.818 [1126189408] >TRACE: clssnmDiskStateChange: state from 2 to 4 disk (1//oracle/102_64/oraocr2/vote2.crs)
[ CSSD]2008-05-28 09:04:58.819 [1147169120] >TRACE: clssnmvKillBlockThread: spawned for disk 0 (/oracle/102_64/oraocr/vote.crs) initial sleep interval (1000)ms
[ CSSD]2008-05-28 09:04:58.819 [1136679264] >TRACE: clssnmDiskStateChange: state from 2 to 4 disk (2//oracle/102_64/crs/oraocr3/vote3.crs)
[ CSSD]2008-05-28 09:04:58.819 [1157658976] >TRACE: clssnmvKillBlockThread: spawned for disk 1 (/oracle/102_64/oraocr2/vote2.crs) initial sleep interval (1000)ms
[ CSSD]2008-05-28 09:04:58.819 [1168148832] >TRACE: clssnmvKillBlockThread: spawned for disk 2 (/oracle/102_64/crs/oraocr3/vote3.crs) initial sleep interval (1000)ms
[ CSSD]2008-05-28 09:04:58.819 [1115699552] >TRACE: clssnmReadDskHeartbeat: node(1) is down. rcfg(10) wrtcnt(566139) LATS(1471681074) Disk lastSeqNo(566139)
[ CSSD]2008-05-28 09:04:58.819 [1115699552] >TRACE: clssnmReadDskHeartbeat: node(3) is down. rcfg(10) wrtcnt(1464143) LATS(1471681074) Disk lastSeqNo(1464143)
[ CSSD]2008-05-28 09:04:58.819 [1115699552] >TRACE: clssnmReadDskHeartbeat: node(4) is down. rcfg(10) wrtcnt(1463439) LATS(1471681074) Disk lastSeqNo(1463439)
[ CSSD]2008-05-28 09:04:58.826 [1126189408] >TRACE: clssnmReadDskHeartbeat: node(1) is down. rcfg(10) wrtcnt(566139) LATS(1471681084) Disk lastSeqNo(566139)
[ CSSD]2008-05-28 09:04:58.826 [1126189408] >TRACE: clssnmReadDskHeartbeat: node(3) is down. rcfg(10) wrtcnt(1464143) LATS(1471681084) Disk lastSeqNo(1464143)
[ CSSD]2008-05-28 09:04:58.826 [1126189408] >TRACE: clssnmReadDskHeartbeat: node(4) is down. rcfg(10) wrtcnt(1463439) LATS(1471681084) Disk lastSeqNo(1463439)
[ CSSD]2008-05-28 09:04:58.828 [1136679264] >TRACE: clssnmReadDskHeartbeat: node(1) is down. rcfg(10) wrtcnt(566139) LATS(1471681084) Disk lastSeqNo(566139)
[ CSSD]2008-05-28 09:04:58.828 [1136679264] >TRACE: clssnmReadDskHeartbeat: node(3) is down. rcfg(10) wrtcnt(1464143) LATS(1471681084) Disk lastSeqNo(1464143)
[ CSSD]2008-05-28 09:04:58.828 [1136679264] >TRACE: clssnmReadDskHeartbeat: node(4) is down. rcfg(10) wrtcnt(1463439) LATS(1471681084) Disk lastSeqNo(1463439)
[ CSSD]2008-05-28 09:04:58.831 [2538232576] >TRACE: clssnmFatalInit: fatal mode enabled
[ CSSD]2008-05-28 09:04:58.831 [1189128544] >TRACE: clssnmconnect: connecting to node 2, flags 0x0001, connector 1
[ CSSD]2008-05-28 09:04:58.832 [1189128544] >TRACE: clssnmClusterListener: Listening on (ADDRESS=(PROTOCOL=tcp)(HOST=ashb016d02pr-priv)(PORT=49895))

[ CSSD]2008-05-28 09:04:58.832 [1189128544] >TRACE: clssnmconnect: connecting to node 0, flags 0x0000, connector 1
[ CSSD]2008-05-28 09:04:58.832 [1189128544] >TRACE: clssnmconnect: connecting to node 1, flags 0x0001, connector 0
[ CSSD]2008-05-28 09:04:58.833 [1189128544] >TRACE: clssnmClusterListener: Probing node 3, con (0x72a750)
[ CSSD]2008-05-28 09:04:58.833 [1189128544] >TRACE: clssnmClusterListener: Probing node 4, con (0x72d340)
[ CSSD]2008-05-28 09:04:58.833 [1189128544] >TRACE: clssnmConnComplete: connected to node 1 (con 0x727b60), state 3 birth 0, unique 1211396400/1211396400 prevConuni(0)
[ CSSD]2008-05-28 09:04:58.834 [1189128544] >TRACE: clssnmConnComplete: connected to node 3 (con 0x2a97b01070), state 3 birth 0, unique 1210493312/1210493312 prevConuni(0)
[ CSSD]2008-05-28 09:04:58.835 [1189128544] >TRACE: clssnmConnComplete: connected to node 4 (con 0x2a97b032a0), state 3 birth 0, unique 1210494013/1210494013 prevConuni(0)
[ CSSD]2008-05-28 09:04:58.836 [1199618400] >TRACE: clssgmclientlsnr: listening on (ADDRESS=(PROTOCOL=ipc)(KEY=Oracle_CSS_LclLstnr_ashb016dpr_2))
[ CSSD]2008-05-28 09:04:58.836 [1199618400] >TRACE: clssgmclientlsnr: listening on (ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_ashb016d02pr_ashb016dpr))
[ CSSD]2008-05-28 09:04:58.837 [1231087968] >TRACE: clssgmPeerListener: Listening on (ADDRESS=(PROTOCOL=tcp)(DEV=26)(HOST=10.252.14.130)(PORT=3779))
[ CSSD]2008-05-28 09:04:58.837 [1241577824] >TRACE: clssnmPollingThread: Connection complete
[ CSSD]2008-05-28 09:04:58.837 [1252067680] >TRACE: clssnmSendingThread: Connection complete
[ CSSD]2008-05-28 09:04:58.837 [1262557536] >TRACE: clssnmRcfgMgrThread: Connection complete
[ CSSD]2008-05-28 09:04:59.109 [1189128544] >TRACE: clssnmHandleSync: Acknowledging sync: src[3] srcName[ashb016d03pr] seq[5] sync[10]
[ CSSD]2008-05-28 09:04:59.109 [1189128544] >TRACE: clssnmHandleSync: diskTimeout set to (197000)ms
[ CSSD]2008-05-28 09:04:59.110 [1189128544] >TRACE: clssnmSendVoteInfo: node(3) syncSeqNo(10)
[ CSSD]2008-05-28 09:04:59.150 [1189128544] >TRACE: clssnmUpdateNodeState: node 0, state (0/0) unique (0/0) prevConuni(0) birth (0/0) (old/new)
[ CSSD]2008-05-28 09:04:59.150 [1189128544] >TRACE: clssnmDeactivateNode: node 0 () left cluster

[ CSSD]2008-05-28 09:04:59.150 [1189128544] >TRACE: clssnmUpdateNodeState: node 1, state (4/3) unique (1211396400/1211396400) prevConuni(0) birth (0/8) (old/new)
[ CSSD]2008-05-28 09:04:59.150 [1189128544] >TRACE: clssnmUpdateNodeState: node 2, state (1/2) unique (1211965496/1211965496) prevConuni(0) birth (0/10) (old/new)
[ CSSD]2008-05-28 09:04:59.150 [1189128544] >TRACE: clssnmUpdateNodeState: node 3, state (4/3) unique (1210493312/1210493312) prevConuni(0) birth (0/2) (old/new)
[ CSSD]2008-05-28 09:04:59.150 [1189128544] >TRACE: clssnmUpdateNodeState: node 4, state (4/3) unique (1210494013/1210494013) prevConuni(0) birth (0/4) (old/new)
[ CSSD]2008-05-28 09:04:59.150 [1189128544] >USER: clssnmHandleUpdate: SYNC(10) from node(3) completed
[ CSSD]2008-05-28 09:04:59.150 [1189128544] >USER: clssnmHandleUpdate: NODE 1 (ashb016d01pr) IS ACTIVE MEMBER OF CLUSTER
[ CSSD]2008-05-28 09:04:59.150 [1189128544] >USER: clssnmHandleUpdate: NODE 2 (ashb016d02pr) IS ACTIVE MEMBER OF CLUSTER
[ CSSD]2008-05-28 09:04:59.150 [1189128544] >USER: clssnmHandleUpdate: NODE 3 (ashb016d03pr) IS ACTIVE MEMBER OF CLUSTER
[ CSSD]2008-05-28 09:04:59.150 [1189128544] >USER: clssnmHandleUpdate: NODE 4 (ashb016d04pr) IS ACTIVE MEMBER OF CLUSTER
[ CSSD]2008-05-28 09:04:59.150 [1189128544] >TRACE: clssnmHandleUpdate: diskTimeout set to (200000)ms
[ CSSD]2008-05-28 09:04:59.245 [2538232576] >USER: NMEVENT_SUSPEND [00][00][00][00]
[ CSSD]2008-05-28 09:04:59.245 [1273047392] >TRACE: clssgmReconfigThread: started for reconfig (10)
[ CSSD]2008-05-28 09:04:59.245 [1273047392] >USER: NMEVENT_RECONFIG [00][00][00][1e]
[ CSSD]2008-05-28 09:04:59.245 [1273047392] >TRACE: clssgmEstablishConnections: 4 nodes in cluster incarn 10
[ CSSD]2008-05-28 09:04:59.246 [1231087968] >TRACE: clssgmInitialRecv: (0x77de40) accepted a new connection from node 1 born at 8 active (4, 2), vers (10,3,1,2)
[ CSSD]2008-05-28 09:04:59.246 [1231087968] >TRACE: clssgmInitialRecv: (0x780920) accepted a new connection from node 3 born at 2 active (4, 3), vers (10,3,1,2)
[ CSSD]2008-05-28 09:04:59.246 [1231087968] >TRACE: clssgmInitialRecv: (0x783510) accepted a new connection from node 4 born at 4 active (4, 4), vers (10,3,1,2)
[ CSSD]2008-05-28 09:04:59.246 [1231087968] >TRACE: clssgmInitialRecv: conns done (4/4)
[ CSSD]2008-05-28 09:04:59.246 [1273047392] >TRACE: clssgmEstablishMasterNode: MASTER for 10 is node(3) birth(2)
[ CSSD]2008-05-28 09:04:59.246 [1273047392] >TRACE: clssgmChangeMasterNode: requeued 0 RPCs
[ CSSD]2008-05-28 09:04:59.255 [1231087968] >TRACE: clssgmHandleDBDone(): src/dest (3/65535) size(72) incarn 10
[ CSSD]CLSS-3000: reconfiguration successful, incarnation 10 with 4 nodes

[ CSSD]CLSS-3001: local node number 2, master node number 3

CRSD.LOG
*** 2008-05-28 09:04:50.253
2008-05-28 09:04:50.253: [ CSSCLNT]clsssRecvMsg: comm error received, comrc 11, con (0x5fbc3c0), msg (0x7fbfffdf00), msgl 144
2008-05-28 09:04:50.269: [ CSSCLNT]clssgsGGetStatus: communications failed (0/3/0)
2008-05-28 09:04:50.269: [ CSSCLNT]clssgsGGetStatus: returning 8
kgxgnpstat: received ABORT event from CLSS
CM problem, please abort

EVMD.LOG
2008-05-28 09:04:50.191: [ CSSCLNT][1491249504]clssgsGGetStatus: communications failed (0/3/53)
2008-05-28 09:04:50.191: [ CSSCLNT][1491249504]clssgsGGetStatus: returning 8
2008-05-28 09:04:50.191: [ CRSEVT][1491249504][PANIC]0Error in clssgsgrpstat rc =8
2008-05-28 09:04:50.214: [ CSSCLNT][1210108256]clsssRecvMsg: comm error received, comrc 11, con (0xc3dd20), msg (0x48209fc0), msgl 144
2008-05-28 09:04:50.214: [ CSSCLNT][1210108256]clssgsGGetStatus: communications failed (0/3/0)
2008-05-28 09:04:50.214: [ CSSCLNT][1210108256]clssgsGGetStatus: returning 8
2008-05-28 09:04:50.214: [ OCRMAS][1210108256]th_master:21:clssgsgrpstat failed css ret code = 8
2008-05-28 09:04:50.214: [ COMMCRS][1210108256]clscsendx: (0xc3dd20) Connection not active
2008-05-28 09:04:50.214: [ CSSCLNT][1210108256]clsssServerRPC: send failed with err 6, msg type 7
2008-05-28 09:04:50.214: [ CSSCLNT][1210108256]clsssCommonClientExit: RPC failure, rc 3
2008-05-28 09:04:50.283: [ CSSCLNT][1294027104]clsssRecvMsg: comm error received, comrc 11, con (0xc58400), msg (0x4d212db0), msgl 144
2008-05-28 09:04:50.283: [ CSSCLNT][1294027104]clssgsGGetStatus: communications failed (0/3/0)
2008-05-28 09:04:50.283: [ CSSCLNT][1294027104]clssgsGGetStatus: returning 8

Locked Post

New comments cannot be posted to this locked post.

Locked on Jun 27 2008

Added on May 28 2008

#performance-availability, #real-application-clusters

1 comment

2,346 views