Skip to Main Content
Forums

Database Software

Announcement

For appeals, questions and feedback about Oracle Forums, please email oracle-forums-moderators_us@oracle.com. Technical questions should be asked in the appropriate category. Thank you!

clusterware nodes evicted too quickly

User754933-OracleOct 26 2009 — edited Oct 27 2009
I have two clusterware nodes -- pdb3 and pdb4.
They each run 10g databases. Clusterware and 10g are at 10.2.0.4

If I pull the crossover cable connecting them, the node from which I pull
(this morning's test, it was pdb4) the cable will very quickly reboot itself.
Not long after, the second node pdb3) reboots ITself. Very annoying.

How can I make sure that if, in the case of the recent test, pdb3 takes
over serving the databases and pdb4 is given more time to correct itself.
I only removed the crossover cable for ONE SECOND, then put it right
back in, and then the reboots were upon us. I don't want this happening
anymore because obviously there's no availability if they just join this
suicide pact and take themselves offline. I thought for a while it was ocfs2
doing this but I have set the heartbeat and network idle timeouts very high for
that so I doubt that's it. The 2 nodes share several SAN-based ocfs2 filesystems.

Logs are here (from PDB4; can upload PDB3's as well if needed):

PDB4's ocssd.log:
[ CSSD]2009-10-26 08:08:40.253 [1205410112] >TRACE: clsc_receive: (0xd17ef50) Error receiving, ns (12535, 12560), transport (505, 110, 0)

[ CSSD]2009-10-26 08:08:40.253 [1205410112] >WARNING: clssnmeventhndlr: Receive failure with node 1 (e1198pdb3), state 3, con(0xd17f650), probe((nil)), rc=3
[ CSSD]2009-10-26 08:08:40.253 [1205410112] >TRACE: clssnmDiscHelper: e1198pdb3, node(1) connection failed, con (0xd17f650), probe((nil))
[ CSSD]2009-10-26 08:09:07.236 [1257859392] >WARNING: clssnmPollingThread: node e1198pdb3 (1) at 50 1.077030e-315artbeat fatal, eviction in 29.450 seconds
[ CSSD]2009-10-26 08:09:07.236 [1257859392] >TRACE: clssnmPollingThread: node e1198pdb3 (1) is impending reconfig, flag 1, misstime 30550
[ CSSD]2009-10-26 08:09:07.237 [1257859392] >TRACE: clssnmPollingThread: diskTimeout set to (57000)ms impending reconfig status(1)
[ CSSD]2009-10-26 08:09:22.236 [1257859392] >WARNING: clssnmPollingThread: node e1198pdb3 (1) at 75 1.077030e-315artbeat fatal, eviction in 14.450 seconds
[ CSSD]2009-10-26 08:09:31.235 [1257859392] >WARNING: clssnmPollingThread: node e1198pdb3 (1) at 90 1.482197e-323artbeat fatal, eviction in 5.450 seconds
[ CSSD]2009-10-26 08:09:32.237 [1257859392] >WARNING: clssnmPollingThread: node e1198pdb3 (1) at 90 1.482197e-323artbeat fatal, eviction in 4.450 seconds
[ CSSD]2009-10-26 08:09:33.240 [1257859392] >WARNING: clssnmPollingThread: node e1198pdb3 (1) at 90 1.482197e-323artbeat fatal, eviction in 3.450 seconds
[ CSSD]2009-10-26 08:09:34.242 [1257859392] >WARNING: clssnmPollingThread: node e1198pdb3 (1) at 90 1.482197e-323artbeat fatal, eviction in 2.450 seconds
[ CSSD]2009-10-26 08:09:35.244 [1257859392] >WARNING: clssnmPollingThread: node e1198pdb3 (1) at 90 1.482197e-323artbeat fatal, eviction in 1.440 seconds

PDB4's /var/log/kern.log:
Oct 26 08:08:37 localhost kernel: bnx2: eth1 NIC Copper Link is Down
Oct 26 08:08:40 localhost kernel: bnx2: eth1 NIC Copper Link is Up, 1000 Mbps full duplex, receive & transmit flow control ON
Oct 26 08:13:30 localhost kernel: klogd 1.4.1, log source = /proc/kmsg started.
Oct 26 08:13:30 localhost kernel: ervention required
Oct 26 08:13:30 localhost kernel:
Oct 26 08:13:30 localhost kernel: end_request: I/O error, dev sdy, sector 280278624

... bunch more of that then ultimately in the same log file the box reboots and we get:

Oct 26 08:13:31 localhost kernel: bnx2: eth0: using MSI
Oct 26 08:13:31 localhost kernel: bnx2: eth0 NIC Copper Link is Up, 1000 Mbps full duplex, receive & transmit flow control ON
Oct 26 08:13:31 localhost kernel: bnx2: eth1: using MSI
Oct 26 08:13:31 localhost kernel: bnx2: eth1 NIC Copper Link is Up, 1000 Mbps full duplex, receive & transmit flow control ON
Oct 26 08:13:31 localhost kernel: OCFS2 Node Manager 1.4.1 Wed Jul 23 12:05:37 PDT 2008 (build 0f78045c75c0174e50e4cf0934bf9eae)
Oct 26 08:13:31 localhost kernel: OCFS2 DLM 1.4.1 Wed Jul 23 12:05:37 PDT 2008 (build 4ce8fae327880c466761f40fb7619490)
Oct 26 08:13:31 localhost kernel: OCFS2 DLMFS 1.4.1 Wed Jul 23 12:05:37 PDT 2008 (build 4ce8fae327880c466761f40fb7619490)
Oct 26 08:13:31 localhost kernel: OCFS2 User DLM kernel interface loaded
Oct 26 08:13:38 localhost kernel: o2net: connected to node e1198pdb3 (num 0) at 10.156.90.13:7777
Oct 26 08:13:42 localhost kernel: OCFS2 1.4.1 Wed Jul 23 12:05:34 PDT 2008 (build 3fc82af4b5669945497b322b6aabd031)
Oct 26 08:14:32 localhost kernel: bnx2: eth1 NIC Copper Link is Down
Oct 26 08:14:34 localhost kernel: bnx2: eth1 NIC Copper Link is Up, 1000 Mbps full duplex, receive & transmit flow control ON
Oct 26 08:14:35 localhost kernel: o2net: no longer connected to node e1198pdb3 (num 0) at 10.156.90.13:7777
Oct 26 08:14:35 localhost kernel: (19230,2):dlm_request_join:1033 ERROR: status = -112
Oct 26 08:14:35 localhost kernel: (19230,2):dlm_try_to_join_domain:1207 ERROR: status = -112
Oct 26 08:14:35 localhost kernel: (19230,2):dlm_join_domain:1485 ERROR: status = -112
Oct 26 08:14:35 localhost kernel: (19230,2):dlm_register_domain:1732 ERROR: status = -112
Oct 26 08:14:35 localhost kernel: (19230,2):ocfs2_dlm_init:2662 ERROR: status = -112
Oct 26 08:14:35 localhost kernel: (19230,2):ocfs2_mount_volume:1251 ERROR: status = -112
Oct 26 08:14:35 localhost kernel: ocfs2: Unmounting device (120,33) on (node 1)
Oct 26 08:14:42 localhost kernel: ocfs2_dlm: Nodes in domain ("FFC760CD2E4E48AF89C03CF9497A055C"): 1
Oct 26 08:14:42 localhost kernel: (19257,4):ocfs2_find_slot:249 slot 1 is already allocated to this node!
Oct 26 08:14:42 localhost kernel: (19257,4):ocfs2_check_volume:1745 File system was not unmounted cleanly, recovering volume.
Oct 26 08:14:42 localhost kernel: kjournald starting. Commit interval 5 seconds
Oct 26 08:14:42 localhost kernel: ocfs2: Mounting device (120,65) on (node 1, slot 1) with ordered data mode.
Oct 26 08:14:42 localhost kernel: (19265,0):ocfs2_replay_journal:1076 Recovering node 0 from slot 0 on device (120,65)
Oct 26 08:14:43 localhost kernel: kjournald starting. Commit interval 5 seconds
Oct 26 08:14:46 localhost kernel: ocfs2_dlm: Nodes in domain ("7FB9B3539B134F56958A9E44CCB25EB0"): 1
Oct 26 08:14:46 localhost kernel: (19269,4):ocfs2_find_slot:249 slot 1 is already allocated to this node!
Oct 26 08:14:46 localhost kernel: (19269,4):ocfs2_check_volume:1745 File system was not unmounted cleanly, recovering volume.
Oct 26 08:14:46 localhost kernel: kjournald starting. Commit interval 5 seconds
Oct 26 08:14:46 localhost kernel: ocfs2: Mounting device (120,49) on (node 1, slot 1) with ordered data mode.
Oct 26 08:14:46 localhost kernel: (19278,0):ocfs2_replay_journal:1076 Recovering node 0 from slot 0 on device (120,49)
Oct 26 08:14:48 localhost kernel: kjournald starting. Commit interval 5 seconds
Oct 26 08:14:50 localhost kernel: ocfs2_dlm: Nodes in domain ("DC8F82EEB7F149CAB24B4802AA4105BD"): 1
Oct 26 08:14:50 localhost kernel: (19282,4):ocfs2_find_slot:249 slot 1 is already allocated to this node!
Oct 26 08:14:50 localhost kernel: (19282,4):ocfs2_check_volume:1745 File system was not unmounted cleanly, recovering volume.
Oct 26 08:14:50 localhost kernel: kjournald starting. Commit interval 5 seconds
Oct 26 08:14:50 localhost kernel: ocfs2: Mounting device (120,129) on (node 1, slot 1) with ordered data mode.
Oct 26 08:14:50 localhost kernel: (19291,0):ocfs2_replay_journal:1076 Recovering node 0 from slot 0 on device (120,129)
Oct 26 08:14:51 localhost kernel: kjournald starting. Commit interval 5 seconds
Oct 26 08:14:54 localhost kernel: ocfs2_dlm: Nodes in domain ("EB71A901424745E6BEB746F3F4408035"): 1
Oct 26 08:14:54 localhost kernel: (19295,4):ocfs2_find_slot:249 slot 1 is already allocated to this node!
Oct 26 08:14:54 localhost kernel: (19295,4):ocfs2_check_volume:1745 File system was not unmounted cleanly, recovering volume.
Oct 26 08:14:54 localhost kernel: kjournald starting. Commit interval 5 seconds
Oct 26 08:14:54 localhost kernel: ocfs2: Mounting device (120,17) on (node 1, slot 1) with ordered data mode.
Oct 26 08:14:54 localhost kernel: (19304,0):ocfs2_replay_journal:1076 Recovering node 0 from slot 0 on device (120,17)
Oct 26 08:14:55 localhost kernel: kjournald starting. Commit interval 5 seconds
Oct 26 08:14:58 localhost kernel: ocfs2_dlm: Nodes in domain ("C99F7F9A514643D6814D54420A7ACA5E"): 1
Oct 26 08:14:58 localhost kernel: (19308,4):ocfs2_find_slot:249 slot 1 is already allocated to this node!
Oct 26 08:14:58 localhost kernel: (19308,4):ocfs2_check_volume:1745 File system was not unmounted cleanly, recovering volume.
Oct 26 08:14:58 localhost kernel: kjournald starting. Commit interval 5 seconds
Oct 26 08:14:58 localhost kernel: ocfs2: Mounting device (120,81) on (node 1, slot 1) with ordered data mode.
Oct 26 08:14:58 localhost kernel: (19317,0):ocfs2_replay_journal:1076 Recovering node 0 from slot 0 on device (120,81)
Oct 26 08:14:59 localhost kernel: kjournald starting. Commit interval 5 seconds
Oct 26 08:15:02 localhost kernel: ocfs2_dlm: Nodes in domain ("224D7E5B2D3143179334711BB8FAA8E4"): 1
Oct 26 08:15:02 localhost kernel: (19321,4):ocfs2_find_slot:249 slot 1 is already allocated to this node!
Oct 26 08:15:02 localhost kernel: (19321,4):ocfs2_check_volume:1745 File system was not unmounted cleanly, recovering volume.
Oct 26 08:15:02 localhost kernel: kjournald starting. Commit interval 5 seconds
Oct 26 08:15:02 localhost kernel: ocfs2: Mounting device (120,1) on (node 1, slot 1) with ordered data mode.
Oct 26 08:15:02 localhost kernel: (19330,0):ocfs2_replay_journal:1076 Recovering node 0 from slot 0 on device (120,1)
Oct 26 08:15:04 localhost kernel: kjournald starting. Commit interval 5 seconds
Oct 26 08:15:05 localhost kernel: NET: Registered protocol family 10
Oct 26 08:15:05 localhost kernel: lo: Disabled Privacy Extensions
Oct 26 08:15:05 localhost kernel: IPv6 over IPv4 tunneling driver
Oct 26 08:15:15 localhost kernel: eth1: no IPv6 routers present
Oct 26 08:15:16 localhost kernel: eth0: no IPv6 routers present
Oct 26 08:18:09 localhost kernel: bnx2: eth1 NIC Copper Link is Down
Oct 26 08:18:11 localhost kernel: bnx2: eth1 NIC Copper Link is Up, 1000 Mbps full duplex, receive & transmit flow control ON
Oct 26 08:18:15 localhost kernel: usb 1-7.1.2: new high speed USB device using ehci_hcd and address 9
Oct 26 08:18:15 localhost kernel: usb 1-7.1.2: configuration #1 chosen from 1 choice
Oct 26 08:18:15 localhost kernel: scsi7 : SCSI emulation for USB Mass Storage devices
Oct 26 08:18:15 localhost kernel: usb-storage: device found at 9
Oct 26 08:18:15 localhost kernel: usb-storage: waiting for device to settle before scanning
Oct 26 08:18:20 localhost kernel: Vendor: USB 2.0 Model: USB Flash Drive Rev: 0.00
Oct 26 08:18:20 localhost kernel: Type: Direct-Access ANSI SCSI revision: 02
Oct 26 08:18:20 localhost kernel: SCSI device sdaf: 15794176 512-byte hdwr sectors (8087 MB)
Oct 26 08:18:20 localhost kernel: sdaf: Write Protect is off
Oct 26 08:18:20 localhost kernel: sdaf: Mode Sense: 00 00 00 00
Oct 26 08:18:20 localhost kernel: sdaf: assuming drive cache: write through
Oct 26 08:18:20 localhost kernel: SCSI device sdaf: 15794176 512-byte hdwr sectors (8087 MB)
Oct 26 08:18:20 localhost kernel: sdaf: Write Protect is off
Oct 26 08:18:20 localhost kernel: sdaf: Mode Sense: 00 00 00 00
Oct 26 08:18:20 localhost kernel: sdaf: assuming drive cache: write through
Oct 26 08:18:20 localhost kernel: sdaf: sdaf1

/var/log/messages on PDB4:

Oct 26 08:07:11 localhost kernel: usb 1-7.1.1: new low speed USB device using ehci_hcd and address 9
Oct 26 08:07:11 localhost kernel: usb 1-7.1.1: configuration #1 chosen from 1 choice
Oct 26 08:07:11 localhost kernel: input: HID 04f3:0103 as /class/input/input7
Oct 26 08:07:11 localhost kernel: input: USB HID v1.10 Keyboard [HID 04f3:0103] on usb-0000:00:1d.7-7.1.1
Oct 26 08:07:11 localhost kernel: input: HID 04f3:0103 as /class/input/input8
Oct 26 08:07:11 localhost kernel: input: USB HID v1.10 Device [HID 04f3:0103] on usb-0000:00:1d.7-7.1.1
Oct 26 08:08:37 localhost kernel: bnx2: eth1 NIC Copper Link is Down
Oct 26 08:08:40 localhost kernel: bnx2: eth1 NIC Copper Link is Up, 1000 Mbps full duplex, receive & transmit flow control ON
Oct 26 08:09:36 localhost logger: Oracle CSSD failure 134.
Oct 26 08:13:30 localhost syslogd 1.4.1: restart.
Oct 26 08:13:30 localhost kernel: klogd 1.4.1, log source = /proc/kmsg started.
Oct 26 08:13:30 localhost kernel: ervention required
Oct 26 08:13:30 localhost kernel:


... bunch lot more of this ... and then ...

Oct 26 08:13:31 localhost kernel: emcpoweri: emcpoweri1
Oct 26 08:13:31 localhost kernel: kjournald starting. Commit interval 5 seconds
Oct 26 08:13:31 localhost kernel: EXT3 FS on sda8, internal journal
Oct 26 08:13:31 localhost kernel: EXT3-fs: mounted filesystem with ordered data mode.
Oct 26 08:13:31 localhost kernel: kjournald starting. Commit interval 5 seconds
Oct 26 08:13:31 localhost kernel: EXT3 FS on sda2, internal journal
Oct 26 08:13:31 localhost kernel: EXT3-fs: mounted filesystem with ordered data mode.
Oct 26 08:13:31 localhost kernel: kjournald starting. Commit interval 5 seconds
Oct 26 08:13:31 localhost kernel: EXT3 FS on sda6, internal journal
Oct 26 08:13:31 localhost kernel: EXT3-fs: mounted filesystem with ordered data mode.
Oct 26 08:13:31 localhost kernel: kjournald starting. Commit interval 5 seconds
Oct 26 08:13:31 localhost kernel: EXT3 FS on sda7, internal journal
Oct 26 08:13:31 localhost kernel: EXT3-fs: mounted filesystem with ordered data mode.
Oct 26 08:13:31 localhost kernel: kjournald starting. Commit interval 5 seconds
Oct 26 08:13:31 localhost kernel: EXT3 FS on sdb1, internal journal
Oct 26 08:13:31 localhost kernel: EXT3-fs: mounted filesystem with ordered data mode.
Oct 26 08:13:31 localhost kernel: kjournald starting. Commit interval 5 seconds
Oct 26 08:13:31 localhost kernel: EXT3 FS on sda1, internal journal
Oct 26 08:13:31 localhost kernel: EXT3-fs: mounted filesystem with ordered data mode.
Oct 26 08:13:31 localhost kernel: Adding 30716240k swap on /dev/sda5. Priority:-1 extents:1 across:30716240k
Oct 26 08:13:31 localhost kernel: bnx2: eth0: using MSI
Oct 26 08:13:31 localhost kernel: bnx2: eth0 NIC Copper Link is Up, 1000 Mbps full duplex, receive & transmit flow control ON
Oct 26 08:13:31 localhost kernel: bnx2: eth1: using MSI
Oct 26 08:13:31 localhost kernel: bnx2: eth1 NIC Copper Link is Up, 1000 Mbps full duplex, receive & transmit flow control ON
Oct 26 08:13:31 localhost kernel: OCFS2 Node Manager 1.4.1 Wed Jul 23 12:05:37 PDT 2008 (build 0f78045c75c0174e50e4cf0934bf9eae)
Oct 26 08:13:31 localhost kernel: OCFS2 DLM 1.4.1 Wed Jul 23 12:05:37 PDT 2008 (build 4ce8fae327880c466761f40fb7619490)
Oct 26 08:13:31 localhost kernel: OCFS2 DLMFS 1.4.1 Wed Jul 23 12:05:37 PDT 2008 (build 4ce8fae327880c466761f40fb7619490)
Oct 26 08:13:31 localhost kernel: OCFS2 User DLM kernel interface loaded
Oct 26 08:13:33 localhost Navisphere Agent[19079]: Agent initializing with pid 19079
Oct 26 08:13:34 localhost EV_AGENT[19213]: Agent daemon process created, pid 19213
Oct 26 08:13:34 localhost EV_AGENT[19213]: Agent has started up.
Oct 26 08:13:38 localhost kernel: o2net: connected to node e1198pdb3 (num 0) at 10.156.90.13:7777
Oct 26 08:13:42 localhost kernel: OCFS2 1.4.1 Wed Jul 23 12:05:34 PDT 2008 (build 3fc82af4b5669945497b322b6aabd031)
Oct 26 08:14:32 localhost kernel: bnx2: eth1 NIC Copper Link is Down
Oct 26 08:14:34 localhost kernel: bnx2: eth1 NIC Copper Link is Up, 1000 Mbps full duplex, receive & transmit flow control ON
Oct 26 08:14:35 localhost kernel: o2net: no longer connected to node e1198pdb3 (num 0) at 10.156.90.13:7777
Oct 26 08:14:35 localhost kernel: (19230,2):dlm_request_join:1033 ERROR: status = -112
Oct 26 08:14:35 localhost kernel: (19230,2):dlm_try_to_join_domain:1207 ERROR: status = -112
Oct 26 08:14:35 localhost kernel: (19230,2):dlm_join_domain:1485 ERROR: status = -112
Oct 26 08:14:35 localhost kernel: (19230,2):dlm_register_domain:1732 ERROR: status = -112
Oct 26 08:14:35 localhost kernel: (19230,2):ocfs2_dlm_init:2662 ERROR: status = -112
Oct 26 08:14:35 localhost kernel: (19230,2):ocfs2_mount_volume:1251 ERROR: status = -112
Oct 26 08:14:35 localhost kernel: ocfs2: Unmounting device (120,33) on (node 1)
Oct 26 08:14:42 localhost kernel: ocfs2_dlm: Nodes in domain ("FFC760CD2E4E48AF89C03CF9497A055C"): 1
Oct 26 08:14:42 localhost kernel: (19257,4):ocfs2_find_slot:249 slot 1 is already allocated to this node!
Oct 26 08:14:42 localhost kernel: (19257,4):ocfs2_check_volume:1745 File system was not unmounted cleanly, recovering volume.
Oct 26 08:14:42 localhost kernel: kjournald starting. Commit interval 5 seconds
Oct 26 08:14:42 localhost kernel: ocfs2: Mounting device (120,65) on (node 1, slot 1) with ordered data mode.
Oct 26 08:14:42 localhost kernel: (19265,0):ocfs2_replay_journal:1076 Recovering node 0 from slot 0 on device (120,65)
Oct 26 08:14:43 localhost kernel: kjournald starting. Commit interval 5 seconds
Oct 26 08:14:46 localhost kernel: ocfs2_dlm: Nodes in domain ("7FB9B3539B134F56958A9E44CCB25EB0"): 1
Oct 26 08:14:46 localhost kernel: (19269,4):ocfs2_find_slot:249 slot 1 is already allocated to this node!
Oct 26 08:14:46 localhost kernel: (19269,4):ocfs2_check_volume:1745 File system was not unmounted cleanly, recovering volume.
Oct 26 08:14:46 localhost kernel: kjournald starting. Commit interval 5 seconds
Oct 26 08:14:46 localhost kernel: ocfs2: Mounting device (120,49) on (node 1, slot 1) with ordered data mode.
Oct 26 08:14:46 localhost kernel: (19278,0):ocfs2_replay_journal:1076 Recovering node 0 from slot 0 on device (120,49)
Oct 26 08:14:48 localhost kernel: kjournald starting. Commit interval 5 seconds
Oct 26 08:14:50 localhost kernel: ocfs2_dlm: Nodes in domain ("DC8F82EEB7F149CAB24B4802AA4105BD"): 1
Oct 26 08:14:50 localhost kernel: (19282,4):ocfs2_find_slot:249 slot 1 is already allocated to this node!
Oct 26 08:14:50 localhost kernel: (19282,4):ocfs2_check_volume:1745 File system was not unmounted cleanly, recovering volume.
Oct 26 08:14:50 localhost kernel: kjournald starting. Commit interval 5 seconds
Oct 26 08:14:50 localhost kernel: ocfs2: Mounting device (120,129) on (node 1, slot 1) with ordered data mode.
Oct 26 08:14:50 localhost kernel: (19291,0):ocfs2_replay_journal:1076 Recovering node 0 from slot 0 on device (120,129)
Oct 26 08:14:51 localhost kernel: kjournald starting. Commit interval 5 seconds
Oct 26 08:14:54 localhost kernel: ocfs2_dlm: Nodes in domain ("EB71A901424745E6BEB746F3F4408035"): 1
Oct 26 08:14:54 localhost kernel: (19295,4):ocfs2_find_slot:249 slot 1 is already allocated to this node!
Oct 26 08:14:54 localhost kernel: (19295,4):ocfs2_check_volume:1745 File system was not unmounted cleanly, recovering volume.
Oct 26 08:14:54 localhost kernel: kjournald starting. Commit interval 5 seconds
Oct 26 08:14:54 localhost kernel: ocfs2: Mounting device (120,17) on (node 1, slot 1) with ordered data mode.
Oct 26 08:14:54 localhost kernel: (19304,0):ocfs2_replay_journal:1076 Recovering node 0 from slot 0 on device (120,17)
Oct 26 08:14:55 localhost kernel: kjournald starting. Commit interval 5 seconds
Oct 26 08:14:58 localhost kernel: ocfs2_dlm: Nodes in domain ("C99F7F9A514643D6814D54420A7ACA5E"): 1
Oct 26 08:14:58 localhost kernel: (19308,4):ocfs2_find_slot:249 slot 1 is already allocated to this node!
Oct 26 08:14:58 localhost kernel: (19308,4):ocfs2_check_volume:1745 File system was not unmounted cleanly, recovering volume.
Oct 26 08:14:58 localhost kernel: kjournald starting. Commit interval 5 seconds
Oct 26 08:14:58 localhost kernel: ocfs2: Mounting device (120,81) on (node 1, slot 1) with ordered data mode.
Oct 26 08:14:58 localhost kernel: (19317,0):ocfs2_replay_journal:1076 Recovering node 0 from slot 0 on device (120,81)
Oct 26 08:14:59 localhost kernel: kjournald starting. Commit interval 5 seconds
Oct 26 08:15:02 localhost kernel: ocfs2_dlm: Nodes in domain ("224D7E5B2D3143179334711BB8FAA8E4"): 1
Oct 26 08:15:02 localhost kernel: (19321,4):ocfs2_find_slot:249 slot 1 is already allocated to this node!
Oct 26 08:15:02 localhost kernel: (19321,4):ocfs2_check_volume:1745 File system was not unmounted cleanly, recovering volume.
Oct 26 08:15:02 localhost kernel: kjournald starting. Commit interval 5 seconds
Oct 26 08:15:02 localhost kernel: ocfs2: Mounting device (120,1) on (node 1, slot 1) with ordered data mode.
Oct 26 08:15:02 localhost kernel: (19330,0):ocfs2_replay_journal:1076 Recovering node 0 from slot 0 on device (120,1)
Oct 26 08:15:03 localhost automount[19366]: lookup_read_master: lookup(nisplus): couldn't locate nis+ table auto.master
Oct 26 08:15:04 localhost kernel: kjournald starting. Commit interval 5 seconds
Oct 26 08:15:04 localhost logger: emcp_mond.sh: Unable to start PowerPath Management Daemon. You must first configure and start net-snmp
Oct 26 08:15:05 localhost xinetd[19435]: xinetd Version 2.3.14 started with libwrap loadavg labeled-networking options compiled in.
Oct 26 08:15:05 localhost xinetd[19435]: Started working: 0 available services
Oct 26 08:15:05 localhost ntpd[19450]: ntpd 4.2.2p1@1.1570-o Mon May 18 21:19:56 UTC 2009 (1)
Oct 26 08:15:05 localhost ntpd[19451]: precision = 1.000 usec
Oct 26 08:15:05 localhost kernel: NET: Registered protocol family 10
Oct 26 08:15:05 localhost kernel: lo: Disabled Privacy Extensions
Oct 26 08:15:05 localhost kernel: IPv6 over IPv4 tunneling driver
Oct 26 08:15:05 localhost ntpd[19451]: Listening on interface wildcard, 0.0.0.0#123 Disabled
Oct 26 08:15:05 localhost ntpd[19451]: Listening on interface wildcard, ::#123 Disabled
Oct 26 08:15:05 localhost ntpd[19451]: Listening on interface lo, ::1#123 Enabled
Oct 26 08:15:05 localhost ntpd[19451]: bind() fd 19, family 10, port 123, scope 3, addr fe80::21e:c9ff:fed8:e98e, in6_is_addr_multicast=0 flags=1 fails: Cannot assign requested address
Oct 26 08:15:05 localhost ntpd[19451]: bind() fd 19, family 10, port 123, scope 2, addr fe80::21e:c9ff:fed8:e98c, in6_is_addr_multicast=0 flags=1 fails: Cannot assign requested address
Oct 26 08:15:05 localhost ntpd[19451]: Listening on interface lo, 127.0.0.1#123 Enabled
Oct 26 08:15:05 localhost ntpd[19451]: Listening on interface eth0, 10.156.90.14#123 Enabled
Oct 26 08:15:05 localhost ntpd[19451]: Listening on interface eth1, 192.168.0.14#123 Enabled
Oct 26 08:15:05 localhost ntpd[19451]: kernel time sync status 0040
Oct 26 08:15:05 localhost gpm[19570]: *** info [startup.c(95)]:
Oct 26 08:15:05 localhost gpm[19570]: Started gpm successfully. Entered daemon mode.
Oct 26 08:15:06 localhost logger: Oracle Cluster Ready Services starting up automatically.
Oct 26 08:15:08 localhost sshd[19894]: Server listening on :: port 22.
Oct 26 08:15:08 localhost sshd[19894]: Server listening on 0.0.0.0 port 22.
Oct 26 08:15:09 localhost logger: Cluster Ready Services waiting on dependencies. Diagnostics in /tmp/crsctl.20095.
Oct 26 08:15:09 localhost logger: Cluster Ready Services waiting on dependencies. Diagnostics in /tmp/crsctl.19971.
Oct 26 08:15:09 localhost logger: Cluster Ready Services waiting on dependencies. Diagnostics in /tmp/crsctl.20024.
Oct 26 08:16:09 localhost logger: Cluster Ready Services waiting on dependencies. Diagnostics in /tmp/crsctl.20095.
Oct 26 08:16:09 localhost logger: Cluster Ready Services waiting on dependencies. Diagnostics in /tmp/crsctl.19971.
Oct 26 08:16:09 localhost logger: Cluster Ready Services waiting on dependencies. Diagnostics in /tmp/crsctl.20024.
Oct 26 08:17:09 localhost logger: Cluster Ready Services waiting on dependencies. Diagnostics in /tmp/crsctl.20095.
Oct 26 08:17:09 localhost logger: Cluster Ready Services waiting on dependencies. Diagnostics in /tmp/crsctl.19971.
Oct 26 08:17:09 localhost logger: Cluster Ready Services waiting on dependencies. Diagnostics in /tmp/crsctl.20024.
Oct 26 08:18:09 localhost kernel: bnx2: eth1 NIC Copper Link is Down
Oct 26 08:18:09 localhost logger: Cluster Ready Services waiting on dependencies. Diagnostics in /tmp/crsctl.19971.
Oct 26 08:18:09 localhost logger: Cluster Ready Services waiting on dependencies. Diagnostics in /tmp/crsctl.20095.
Oct 26 08:18:09 localhost logger: Cluster Ready Services waiting on dependencies. Diagnostics in /tmp/crsctl.20024.
Oct 26 08:18:11 localhost kernel: bnx2: eth1 NIC Copper Link is Up, 1000 Mbps full duplex, receive & transmit flow control ON
Oct 26 08:18:15 localhost kernel: usb 1-7.1.2: new high speed USB device using ehci_hcd and address 9
Oct 26 08:18:15 localhost kernel: usb 1-7.1.2: configuration #1 chosen from 1 choice
Oct 26 08:18:15 localhost kernel: scsi7 : SCSI emulation for USB Mass Storage devices
Oct 26 08:18:20 localhost kernel: Vendor: USB 2.0 Model: USB Flash Drive Rev: 0.00
Oct 26 08:18:20 localhost kernel: Type: Direct-Access ANSI SCSI revision: 02
Oct 26 08:18:20 localhost kernel: SCSI device sdaf: 15794176 512-byte hdwr sectors (8087 MB)
Oct 26 08:18:20 localhost kernel: sdaf: Write Protect is off
Oct 26 08:18:20 localhost kernel: sdaf: assuming drive cache: write through
Oct 26 08:18:20 localhost kernel: SCSI device sdaf: 15794176 512-byte hdwr sectors (8087 MB)
Oct 26 08:18:20 localhost kernel: sdaf: Write Protect is off
Oct 26 08:18:20 localhost kernel: sdaf: assuming drive cache: write through
Oct 26 08:18:20 localhost kernel: sdaf: sdaf1
Oct 26 08:18:20 localhost kernel: sd 7:0:0:0: Attached scsi removable disk sdaf
Oct 26 08:18:20 localhost kernel: sd 7:0:0:0: Attached scsi generic sg33 type 0
Oct 26 08:18:32 localhost kernel: o2net: connected to node e1198pdb3 (num 0) at 10.156.90.13:7777
Oct 26 08:18:33 localhost kernel: ocfs2_dlm: Node 0 joins domain FFC760CD2E4E48AF89C03CF9497A055C
Oct 26 08:18:33 localhost kernel: ocfs2_dlm: Nodes in domain ("FFC760CD2E4E48AF89C03CF9497A055C"): 0 1
Oct 26 08:18:37 localhost kernel: ocfs2_dlm: Node 0 joins domain 7FB9B3539B134F56958A9E44CCB25EB0
Oct 26 08:18:37 localhost kernel: ocfs2_dlm: Nodes in domain ("7FB9B3539B134F56958A9E44CCB25EB0"): 0 1
Oct 26 08:18:41 localhost kernel: ocfs2_dlm: Node 0 joins domain DC8F82EEB7F149CAB24B4802AA4105BD
Oct 26 08:18:41 localhost kernel: ocfs2_dlm: Nodes in domain ("DC8F82EEB7F149CAB24B4802AA4105BD"): 0 1
Oct 26 08:18:43 localhost kernel: usb 1-7.1.1: USB disconnect, address 8
Oct 26 08:18:45 localhost kernel: ocfs2_dlm: Node 0 joins domain EB71A901424745E6BEB746F3F4408035
Oct 26 08:18:45 localhost kernel: ocfs2_dlm: Nodes in domain ("EB71A901424745E6BEB746F3F4408035"): 0 1
Oct 26 08:18:49 localhost kernel: ocfs2_dlm: Node 0 joins domain C99F7F9A514643D6814D54420A7ACA5E
Oct 26 08:18:49 localhost kernel: ocfs2_dlm: Nodes in domain ("C99F7F9A514643D6814D54420A7ACA5E"): 0 1
Oct 26 08:18:53 localhost kernel: ocfs2_dlm: Node 0 joins domain 224D7E5B2D3143179334711BB8FAA8E4
Oct 26 08:18:53 localhost kernel: ocfs2_dlm: Nodes in domain ("224D7E5B2D3143179334711BB8FAA8E4"): 0 1
Oct 26 08:19:09 localhost logger: Cluster Ready Services waiting on dependencies. Diagnostics in /tmp/crsctl.20024.
Oct 26 08:19:09 localhost logger: Cluster Ready Services waiting on dependencies. Diagnostics in /tmp/crsctl.19971.
Oct 26 08:19:09 localhost logger: Cluster Ready Services waiting on dependencies. Diagnostics in /tmp/crsctl.20095.
Oct 26 08:19:17 localhost kernel: usb 1-7.1.1: new low speed USB device using ehci_hcd and address 10
Oct 26 08:19:17 localhost kernel: usb 1-7.1.1: configuration #1 chosen from 1 choice
Oct 26 08:19:17 localhost kernel: input: HID 04f3:0103 as /class/input/input7
Oct 26 08:19:17 localhost kernel: input: USB HID v1.10 Keyboard [HID 04f3:0103] on usb-0000:00:1d.7-7.1.1
Oct 26 08:19:17 localhost kernel: input: HID 04f3:0103 as /class/input/input8
Oct 26 08:19:17 localhost kernel: input: USB HID v1.10 Device [HID 04f3:0103] on usb-0000:00:1d.7-7.1.1
Oct 26 08:20:09 localhost logger: Cluster Ready Services waiting on dependencies. Diagnostics in /tmp/crsctl.20024.
Oct 26 08:20:09 localhost logger: Cluster Ready Services waiting on dependencies. Diagnostics in /tmp/crsctl.20095.
Oct 26 08:20:09 localhost logger: Cluster Ready Services waiting on dependencies. Diagnostics in /tmp/crsctl.19971.
Oct 26 08:21:09 localhost logger: Cluster Ready Services waiting on dependencies. Diagnostics in /tmp/crsctl.20095.
Oct 26 08:21:09 localhost logger: Cluster Ready Services waiting on dependencies. Diagnostics in /tmp/crsctl.19971.
Oct 26 08:21:09 localhost logger: Cluster Ready Services waiting on dependencies. Diagnostics in /tmp/crsctl.20024.
Oct 26 08:22:10 localhost logger: Cluster Ready Services waiting on dependencies. Diagnostics in /tmp/crsctl.20095.
Oct 26 08:22:10 localhost logger: Cluster Ready Services waiting on dependencies. Diagnostics in /tmp/crsctl.20024.
Oct 26 08:22:10 localhost logger: Cluster Ready Services waiting on dependencies. Diagnostics in /tmp/crsctl.19971.
Oct 26 08:23:10 localhost logger: Cluster Ready Services waiting on dependencies. Diagnostics in /tmp/crsctl.20095.
Oct 26 08:23:10 localhost logger: Cluster Ready Services waiting on dependencies. Diagnostics in /tmp/crsctl.20024.
Oct 26 08:23:10 localhost logger: Cluster Ready Services waiting on dependencies. Diagnostics in /tmp/crsctl.19971.
Oct 26 08:24:10 localhost logger: Cluster Ready Services waiting on dependencies. Diagnostics in /tmp/crsctl.19971.
Oct 26 08:24:10 localhost logger: Cluster Ready Services waiting on dependencies. Diagnostics in /tmp/crsctl.20024.
Oct 26 08:24:10 localhost logger: Cluster Ready Services waiting on dependencies. Diagnostics in /tmp/crsctl.20095.
Comments
Locked Post
New comments cannot be posted to this locked post.
Post Details
Locked on Nov 24 2009
Added on Oct 26 2009
#oracle-clusterware, #performance-availability
21 comments
6,757 views