Failed to start Oracle Clusterware stack on node2
972371Nov 4 2012 — edited Nov 5 2012Hi,
I find problem when install Grid Infrastructure, node 2 failed to start clusterware. Please help me to solve this.
on node1
==============================================================
root@may50 # /u01/app/11.2.0/grid/root.sh
Running Oracle 11g root.sh script...
The following environment variables are set as:
ORACLE_OWNER= oracle
ORACLE_HOME= /u01/app/11.2.0/grid
Enter the full pathname of the local bin directory: [usr/local/bin]:
The file "dbhome" already exists in /usr/local/bin. Overwrite it? (y/n) [n]: y
Copying dbhome to /usr/local/bin ...
The file "oraenv" already exists in /usr/local/bin. Overwrite it? (y/n) [n]: y
Copying oraenv to /usr/local/bin ...
The file "coraenv" already exists in /usr/local/bin. Overwrite it? (y/n) [n]: y
Copying coraenv to /usr/local/bin ...
Entries will be added to the /var/opt/oracle/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root.sh script.
Now product-specific root actions will be performed.
2000-01-01 09:57:09: Parsing the host name
2000-01-01 09:57:09: Checking for super user privileges
2000-01-01 09:57:09: User has super user privileges
Using configuration parameter file: /u01/app/11.2.0/grid/crs/install/crsconfig_params
Creating trace directory
LOCAL ADD MODE
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
root wallet
root wallet cert
root cert export
peer wallet
profile reader wallet
pa wallet
peer wallet keys
pa wallet keys
peer cert request
pa cert request
peer cert
pa cert
peer root cert TP
profile reader root cert TP
pa root cert TP
peer pa cert TP
pa peer cert TP
profile reader pa cert TP
profile reader peer cert TP
peer user cert
pa user cert
Adding daemon to inittab
Jan 1 09:58:54 may50 root: exec /u01/app/11.2.0/grid/perl/bin/perl -I/u01/app/11.2.0/grid/perl/lib /u01/app/11.2.0/grid/bin/crswrapexece.pl /u01/app/11.2.0/grid/crs/install/s_crsconfig_may50_env.txt /u01/app/11.2.0/grid/bin/ohasd.bin "reboot"
CRS-4123: Oracle High Availability Services has been started.
ohasd is starting
Jan 1 09:59:27 may50 mDNSResponder (Engineering Build) (Nov 2 2009 05:02:07) [5180]: starting
Jan 1 09:59:28 may50 mDNSResponder: Oracle mDNSResponder starting
Jan 1 09:59:28 may50 mDNSResponder: mdnsd found interface name=lo0 AF=2
Jan 1 09:59:28 may50 mDNSResponder: mdnsd found interface name=ce0 AF=2
Jan 1 09:59:28 may50 mDNSResponder: mdnsd found interface name=ce2 AF=2
Jan 1 09:59:28 may50 mDNSResponder: Unable to parse DNS server list. Unicast DNS-SD unavailable
CRS-2672: Attempting to start 'ora.gipcd' on 'may50'
CRS-2672: Attempting to start 'ora.mdnsd' on 'may50'
CRS-2676: Start of 'ora.gipcd' on 'may50' succeeded
CRS-2676: Start of 'ora.mdnsd' on 'may50' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'may50'
CRS-2676: Start of 'ora.gpnpd' on 'may50' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'may50'
CRS-2676: Start of 'ora.cssdmonitor' on 'may50' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'may50'
CRS-2672: Attempting to start 'ora.diskmon' on 'may50'
CRS-2676: Start of 'ora.diskmon' on 'may50' succeeded
CRS-2676: Start of 'ora.cssd' on 'may50' succeeded
CRS-2672: Attempting to start 'ora.ctssd' on 'may50'
CRS-2676: Start of 'ora.ctssd' on 'may50' succeeded
ASM created and started successfully.
DiskGroup DATA created successfully.
clscfg: -install mode specified
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
CRS-2672: Attempting to start 'ora.crsd' on 'may50'
CRS-2676: Start of 'ora.crsd' on 'may50' succeeded
CRS-4256: Updating the profile
Successful addition of voting disk 103138bfd53b4f84bfb7c36adfeeb442.
Successful addition of voting disk 8b7d610bedc74f65bf91b46ff4a816ec.
Successful addition of voting disk 6d360ad323a64fd9bf66ff5638e6ede6.
Successfully replaced voting disk group with +DATA.
CRS-4256: Updating the profile
CRS-4266: Voting file(s) successfully replaced
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE 103138bfd53b4f84bfb7c36adfeeb442 (/dev/rdsk/c0t40d2s3) [DATA]
2. ONLINE 8b7d610bedc74f65bf91b46ff4a816ec (/dev/rdsk/c0t40d2s4) [DATA]
3. ONLINE 6d360ad323a64fd9bf66ff5638e6ede6 (/dev/rdsk/c0t40d2s5) [DATA]
Located 3 voting disk(s).
CRS-2673: Attempting to stop 'ora.crsd' on 'may50'
CRS-2677: Stop of 'ora.crsd' on 'may50' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'may50'
CRS-2677: Stop of 'ora.asm' on 'may50' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'may50'
CRS-2677: Stop of 'ora.ctssd' on 'may50' succeeded
CRS-2673: Attempting to stop 'ora.cssdmonitor' on 'may50'
CRS-2677: Stop of 'ora.cssdmonitor' on 'may50' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'may50'
CRS-2677: Stop of 'ora.cssd' on 'may50' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'may50'
CRS-2677: Stop of 'ora.gpnpd' on 'may50' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'may50'
CRS-2677: Stop of 'ora.gipcd' on 'may50' succeeded
CRS-2673: Attempting to stop 'ora.mdnsd' on 'may50'
Jan 1 10:01:45 may50 mDNSResponder: clsdm requested mdnsd exit
Jan 1 10:01:45 may50 mDNSResponder: mdnsd exit
Jan 1 10:01:48 may50 mDNSResponder (Engineering Build) (Nov 2 2009 05:02:07) [5180]: stopping
CRS-2677: Stop of 'ora.mdnsd' on 'may50' succeeded
CRS-2672: Attempting to start 'ora.mdnsd' on 'may50'
Jan 1 10:01:51 may50 mDNSResponder (Engineering Build) (Nov 2 2009 05:02:07) [5679]: starting
Jan 1 10:01:51 may50 mDNSResponder: Oracle mDNSResponder starting
Jan 1 10:01:51 may50 mDNSResponder: mdnsd found interface name=lo0 AF=2
Jan 1 10:01:51 may50 mDNSResponder: mdnsd found interface name=ce0 AF=2
Jan 1 10:01:51 may50 mDNSResponder: mdnsd found interface name=ce2 AF=2
Jan 1 10:01:51 may50 mDNSResponder: Unable to parse DNS server list. Unicast DNS-SD unavailable
CRS-2676: Start of 'ora.mdnsd' on 'may50' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'may50'
CRS-2676: Start of 'ora.gipcd' on 'may50' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'may50'
CRS-2676: Start of 'ora.gpnpd' on 'may50' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'may50'
CRS-2676: Start of 'ora.cssdmonitor' on 'may50' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'may50'
CRS-2672: Attempting to start 'ora.diskmon' on 'may50'
CRS-2676: Start of 'ora.diskmon' on 'may50' succeeded
CRS-2676: Start of 'ora.cssd' on 'may50' succeeded
CRS-2672: Attempting to start 'ora.ctssd' on 'may50'
CRS-2676: Start of 'ora.ctssd' on 'may50' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'may50'
CRS-2676: Start of 'ora.asm' on 'may50' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'may50'
CRS-2676: Start of 'ora.crsd' on 'may50' succeeded
CRS-2672: Attempting to start 'ora.evmd' on 'may50'
CRS-2676: Start of 'ora.evmd' on 'may50' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'may50'
CRS-2676: Start of 'ora.asm' on 'may50' succeeded
CRS-2672: Attempting to start 'ora.DATA.dg' on 'may50'
CRS-2676: Start of 'ora.DATA.dg' on 'may50' succeeded
Jan 1 10:04:47 may50 mDNSResponder: mdnsd found interface name=lo0 AF=2
Jan 1 10:04:47 may50 mDNSResponder: mdnsd found interface name=ce0 AF=2
Jan 1 10:04:47 may50 mDNSResponder: mdnsd found interface name=ce2 AF=2
Jan 1 10:04:47 may50 mDNSResponder: mdnsd found interface name=lo0 AF=2
Jan 1 10:04:47 may50 mDNSResponder: mdnsd found interface name=ce0 AF=2
Jan 1 10:04:47 may50 mDNSResponder: mdnsd found interface name=ce0:1 AF=2
Jan 1 10:04:47 may50 mDNSResponder: mdnsd found interface name=ce2 AF=2
Jan 1 10:04:47 may50 mDNSResponder: mdnsd found interface name=lo0 AF=2
Jan 1 10:04:47 may50 mDNSResponder: mdnsd found interface name=ce0 AF=2
Jan 1 10:04:47 may50 mDNSResponder: mdnsd found interface name=ce0:1 AF=2
Jan 1 10:04:47 may50 mDNSResponder: mdnsd found interface name=ce2 AF=2
Jan 1 10:04:47 may50 mDNSResponder: mdnsd found interface name=lo0 AF=2
Jan 1 10:04:47 may50 mDNSResponder: mdnsd found interface name=ce0 AF=2
Jan 1 10:04:47 may50 mDNSResponder: mdnsd found interface name=ce0:1 AF=2
Jan 1 10:04:47 may50 mDNSResponder: mdnsd found interface name=ce2 AF=2
Jan 1 10:05:07 may50 mDNSResponder: mdnsd found interface name=lo0 AF=2
Jan 1 10:05:07 may50 mDNSResponder: mdnsd found interface name=ce0 AF=2
Jan 1 10:05:07 may50 mDNSResponder: mdnsd found interface name=ce0:1 AF=2
Jan 1 10:05:07 may50 mDNSResponder: mdnsd found interface name=ce2 AF=2
Jan 1 10:05:07 may50 mDNSResponder: mdnsd found interface name=lo0 AF=2
Jan 1 10:05:07 may50 mDNSResponder: mdnsd found interface name=ce0 AF=2
Jan 1 10:05:07 may50 mDNSResponder: mdnsd found interface name=ce0:1 AF=2
Jan 1 10:05:07 may50 mDNSResponder: mdnsd found interface name=ce0:2 AF=2
Jan 1 10:05:07 may50 mDNSResponder: mdnsd found interface name=ce2 AF=2
Jan 1 10:05:07 may50 mDNSResponder: mdnsd found interface name=lo0 AF=2
Jan 1 10:05:07 may50 mDNSResponder: mdnsd found interface name=ce0 AF=2
Jan 1 10:05:07 may50 mDNSResponder: mdnsd found interface name=ce0:1 AF=2
Jan 1 10:05:07 may50 mDNSResponder: mdnsd found interface name=ce0:2 AF=2
Jan 1 10:05:07 may50 mDNSResponder: mdnsd found interface name=ce2 AF=2
Jan 1 10:05:07 may50 mDNSResponder: mdnsd found interface name=lo0 AF=2
Jan 1 10:05:07 may50 mDNSResponder: mdnsd found interface name=ce0 AF=2
Jan 1 10:05:07 may50 mDNSResponder: mdnsd found interface name=ce0:1 AF=2
Jan 1 10:05:07 may50 mDNSResponder: mdnsd found interface name=ce0:2 AF=2
Jan 1 10:05:07 may50 mDNSResponder: mdnsd found interface name=ce2 AF=2
may50 2000/01/01 10:05:15 /u01/app/11.2.0/grid/cdata/may50/backup_20000101_100515.olr
Configure Oracle Grid Infrastructure for a Cluster ... succeeded
Updating inventory properties for clusterware
Starting Oracle Universal Installer...
Checking swap space: must be greater than 500 MB. Actual 14511 MB Passed
The inventory pointer is located at /var/opt/oracle/oraInst.loc
The inventory is located at /u01/app/oraInventory
'UpdateNodeList' was successful.
root@may50 #
==============================================================
on node2
==============================================================
bash-3.2# /u01/app/11.2.0/grid/root.sh
Running Oracle 11g root.sh script...
The following environment variables are set as:
ORACLE_OWNER= oracle
ORACLE_HOME= /u01/app/11.2.0/grid
Enter the full pathname of the local bin directory: [usr/local/bin]:
The file "dbhome" already exists in /usr/local/bin. Overwrite it? (y/n) [n]: y
Copying dbhome to /usr/local/bin ...
The file "oraenv" already exists in /usr/local/bin. Overwrite it? (y/n) [n]: y
Copying oraenv to /usr/local/bin ...
The file "coraenv" already exists in /usr/local/bin. Overwrite it? (y/n) [n]: y
Copying coraenv to /usr/local/bin ...
Entries will be added to the /var/opt/oracle/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root.sh script.
Now product-specific root actions will be performed.
2000-01-01 10:01:50: Parsing the host name
2000-01-01 10:01:50: Checking for super user privileges
2000-01-01 10:01:50: User has super user privileges
Using configuration parameter file: /u01/app/11.2.0/grid/crs/install/crsconfig_params
Creating trace directory
LOCAL ADD MODE
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
Adding daemon to inittab
Jan 1 10:02:00 may52 root: exec /u01/app/11.2.0/grid/perl/bin/perl -I/u01/app/11.2.0/grid/perl/lib /u01/app/11.2.0/grid/bin/crswrapexece.pl /u01/app/11.2.0/grid/crs/install/s_crsconfig_may52_env.txt /u01/app/11.2.0/grid/bin/ohasd.bin "reboot"
CRS-4123: Oracle High Availability Services has been started.
ohasd is starting
Jan 1 10:02:34 may52 mDNSResponder (Engineering Build) (Nov 2 2009 05:02:07) [3613]: starting
Jan 1 10:02:34 may52 mDNSResponder: Oracle mDNSResponder starting
Jan 1 10:02:34 may52 mDNSResponder: mdnsd found interface name=lo0 AF=2
Jan 1 10:02:34 may52 mDNSResponder: mdnsd found interface name=ce0 AF=2
Jan 1 10:02:34 may52 mDNSResponder: mdnsd found interface name=ce2 AF=2
Jan 1 10:02:34 may52 mDNSResponder: Unable to parse DNS server list. Unicast DNS-SD unavailable
CRS-4402: The CSS daemon was started in exclusive mode but found an active CSS daemon on node may50, number 1, and is terminating
CRS-2673: Attempting to stop 'ora.cssdmonitor' on 'may52'
CRS-2677: Stop of 'ora.cssdmonitor' on 'may52' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'may52'
CRS-2677: Stop of 'ora.gpnpd' on 'may52' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'may52'
CRS-2677: Stop of 'ora.gipcd' on 'may52' succeeded
CRS-2673: Attempting to stop 'ora.mdnsd' on 'may52'
Jan 1 10:03:34 may52 mDNSResponder: clsdm requested mdnsd exit
Jan 1 10:03:34 may52 mDNSResponder: mdnsd exit
Jan 1 10:03:37 may52 mDNSResponder (Engineering Build) (Nov 2 2009 05:02:07) [3613]: stopping
CRS-2677: Stop of 'ora.mdnsd' on 'may52' succeeded
An active cluster was found during exclusive startup, restarting to join the cluster
CRS-2672: Attempting to start 'ora.mdnsd' on 'may52'
Jan 1 10:03:45 may52 mDNSResponder (Engineering Build) (Nov 2 2009 05:02:07) [3775]: starting
Jan 1 10:03:46 may52 mDNSResponder: Oracle mDNSResponder starting
Jan 1 10:03:46 may52 mDNSResponder: mdnsd found interface name=lo0 AF=2
Jan 1 10:03:46 may52 mDNSResponder: mdnsd found interface name=ce0 AF=2
Jan 1 10:03:46 may52 mDNSResponder: mdnsd found interface name=ce2 AF=2
Jan 1 10:03:46 may52 mDNSResponder: Unable to parse DNS server list. Unicast DNS-SD unavailable
CRS-2676: Start of 'ora.mdnsd' on 'may52' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'may52'
CRS-2676: Start of 'ora.gipcd' on 'may52' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'may52'
CRS-2676: Start of 'ora.gpnpd' on 'may52' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'may52'
CRS-2676: Start of 'ora.cssdmonitor' on 'may52' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'may52'
CRS-2672: Attempting to start 'ora.diskmon' on 'may52'
CRS-2674: Start of 'ora.diskmon' on 'may52' failed
CRS-2679: Attempting to clean 'ora.diskmon' on 'may52'
CRS-2674: Start of 'ora.cssd' on 'may52' failed
CRS-2679: Attempting to clean 'ora.cssd' on 'may52'
CRS-2681: Clean of 'ora.cssd' on 'may52' succeeded
CRS-2679: Attempting to clean 'ora.diskmon' on 'may52'
CRS-2680: Clean of 'ora.diskmon' on 'may52' failed
CRS-2681: Clean of 'ora.diskmon' on 'may52' succeeded
CRS-4000: Command Start failed, or completed with errors.
CRS-2672: Attempting to start 'ora.cssd' on 'may52'
CRS-2672: Attempting to start 'ora.diskmon' on 'may52'
CRS-2674: Start of 'ora.diskmon' on 'may52' failed
CRS-2679: Attempting to clean 'ora.diskmon' on 'may52'
CRS-5016: Process "/u01/app/11.2.0/grid/bin/diskmon" spawned by agent "/u01/app/11.2.0/grid/bin/orarootagent.bin" for action "clean" failed: details at "(:CLSN00010:)" in "/u01/app/11.2.0/grid/log/may52/agent/ohasd/orarootagent_root/orarootagent_root.log"
CRS-2681: Clean of 'ora.diskmon' on 'may52' succeeded
CRS-2674: Start of 'ora.cssd' on 'may52' failed
CRS-2679: Attempting to clean 'ora.cssd' on 'may52'
CRS-2678: 'ora.cssd' on 'may52' has experienced an unrecoverable failure
CRS-0267: Human intervention required to resume its availability.
CRS-4000: Command Start failed, or completed with errors.
Command return code of 1 (256) from command: /u01/app/11.2.0/grid/bin/crsctl start resource ora.ctssd -init -env USR_ORA_ENV=CTSS_REBOOT=TRUE
Start of resource "ora.ctssd -init -env USR_ORA_ENV=CTSS_REBOOT=TRUE" failed
Failed to start CTSS
Failed to start Oracle Clusterware stack
bash-3.2#
==============================================================
Thank you.
Tuan