Hi All,
I am doing RAC GI 12.1.0.2 installation on RHEL 7.2 over ASM Disk using asmlib , 3 Nodes cluster of Physical HP box.While GI software installation and configuration ,on the primary node where the installation triggered the root.sh script execution passed , i am seeing failure on other 2 nodes for root script execution as follows.
I am doing the installation second time on the same setup. How do i troubleshoot and resolve this issue?
When i run the root script manually i get the following failure.
[grid@cft-cl20-7s4 cft-cl20-7s4]$ uname -a
Linux cft-cl20-7s4.qpnt.lab.hp.com 3.10.0-327.el7.x86_64 #1 SMP Thu Oct 29 17:29:29 EDT 2015 x86_64 x86_64 x86_64 GNU/Linux
[root@cft-cl20-7s10 ~]# /home/app/12/grid/root.sh
Performing root user operation.
The following environment variables are set as:
ORACLE_OWNER= grid
ORACLE_HOME= /home/app/12/grid
Copying dbhome to /usr/local/bin ...
Copying oraenv to /usr/local/bin ...
Copying coraenv to /usr/local/bin ...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /home/app/12/grid/crs/install/crsconfig_params
2016/11/22 16:23:31 CLSRSC-4001: Installing Oracle Trace File Analyzer (TFA) Collector.
2016/11/22 16:23:31 CLSRSC-4002: Successfully installed Oracle Trace File Analyzer (TFA) Collector.
2016/11/22 16:23:32 CLSRSC-363: User ignored prerequisites during installation
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'cft-cl20-7s10'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'cft-cl20-7s10'
CRS-2673: Attempting to stop 'ora.gipcd' on 'cft-cl20-7s10'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'cft-cl20-7s10'
CRS-2673: Attempting to stop 'ora.evmd' on 'cft-cl20-7s10'
CRS-2673: Attempting to stop 'ora.cssdmonitor' on 'cft-cl20-7s10'
CRS-2677: Stop of 'ora.cssdmonitor' on 'cft-cl20-7s10' succeeded
CRS-2677: Stop of 'ora.gipcd' on 'cft-cl20-7s10' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'cft-cl20-7s10' succeeded
CRS-2677: Stop of 'ora.gpnpd' on 'cft-cl20-7s10' succeeded
CRS-2677: Stop of 'ora.evmd' on 'cft-cl20-7s10' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'cft-cl20-7s10' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Starting Oracle High Availability Services-managed resources
CRS-2672: Attempting to start 'ora.mdnsd' on 'cft-cl20-7s10'
CRS-2672: Attempting to start 'ora.evmd' on 'cft-cl20-7s10'
CRS-2676: Start of 'ora.mdnsd' on 'cft-cl20-7s10' succeeded
CRS-2676: Start of 'ora.evmd' on 'cft-cl20-7s10' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'cft-cl20-7s10'
CRS-2676: Start of 'ora.gpnpd' on 'cft-cl20-7s10' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'cft-cl20-7s10'
CRS-2676: Start of 'ora.gipcd' on 'cft-cl20-7s10' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'cft-cl20-7s10'
CRS-2676: Start of 'ora.cssdmonitor' on 'cft-cl20-7s10' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'cft-cl20-7s10'
CRS-2672: Attempting to start 'ora.diskmon' on 'cft-cl20-7s10'
CRS-2676: Start of 'ora.diskmon' on 'cft-cl20-7s10' succeeded
CRS-2883: Resource 'ora.cssdmonitor' failed during Clusterware stack start.
CRS-4406: Oracle High Availability Services synchronous start failed.
CRS-4000: Command Start failed, or completed with errors.
2016/11/22 16:35:35 CLSRSC-117: Failed to start Oracle Clusterware stack
Died at /home/app/12/grid/crs/install/crsinstall.pm line 914.
The command '/home/app/12/grid/perl/bin/perl -I/home/app/12/grid/perl/lib -I/home/app/12/grid/crs/install /home/app/12/grid/crs/install/rootcrs.pl ' execution failed
[root@cft-cl20-7s9 OraInstall2016-11-22_12-31-21AM]# oracleasm listdisks
ASM_DISK01
ASM_DISK02
ASM_DISK03
ASM_DISK04
OCR_VOTE01
OCR_VOTE02
OCR_VOTE03
OCR_VOTE04
OCR_VOTE05
OCR_VOTE06
From Installation logs
INFO: *********************************************
INFO: Execute root script on all nodes failed.
INFO: Overall status of execution of root/configuration scripts : failed
INFO: Shutting down OUISetupDriver.JobExecutorThread
INFO: Cleaning up, please wait...
INFO: All forked task are completed at state setup
INFO: Completed background operations
INFO: Moved to state <setup>
INFO: Number of scripts to be executed as root user = 2
INFO: Selecting job named 'Execute root script on all nodes' for retry
INFO: Execute root script on all nodes in progress.
INFO: executing /home/app/oraInventory/orainstRoot.sh script on nodes: [cft-cl20-7s9, cft-cl20-7s10]
INFO: status of /home/app/oraInventory/orainstRoot.sh script execution is:true
INFO: *********************************************
INFO: Execution of /home/app/oraInventory/orainstRoot.sh script is successful on nodes : [cft-cl20-7s10, cft-cl20-7s9]
Execution status of succeeded node:cft-cl20-7s10
Standard output : Changing permissions of /home/app/oraInventory.
Adding read,write permissions for group.
Removing read,write,execute permissions for world.
Changing groupname of /home/app/oraInventory to oinstall.
The execution of the script is complete.
Execution status of succeeded node:cft-cl20-7s9
Standard output : Changing permissions of /home/app/oraInventory.
Adding read,write permissions for group.
Removing read,write,execute permissions for world.
Changing groupname of /home/app/oraInventory to oinstall.
The execution of the script is complete.
Execution status of cft-cl20-7s10 is:true
Execution exit code of cft-cl20-7s10 is:0
Execution status of cft-cl20-7s9 is:true
Execution exit code of cft-cl20-7s9 is:0
INFO: *********************************************
INFO: executing GI Install script on nodes: [cft-cl20-7s4, cft-cl20-7s9, cft-cl20-7s10]
INFO: PRCZ-2012 : cft-cl20-7s4 configuration started
INFO: PRCZ-2013 : cft-cl20-7s4 configuration ended
INFO: Progress:33
INFO: PRCZ-2012 : cft-cl20-7s9 configuration started
INFO: PRCZ-2012 : cft-cl20-7s10 configuration started
INFO: PRCZ-2013 : cft-cl20-7s9 configuration ended
INFO: Progress:66
INFO: PRCZ-2013 : cft-cl20-7s10 configuration ended
INFO: Progress:100
INFO: status of GI Install script execution is:false
INFO: *********************************************
INFO: Execution of GI Install script is successful on nodes : [cft-cl20-7s4]
Execution of GI Install script is failed on nodes : [cft-cl20-7s10, cft-cl20-7s9]
Execution status of failed node:cft-cl20-7s10
Standard output : Performing root user operation.
The following environment variables are set as:
ORACLE_OWNER= grid
ORACLE_HOME= /home/app/12/grid
Copying dbhome to /usr/local/bin ...
Copying oraenv to /usr/local/bin ...
Copying coraenv to /usr/local/bin ...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /home/app/12/grid/crs/install/crsconfig_params
2016/11/22 14:02:09 CLSRSC-4001: Installing Oracle Trace File Analyzer (TFA) Collector.
2016/11/22 14:02:09 CLSRSC-4002: Successfully installed Oracle Trace File Analyzer (TFA) Collector.
2016/11/22 14:02:10 CLSRSC-363: User ignored prerequisites during installation
2016/11/22 14:02:44 CLSRSC-507: The root script cannot proceed on this node cft-cl20-7s10 because either the first-node operations have not completed on node cft-cl20-7s4 or there was an error in obtaining the status of the first-node operations.
Died at /home/app/12/grid/crs/install/crsutils.pm line 3681.
The command '/home/app/12/grid/perl/bin/perl -I/home/app/12/grid/perl/lib -I/home/app/12/grid/crs/install /home/app/12/grid/crs/install/rootcrs.pl -auto -lang=en_US.UTF-8' execution failed
Execution status of failed node:cft-cl20-7s9
Standard output : Performing root user operation.
The following environment variables are set as:
ORACLE_OWNER= grid
ORACLE_HOME= /home/app/12/grid
Copying dbhome to /usr/local/bin ...
Copying oraenv to /usr/local/bin ...
Copying coraenv to /usr/local/bin ...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /home/app/12/grid/crs/install/crsconfig_params
2016/11/22 14:02:09 CLSRSC-4001: Installing Oracle Trace File Analyzer (TFA) Collector.
2016/11/22 14:02:09 CLSRSC-4002: Successfully installed Oracle Trace File Analyzer (TFA) Collector.
2016/11/22 14:02:10 CLSRSC-363: User ignored prerequisites during installation
2016/11/22 14:02:44 CLSRSC-507: The root script cannot proceed on this node cft-cl20-7s9 because either the first-node operations have not completed on node cft-cl20-7s4 or there was an error in obtaining the status of the first-node operations.
Died at /home/app/12/grid/crs/install/crsutils.pm line 3681.
The command '/home/app/12/grid/perl/bin/perl -I/home/app/12/grid/perl/lib -I/home/app/12/grid/crs/install /home/app/12/grid/crs/install/rootcrs.pl -auto -lang=en_US.UTF-8' execution failed
Execution status of succeeded node:cft-cl20-7s4
Standard output :
Performing root user operation.
The following environment variables are set as:
ORACLE_OWNER= grid
ORACLE_HOME= /home/app/12/grid
Copying dbhome to /usr/local/bin ...
Copying oraenv to /usr/local/bin ...
Copying coraenv to /usr/local/bin ...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /home/app/12/grid/crs/install/crsconfig_params
2016/11/22 14:02:06 CLSRSC-4001: Installing Oracle Trace File Analyzer (TFA) Collector.
2016/11/22 14:02:07 CLSRSC-4002: Successfully installed Oracle Trace File Analyzer (TFA) Collector.
2016/11/22 14:02:08 CLSRSC-456: The Oracle Grid Infrastructure has already been configured.
Execution status of cft-cl20-7s4 is:true
Execution exit code of cft-cl20-7s4 is:0
Execution status of cft-cl20-7s10 is:true
Execution exit code of cft-cl20-7s10 is:25
Execution status of cft-cl20-7s9 is:true
Execution exit code of cft-cl20-7s9 is:25
INFO: *********************************************
INFO: Execute root script on all nodes failed.
INFO: Overall status of execution of root/configuration scripts : failed
INFO: Dispose the current Session instance
INFO: Dispose the install area control object
INFO: Update the state machine to STATE_CLEAN
INFO: Adding ExitStatus SUCCESS_WITH_WARNINGS to the exit status set
INFO: Finding the most appropriate exit status for the current application
INFO: Exit Status is 6
INFO: List of warnings encountered in this Application:
INFO: PREREQS_FAILED_WITH_WARNING
INFO: Shutdown Oracle Grid Infrastructure 12c Release 1 Installer
Please help in resolving error.
Thanks