I am trying to install the two node RAC in the virtual machine.
The version:-
uname -a
Linux fairfax.sun 3.8.13-16.2.1.el6uek.x86_64 #1 SMP Thu Nov 7 17:01:44 PST 2013 x86_64 x86_64 x86_64 GNU/Linux
The error what I am getting during run of root.sh in the screen:-
RS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'fairfax'
CRS-2672: Attempting to start 'ora.ctssd' on 'fairfax'
CRS-2883: Resource 'ora.ctssd' failed during Clusterware stack start.
CRS-4406: Oracle High Availability Services synchronous start failed.
CRS-4000: Command Start failed, or completed with errors.
2018/10/16 13:50:36 CLSRSC-117: Failed to start Oracle Clusterware stack
Died at /u01/app/12.1.0/grid/crs/install/crsinstall.pm line 914.
The command '/u01/app/12.1.0/grid/perl/bin/perl -I/u01/app/12.1.0/grid/perl/lib -I/u01/app/12.1.0/grid/crs/install /u01/app/12.1.0/grid/crs/install/rootcrs.pl ' execution failed
Now checking on the failed node:-
/u01/app/12.1.0/grid/bin/crsctl stat res -t -init
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.asm
1 ONLINE OFFLINE STABLE
ora.cluster_interconnect.haip
1 ONLINE ONLINE fairfax STABLE
ora.crf
1 ONLINE OFFLINE STABLE
ora.crsd
1 ONLINE OFFLINE STABLE
ora.cssd
1 ONLINE ONLINE fairfax STABLE
ora.cssdmonitor
1 ONLINE ONLINE fairfax STABLE
ora.ctssd
1 ONLINE OFFLINE STABLE
ora.diskmon
1 OFFLINE OFFLINE STABLE
ora.drivers.acfs
1 ONLINE ONLINE fairfax STABLE
ora.evmd
1 ONLINE INTERMEDIATE fairfax STABLE
ora.gipcd
1 ONLINE ONLINE fairfax STABLE
ora.gpnpd
1 ONLINE ONLINE fairfax STABLE
ora.mdnsd
1 ONLINE ONLINE fairfax STABLE
ora.storage
1 ONLINE OFFLINE STABLE
Now staring to stop on the failed node:-
/u01/app/12.1.0/grid/bin/crsctl stop crs -f
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'fairfax'
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'fairfax'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'fairfax'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'fairfax'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'fairfax'
CRS-2677: Stop of 'ora.drivers.acfs' on 'fairfax' succeeded
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'fairfax' succeeded
CRS-2673: Attempting to stop 'ora.evmd' on 'fairfax'
CRS-2677: Stop of 'ora.mdnsd' on 'fairfax' succeeded
CRS-2677: Stop of 'ora.gpnpd' on 'fairfax' succeeded
CRS-2677: Stop of 'ora.evmd' on 'fairfax' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'fairfax'
CRS-2677: Stop of 'ora.cssd' on 'fairfax' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'fairfax'
CRS-2677: Stop of 'ora.gipcd' on 'fairfax' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'fairfax' has completed
CRS-4133: Oracle High Availability Services has been stopped.
Now again checking the status:-
/u01/app/12.1.0/grid/bin/crsctl stat res -t -init
CRS-4639: Could not contact Oracle High Availability Services
CRS-4000: Command Status failed, or completed with errors.
Now again restarting:-
BEfore that I have opened the ocssd.trc as tail - f:-
output of ocssd.trc in the failed node:-
/u01/app/12.1.0/grid/bin/crsctl start crs
CRS-4123: Oracle High Availability Services has been started.
Here I am only pasting the failed parts:-
| 2018-10-16 14:14:49.073724 : | CSSD:3866981952: clsu_load_ENV_levels: Module = CLSFA, LogLevel = 3, TraceLevel = 0 |
| CSSD:3866981952: clsugetconf : Configuration type [4]. |
| 2018-10-16 14:14:49.078131 : | CSSD:3866981952: clssscGetParameterOLR: OLR fetch for parameter auth rep (9) failed with rc 21 |
| 2018-10-16 14:14:49.078134 : | CSSD:3866981952: clssscagStartAgLsnr: Failed to get auth location from OLR, constructing manually |
| 2018-10-16 14:14:49.078138 : | CSSD:3866981952: clssscagStartAgLsnr: auth location '/u01/app/12.1.0/grid/auth/css/' |
| 2018-10-16 14:14:49.078346 : | CSSD:3866981952: clssscGPNPInit: PERF_TIME Initializing GPNP |
| 2018-10-16 14:14:49.078378 : | GPNP:3866981952: clsgpnp_Init: [at clsgpnp0.c:654] '/u01/app/12.1.0/grid' in effect as GPnP home base. |
| 2018-10-16 14:14:49.078381 : | GPNP:3866981952: clsgpnp_Init: [at clsgpnp0.c:708] GPnP pid=27443, cli= GPNP comp tracelevel=1, depcomp tracelevel=0, tlsrc:ORA_DAEMON_LOGGING_LEVELS, apitl:0, complog:1, tstenv:0, devenv:0, envopt:0, flags=3 |
| 2018-10-16 14:14:49.080586 : | GPNP:3866981952: clsgpnpkwf_initwfloc: [at clsgpnpkwf.c:402] Using FS Wallet Location : /u01/app/12.1.0/grid/gpnp/ |
| 2018-10-16 14:14:49.130369 : | CSSD:3866981952: clssscInitGlobalCTX: GIPCHA down 0 | | 2018-10-16 14:14:49.131591 : | CSSD:3866981952: clssscGetParameterOLR: OLR fetch for parameter logsize (8) failed with rc 21 | | 2018-10-16 14:14:49.131606 : | CSSD:3866981952: clssscqueue_init: queue(0xee7d10), max(0) | | 2018-10-16 14:14:49.131606 : | CSSD:3866981952: clssscUpdateInitState: Set state to 0x00080002, based on prior state of 0x00080000 and requested change of 0x00000002 |
| |
| |
| 2018-10-16 14:14:49.134280 : | CSSD:3866981952: clssscWaitOnInitState: returning 1, requested state 0x00001000, current state 0x00081002 |
| 2018-10-16 14:14:49.134297 : | CSSD:3866981952: clssscGetParameterProfile: profile fetch failed for parameter ocrid (4) with return code 5 |
| 2018-10-16 14:14:49.134298 : | CSSD:3866981952: clssscmain: OCRID is 0 |
| 2018-10-16 14:14:49.145247 : | CSSD:3846567680: clssgmGMCRequestThread: Starting |
| 2018-10-16 14:14:49.148472 : | CSSD:3866981952: clssscGetParameterOLR: OLR fetch for parameter auth rep (9) failed with rc 21 |
| 2018-10-16 14:14:49.181807 : | CLSF:3838940928: checksum failed for disk:ORCL:DISK1: |
| 2018-10-16 14:14:49.181815 : | CLSF:3838940928: Error: obj 2147483648 blk 0 name 'hard_kfbh' flags 0x65 first 1 |
| 2018-10-16 14:14:49.181816 : | CLSF:3838940928: 0: expected 1 actual 1 |
| 2018-10-16 14:14:49.181818 : | CLSF:3838940928: 1: expected 2 actual 130 |
| 2018-10-16 14:14:49.181819 : | CLSF:3838940928: 2: expected 1 actual 1 |
| 2018-10-16 14:14:49.181820 : | CLSF:3838940928: 3: expected 0 actual 0 |
| 2018-10-16 14:14:49.181821 : | CLSF:3838940928: 4: expected 0 actual 0 |
| 2018-10-16 14:14:49.181822 : | CLSF:3838940928: 5: expected 1 actual 1 |
| 2018-10-16 14:14:49.181823 : | CLSF:3838940928: 6: expected 1006530136 actual 1006530136 |
| 2018-10-16 14:14:49.181825 : | CLSF:3838940928: bh: ptr 0x7f9ad0244200 size 512 |
2018-10-16 14:14:49.181828 : SKGFD:3838940928: bh: dump of 0x0x7f9ad0244200, len 512
2018-10-16 14:14:49.181837 : SKGFD:3838940928: 0x0x7f9ad0244200 01 82 01 01 00 00 00 00 - 00 00 00 80 58 6e fe 3b ............Xn.;
2018-10-16 14:14:49.181844 : SKGFD:3838940928: 0x0x7f9ad0244210 00 00 00 00 00 00 00 00 - 00 00 00 00 00 00 00 00 ................
2018-10-16 14:14:49.181850 : SKGFD:3838940928: 0x0x7f9ad0244220 4f 52 43 4c 44 49 53 4b - 44 49 53 4b 31 00 00 00 ORCLDISKDISK1...
| 2018-10-16 14:14:49.191247 : | CLSF:3838940928: checksum failed for disk:ORCL:DISK2: |
| 2018-10-16 14:14:49.191252 : | CLSF:3838940928: Error: obj 2147483649 blk 0 name 'hard_kfbh' flags 0x65 first 1 |
| 2018-10-16 14:14:49.191253 : | CLSF:3838940928: 0: expected 1 actual 1 |
| 2018-10-16 14:14:49.191254 : | CLSF:3838940928: 1: expected 2 actual 130 |
| 2018-10-16 14:14:49.191255 : | CLSF:3838940928: 2: expected 1 actual 1 |
| 2018-10-16 14:14:49.191255 : | CLSF:3838940928: 3: expected 0 actual 0 |
| 2018-10-16 14:14:49.191256 : | CLSF:3838940928: 4: expected 0 actual 0 |
| 2018-10-16 14:14:49.191257 : | CLSF:3838940928: 5: expected 1 actual 1 |
| 2018-10-16 14:14:49.191257 : | CLSF:3838940928: 6: expected 1006530150 actual 1006530150 |
| 2018-10-16 14:14:49.191258 : | CLSF:3838940928: bh: ptr 0x7f9ad023fc00 size 512 |
| 2018-10-16 14:14:49.199462 : | CLSF:3838940928: checksum failed for disk:ORCL:DISK3: |
| 2018-10-16 14:14:49.199468 : | CLSF:3838940928: Error: obj 2147483650 blk 0 name 'hard_kfbh' flags 0x65 first 1 |
| 2018-10-16 14:14:49.199469 : | CLSF:3838940928: 0: expected 1 actual 1 |
| 2018-10-16 14:14:49.199470 : | CLSF:3838940928: 1: expected 2 actual 130 |
| 2018-10-16 14:14:49.199470 : | CLSF:3838940928: 2: expected 1 actual 1 |
| 2018-10-16 14:14:49.199471 : | CLSF:3838940928: 3: expected 0 actual 0 |
| 2018-10-16 14:14:49.199472 : | CLSF:3838940928: 4: expected 0 actual 0 |
| 2018-10-16 14:14:49.199472 : | CLSF:3838940928: 5: expected 1 actual 1 |
| 2018-10-16 14:14:49.199473 : | CLSF:3838940928: 6: expected 1006530138 actual 1006530138 |
| 2018-10-16 14:14:49.199474 : | CLSF:3838940928: bh: ptr 0x7f9ad0240400 size 512 |
GPnP service profile: local gpnpd is up and running. Use getProfile instead.
2018-10-16 14:14:49.276990 : GPNP:3866981952: clsgpnp_getCachedProfileEx: [at clsgpnp.c:678] Result: (26) CLSGPNP_NO_PROFILE. (:GPNP00108:)Failed to get offline GPnP service profile.
2018-10-16 14:14:49.277313 : GPNP:3866981952: clsgpnp_profileCallUrlInt: [at clsgpnp.c:2211] get-profile call to url "ipc://GPNPD_fairfax" disco "" [f=2 claimed- host: cname: cguid: cli:gipc p:27443 role: seq: ep: auth: diag:[]]
2018-10-16 14:14:49.295813 : GPNP:3866981952: clsgpnp_profileCallUrlInt: [at clsgpnp.c:2364] Result: (0) CLSGPNP_OK. Successful get-profile CALL to remote "ipc://GPNPD_fairfax" disco ""
2018-10-16 14:14:49.295842 :GIPCXCPT:3866981952: gipchaInternalReadGpnp: No HAIP network info configured in GPNP profile, using defaults, ret gipcretFail (1)
2018-10-16 14:14:49.295877 : GPNP:3866981952: clsgpnp_Term: [at clsgpnp0.c:1347] GPnP cli=gipc
2018-10-16 14:14:49.297181 :GIPCHGEN:3866981952: gipchaInternalReadGpnp: configuring default multicast addresses
2018-10-16 14:14:49.297183 :GIPCHGEN:3866981952: gipchaInternalReadGpnp: configuring default bootstrap communications modes
2018-10-16 14:14:49.297185 :GIPCHGEN:3866981952: gipchaInternalReadGpnp: configuring bootstrap communications
GPnP service profile: local gpnpd is up and running. Use getProfile instead.
2018-10-16 14:14:49.329199 : GPNP:3833132800: clsgpnp_getCachedProfileEx: [at clsgpnp.c:678] Result: (26) CLSGPNP_NO_PROFILE. (:GPNP00108:)Failed to get offline GPnP service profile.
2018-10-16 14:14:49.329575 : GPNP:3833132800: clsgpnp_profileCallUrlInt: [at clsgpnp.c:2211] get-profile call to url "ipc://GPNPD_fairfax" disco "" [f=2 claimed- host: cname: cguid: cli:gipc p:27443 role: seq: ep: auth: diag:[]]
2018-10-16 14:14:49.346015 : GPNP:3833132800: clsgpnp_profileCallUrlInt: [at clsgpnp.c:2364] Result: (0) CLSGPNP_OK. Successful get-profile CALL to remote "ipc://GPNPD_fairfax
| 018-10-16 14:14:49.363622 : | GPNP:3833132800: clsgpnp_getCachedProfileEx: [at clsgpnp.c:667] Result: (26) CLSGPNP_NO_PROFILE. (:GPNP00107:)Can't get offline GPnP service profile: local gpnpd is up and running. Use getProfile instead. |
| 2018-10-16 14:14:49.363628 : | GPNP:3833132800: clsgpnp_getCachedProfileEx: [at clsgpnp.c:678] Result: (26) CLSGPNP_NO_PROFILE. (:GPNP00108:)Failed to get offline GPnP service profile. |
| 2018-10-16 14:14:49.363794 : | GPNP:3833132800: clsgpnp_profileCallUrlInt: [at clsgpnp.c:2211] get-profile call to url "ipc://GPNPD_fairfax" disco "" [f=2 claimed- host: cname: cguid: cli:gipc p:27443 role: seq: ep: auth: diag:[]] |
| 2018-10-16 14:14:49.380894 : | GPNP:3833132800: clsgpnp_profileCallUrlInt: [at clsgpnp.c:2364] Result: (0) CLSGPNP_OK. Successful get-profile CALL to remote "ipc://GPNPD_fairfax" disco "" |
2018-10-16 14:14:49.380922 : GIPCLIB:3833132800: gipclibGetClusterGuid: retrieved cluster guid 23c80cdc0b954f4abf459345a059bef2
| 2018-10-16 14:14:49.380928 : | GPNP:3833132800: clsgpnp_Term: [at clsgpnp0.c:1347] GPnP cli=gipc |
| 2018-10-16 14:14:49.390696 : | GPNP:3833132800: clsgpnp_Init: [at clsgpnp0.c:654] '/u01/app/12.1.0/grid' in effect as GPnP home base. |
2018-10-16 14:14:49.3907