Hi all,
need help in fixing the state.
ora.gipcd went to UNKNOWN state on both the nodes as below... this happened after I rolled back the latest CPU patch. Not able to bring down the CRS service.
crsctl stat res -t -init
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.asm
1 OFFLINE OFFLINE STABLE
ora.cluster_interconnect.haip
1 OFFLINE OFFLINE STABLE
ora.crf
1 ONLINE OFFLINE STABLE
ora.crsd
1 OFFLINE OFFLINE STABLE
ora.cssd
1 ONLINE OFFLINE STABLE
ora.cssdmonitor
1 OFFLINE OFFLINE STABLE
ora.ctssd
1 OFFLINE OFFLINE STABLE
ora.diskmon
1 OFFLINE OFFLINE STABLE
ora.drivers.acfs
1 ONLINE OFFLINE STABLE
ora.evmd
1 ONLINE OFFLINE STABLE
ora.gipcd
1 ONLINE UNKNOWN node 2 STABLE
ora.gpnpd
1 ONLINE OFFLINE STABLE
ora.mdnsd
1 ONLINE OFFLINE STABLE
ora.storage
1 OFFLINE OFFLINE STABLE
----------------------------------------------------------------------------
As root trying to bring down the CRS service got the below error message.
./crsctl stop crs
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'node2'
CRS-2679: Attempting to clean 'ora.gipcd' on 'node2'
CRS-2680: Clean of 'ora.gipcd' on 'node2' failed
CRS-2799: Failed to shut down resource 'ora.gipcd' on 'node2'
CRS-2795: Shutdown of Oracle High Availability Services-managed resources on 'node2' has failed
CRS-4687: Shutdown command has completed with errors.
CRS-4000: Command Stop failed, or completed with errors.
Alert log information :
2015-08-15 08:36:38.376 [OHASD(26365)]CRS-5828: Could not start agent '/u01/app/crs/12.1.0.2/grid/bin/orarootagent'. Details at (:CRSAGF00123:) {0:0:2401} in /u01/app/oracle/diag/crs/node2/crs/trace/ohasd.trc.
in the ohasd.trc file:-
2015-08-15 08:36:38.380805 :UiServer:1251952384: {0:0:2401} Response: c4|5!ORDERk7|MESSAGEt55|CRS-2679: Attempting to clean 'ora.gipcd' on 'node2'k7|MSGTYPEt1|3k5|OBJIDt9|ora.gipcdk4|WAITt1|0
2015-08-15 08:36:38.382291 : CRSMAIN:1264559872: {0:0:2401} {0:0:2401} Created alert : (:CRSAGF00123:) : Failed to start the agent process: /u01/app/crs/12.1.0.2/grid/bin/orarootagent Category: -1 Operation: fail Loc: canexec2 OS error: 0 Other : no exe permission, file [/u01/app/crs/12.1.0.2/grid/bin/orarootagent]
2015-08-15 08:36:38.386866 : CRSMAIN:1264559872: {0:0:2401} {0:0:2401} Created alert : (:CRSAGF00126:) : Agent start failed
2015-08-15 08:36:38.386874 : AGFW:1264559872: {0:0:2401} Rejecting pending msgs for ora.gipcd 1 1
2015-08-15 08:36:38.386934 : AGFW:1264559872: {0:0:2401} Could not forward message [RESOURCE_CLEAN[ora.gipcd 1 1] ID 4100:21434] to agent. /u01/app/crs/12.1.0.2/grid/bin/orarootagent_grid is not running
2015-08-15 08:36:38.389439 : AGFW:1264559872: {0:0:2401} Starting the agent: /u01/app/crs/12.1.0.2/grid/bin/orarootagent with user id: grid and incarnation:12
2015-08-15 08:36:38.398345 : CRSMAIN:1264559872: {0:0:2401} {0:0:2401} Created alert : (:CRSAGF00123:) : Failed to start the agent process: /u01/app/crs/12.1.0.2/grid/bin/orarootagent Category: -1 Operation: fail Loc: canexec2 OS error: 0 Other : no exe permission, file [/u01/app/crs/12.1.0.2/grid/bin/orarootagent]
2015-08-15 08:36:38.402443 : CRSMAIN:1264559872: {0:0:2401} {0:0:2401} Created alert : (:CRSAGF00126:) : Agent start failed
2015-08-15 08:36:38.402450 : AGFW:1264559872: {0:0:2401} Rejecting pending msgs for ora.gipcd 1 1
2015-08-15 08:36:38.402458 : AGFW:1264559872: {0:0:2401} Rejecting msg: 4100
2015-08-15 08:36:38.402488 : AGFW:1264559872: {0:0:2401} Agfw Proxy Server sending the last reply to PE for message:RESOURCE_CLEAN[ora.gipcd 1 1] ID 4100:21434
2015-08-15 08:36:38.402562 : CRSPE:1254053632: {0:0:2401} Received reply to action [Clean] message ID: 21434
2015-08-15 08:36:38.402571 : CRSPE:1254053632: {0:0:2401} RI [ora.gipcd 1 1] new internal state: [STABLE] old value: [CLEANING]
2015-08-15 08:36:38.407883 : CRSPE:1254053632: {0:0:2401} Fatal Error from AGFW Proxy: Unable to start the agent process
2015-08-15 08:36:38.423274 : CRSPE:1254053632: {0:0:2401} CRS-2680: Clean of 'ora.gipcd' on 'node2' failed
Need help in resolving this issue.
Cheers ! ! !
-Doll