As part of upgrade from 12cR1 to 19c applying patch 28553832.
Patch gets applied successfully but when trying to start the clsuterware using ./rootcrs.sh -postpatch the command fails with below messages
RS-2883: Resource 'ora.cssdmonitor' failed during Clusterware stack start.
CRS-4406: Oracle High Availability Services synchronous start failed.
CRS-4000: Command Start failed, or completed with errors.
2021/01/01 18:23:57 CLSRSC-117: Failed to start Oracle Clusterware stack
ERROR:
Died at /u01/app/12.0.0/grid/opatchautocfg/db/dbtmp/bootstrap_rac1/patchwork/crs/install/crspatch.pm line 946.
can someone help me identify the problem in here.
Thanks,
Kiran
Additional information from the crspatch_rac1_2021-01-01_06-41-15PM.log
>End Command output
2021-01-01 18:41:39: Oracle High Availability Services is online
2021-01-01 18:41:39: Modifying resource ora.gipcd to absorb changes to its type
2021-01-01 18:41:39: delete 'ora.gipcd' ...
2021-01-01 18:41:39: Executing cmd: /u01/app/12.0.0/grid/bin/crsctl delete resource ora.gipcd -f -init
2021-01-01 18:41:39: Succesfully executed action: delete, for resource ora.gipcd
2021-01-01 18:41:39: Executing cmd: /u01/app/12.0.0/grid/bin/crsctl delete type ora.gipc.type -init
2021-01-01 18:41:39: Registering resource type 'ora.gipc.type'
2021-01-01 18:41:39: Executing /u01/app/12.0.0/grid/bin/crsctl add type ora.gipc.type -basetype ora.daemon.type -file /u01/app/12.0.0/grid/log/rac1/ohasd/ora.gipc.type -init
2021-01-01 18:41:39: Executing cmd: /u01/app/12.0.0/grid/bin/crsctl add type ora.gipc.type -basetype ora.daemon.type -file /u01/app/12.0.0/grid/log/rac1/ohasd/ora.gipc.type -init
2021-01-01 18:41:39: Type 'ora.gipc.type' added successfully
2021-01-01 18:41:39: add 'ora.gipcd' ...
2021-01-01 18:41:39: Executing /u01/app/12.0.0/grid/bin/crsctl add resource ora.gipcd -attr " ACL='owner:root:rw-,pgrp:oinstall:rw-,other::r--,user:grid:rwx',ORACLE_USER=grid " -type ora.gipc.type -init
2021-01-01 18:41:39: Executing cmd: /u01/app/12.0.0/grid/bin/crsctl add resource ora.gipcd -attr " ACL='owner:root:rw-,pgrp:oinstall:rw-,other::r--,user:grid:rwx',ORACLE_USER=grid " -type ora.gipc.type -init
2021-01-01 18:41:39: Succesfully executed action: add, for resource ora.gipcd
2021-01-01 18:41:39: Executing cmd: /u01/app/12.0.0/grid/bin/crsctl stop crs -f
2021-01-01 18:41:39: Command output:
> CRS-4133: Oracle High Availability Services has been stopped.
>End Command output
2021-01-01 18:41:39: The return value of stop of CRS: 0
2021-01-01 18:41:39: Executing cmd: /u01/app/12.0.0/grid/bin/crsctl check crs
2021-01-01 18:41:40: Command output:
> CRS-4639: Could not contact Oracle High Availability Services
>End Command output
2021-01-01 18:41:40: Oracle CRS stack has been shut down
2021-01-01 18:41:40: Setting the modification time on file /u01/app/12.0.0/grid/oc4j/j2ee/home/applications/gridhome.ear
2021-01-01 18:41:40: Setting the modification time on file /u01/app/12.0.0/grid/oc4j/j2ee/home/applications/dbwlm.ear
2021-01-01 18:41:40: Starting Oracle Clusterware stack for rolling patch ...
2021-01-01 18:41:40: Executing cmd: /u01/app/12.0.0/grid/bin/crsctl start crs -wait
2021-01-01 18:52:02: Command output:
> CRS-4123: Starting Oracle High Availability Services-managed resources
> CRS-2672: Attempting to start 'ora.mdnsd' on 'rac1'
> CRS-2672: Attempting to start 'ora.evmd' on 'rac1'
> CRS-2676: Start of 'ora.mdnsd' on 'rac1' succeeded
> CRS-2676: Start of 'ora.evmd' on 'rac1' succeeded
> CRS-2672: Attempting to start 'ora.gpnpd' on 'rac1'
> CRS-2676: Start of 'ora.gpnpd' on 'rac1' succeeded
> CRS-2672: Attempting to start 'ora.gipcd' on 'rac1'
> CRS-2676: Start of 'ora.gipcd' on 'rac1' succeeded
> CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rac1'
> CRS-2676: Start of 'ora.cssdmonitor' on 'rac1' succeeded
> CRS-2672: Attempting to start 'ora.cssd' on 'rac1'
> CRS-2672: Attempting to start 'ora.diskmon' on 'rac1'
> CRS-2676: Start of 'ora.diskmon' on 'rac1' succeeded
> CRS-2883: Resource 'ora.cssdmonitor' failed during Clusterware stack start.
> CRS-4406: Oracle High Availability Services synchronous start failed.
> CRS-4000: Command Start failed, or completed with errors.
>End Command output
2021-01-01 18:52:02: The return value of blocking start of CRS: 1
2021-01-01 18:52:02: Failed to start the Oracle CRS stack