GI version: 11.2.0.4
Platform : Oracle Linux 6.4
Since the cluster was hung, I was trying bring down the cluster. It didn't work with crsctl stop crs . So , I used -f option.
Still couldn't shutdown the cluster .
Tried killing crsd hoping that all the remaining processes will crash. It didn't. The remaining processes like evmd, occssd were still alive . Is there a tricky way to bring down the cluster like we kill pmon/smon process to bring down DB instance ?
[root@kpmsdbp1004 ~]# /crs/product/11.2.0/bin/crsctl stop crs -f
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'kpmsdbp1004'
CRS-2673: Attempting to stop 'ora.crsd' on 'kpmsdbp1004'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'kpmsdbp1004'
CRS-2679: Attempting to clean 'ora.bsblprd.db' on 'kpmsdbp1004'
CRS-2680: Clean of 'ora.bsblprd.db' on 'kpmsdbp1004' failed
CRS-2799: Failed to shut down resource 'ora.bsblprd.db' on 'kpmsdbp1004'
CRS-2794: Shutdown of Cluster Ready Services-managed resources on 'kpmsdbp1004' has failed
CRS-5022: Stop of resource "ora.crsd" failed: current state is "UNKNOWN"
CRS-2675: Stop of 'ora.crsd' on 'kpmsdbp1004' failed
CRS-2799: Failed to shut down resource 'ora.crsd' on 'kpmsdbp1004'
CRS-2795: Shutdown of Oracle High Availability Services-managed resources on 'kpmsdbp1004' has failed
CRS-4687: Shutdown command has completed with errors.
CRS-4000: Command Stop failed, or completed with errors.
[root@kpmsdbp1004 ~]#
[root@kpmsdbp1004 ~]#
[root@kpmsdbp1004 ~]#
[root@kpmsdbp1004 ~]# ps -ef | grep d.bin
root 27441 25147 0 16:27 pts/1 00:00:00 grep d.bin
root 35432 1 0 11:49 ? 00:01:43 /crs/product/11.2.0/bin/ohasd.bin reboot
grid 35566 1 0 11:49 ? 00:00:01 /crs/product/11.2.0/bin/mdnsd.bin
grid 35579 1 0 11:49 ? 00:00:12 /crs/product/11.2.0/bin/gpnpd.bin
grid 35594 1 0 11:49 ? 00:01:44 /crs/product/11.2.0/bin/gipcd.bin
root 35612 1 8 11:49 ? 00:24:51 /crs/product/11.2.0/bin/osysmond.bin
grid 35664 1 1 11:49 ? 00:03:08 /crs/product/11.2.0/bin/ocssd.bin
root 35834 1 0 11:50 ? 00:00:56 /crs/product/11.2.0/bin/octssd.bin reboot
grid 35868 1 0 11:50 ? 00:00:54 /crs/product/11.2.0/bin/evmd.bin
root 36218 1 0 11:50 ? 00:02:28 /crs/product/11.2.0/bin/crsd.bin reboot
[root@kpmsdbp1004 ~]#
[root@kpmsdbp1004 ~]# kill -9 36218
[root@kpmsdbp1004 ~]#
Killing crsd didn't help. All the remaining processes (evmd, occssd,...) still stayed alive in a hung mode.
[root@kpmsdbp1004 ~]# ps -ef | grep d.bin
root 27539 1 3 16:27 ? 00:00:00 /crs/product/11.2.0/bin/crsd.bin reboot
root 27572 25147 0 16:27 pts/1 00:00:00 grep d.bin
root 35432 1 0 11:49 ? 00:01:43 /crs/product/11.2.0/bin/ohasd.bin reboot
grid 35566 1 0 11:49 ? 00:00:01 /crs/product/11.2.0/bin/mdnsd.bin
grid 35579 1 0 11:49 ? 00:00:12 /crs/product/11.2.0/bin/gpnpd.bin
grid 35594 1 0 11:49 ? 00:01:44 /crs/product/11.2.0/bin/gipcd.bin
root 35612 1 8 11:49 ? 00:24:53 /crs/product/11.2.0/bin/osysmond.bin
grid 35664 1 1 11:49 ? 00:03:08 /crs/product/11.2.0/bin/ocssd.bin
root 35834 1 0 11:50 ? 00:00:56 /crs/product/11.2.0/bin/octssd.bin reboot
grid 35868 1 0 11:50 ? 00:00:54 /crs/product/11.2.0/bin/evmd.bin
[root@kpmsdbp1004 ~]#
[root@kpmsdbp1004 ~]#
[root@kpmsdbp1004 ~]# ps -ef | grep d.bin
root 27539 1 3 16:27 ? 00:00:00 /crs/product/11.2.0/bin/crsd.bin reboot
root 27614 25147 0 16:27 pts/1 00:00:00 grep d.bin
root 35432 1 0 11:49 ? 00:01:43 /crs/product/11.2.0/bin/ohasd.bin reboot
grid 35566 1 0 11:49 ? 00:00:01 /crs/product/11.2.0/bin/mdnsd.bin
grid 35579 1 0 11:49 ? 00:00:12 /crs/product/11.2.0/bin/gpnpd.bin
grid 35594 1 0 11:49 ? 00:01:44 /crs/product/11.2.0/bin/gipcd.bin
root 35612 1 8 11:49 ? 00:24:54 /crs/product/11.2.0/bin/osysmond.bin
grid 35664 1 1 11:49 ? 00:03:08 /crs/product/11.2.0/bin/ocssd.bin
root 35834 1 0 11:50 ? 00:00:56 /crs/product/11.2.0/bin/octssd.bin reboot
grid 35868 1 0 11:50 ? 00:00:54 /crs/product/11.2.0/bin/evmd.bin
[root@kpmsdbp1004 ~]#