We're running into issues trying to shutdown ASM on a 2 node 11.2.0.3 RAC cluster in one of our environments. This is on RHEL 5, kernel 2.6.18-274.17.1.el5
when issuing crsctl stop crs we get the following output
$ sudo /u01/app/11.2.0.3/grid/bin/crsctl stop has
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'opbld10'
CRS-2673: Attempting to stop 'ora.crsd' on 'opbld10'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'opbld10'
.....
CRS-2673: Attempting to stop 'ora.asm' on 'opbld10'
CRS-2677: Stop of 'ora.asm' on 'opbld10' succeeded
CRS-5014: Agent "/u01/app/11.2.0.3/grid/bin/orarootagent.bin" timed out starting process "/u01/app/11.2.0.3/grid/bin/acfsload" for action "stop": details at "(:CLSN00009:)" in "/u01/app/11.2.0.3/grid/log/opbld10/agent/crsd/orarootagent_root/orarootagent_root.log"
CRS-5017: The resource action "ora.drivers.acfs stop" encountered the following error:
(:CLSN00009:)Utils:execCmd aborted. For details refer to "(:CLSN00108:)" in "/u01/app/11.2.0.3/grid/log/opbld10/agent/crsd/orarootagent_root/orarootagent_root.log".
CRS-2675: Stop of 'ora.drivers.acfs' on 'opbld10' failed
CRS-2679: Attempting to clean 'ora.drivers.acfs' on 'opbld10'
CRS-5014: Agent "/u01/app/11.2.0.3/grid/bin/orarootagent.bin" timed out starting process "/u01/app/11.2.0.3/grid/bin/acfsload" for action "clean": details at "(:CLSN00009:)" in "/u01/app/11.2.0.3/grid/log/opbld10/agent/crsd/orarootagent_root/orarootagent_root.log"
CRS-5017: The resource action "ora.drivers.acfs clean" encountered the following error:
(:CLSN00009:)Utils:execCmd aborted. For details refer to "(:CLSN00106:)" in "/u01/app/11.2.0.3/grid/log/opbld10/agent/crsd/orarootagent_root/orarootagent_root.log".
CRS-2678: 'ora.drivers.acfs' on 'opbld10' has experienced an unrecoverable failure
CRS-2799: Failed to shut down resource 'ora.drivers.acfs' on 'opbld10'
CRS-2794: Shutdown of Cluster Ready Services-managed resources on 'opbld10' has failed
CRS-2675: Stop of 'ora.crsd' on 'opbld10' failed
CRS-2799: Failed to shut down resource 'ora.crsd' on 'opbld10'
CRS-2795: Shutdown of Oracle High Availability Services-managed resources on 'opbld10' has failed
CRS-4687: Shutdown command has completed with errors.
CRS-4000: Command Stop failed, or completed with errors.
Something appears to prevent the acfs driver from unloading. After some searching I found an old SR for this environment where acfs was not auto loading. The solution was a previous DBA adding ora.drivers.acfs as a weak start dependency of ora.asm.
The steps performed were as follows:
==============
1. On each instance of the cluster, add ora.drivers.acfs as a weak dependency
of ora.asm reaource.
As grid - crsctl modify resource ora.asm -attr
"START_DEPENDENCIES='weak(ora.LISTENER.lsnr,ora.drivers.acfs)'"
2. As Grid - [grid@opbld10 bin]$ /u01/app/11.2.0/grid/bin/crsctl add type
ora.registry.acfs.type -basetype ora.local_resource.type -file
/u01/app/11.2.0/grid/crs/template/registry.acfs.type
3.As Root - [root@opbld10 bin]$ /u01/app/11.2.0/grid/bin/crsctl add resource
ora.registry.acfs -attr ACL=\'owner:root:rwx,pgrp:oinstall:r-x,other::r--\'
-type ora.registry.acfs.type -f
4. As Grid - [grid@opbld10 bin]$ /u01/app/11.2.0/grid/bin/crsctl add type
ora.drivers.acfs.type -basetype ora.local_resource.type -file
/u01/app/11.2.0/grid/crs/template/drivers.acfs.type
5. As Root - [root@opbld10 bin]$ /u01/app/11.2.0/grid/bin/crsctl add resource
ora.drivers.acfs -attr ACL=\'owner:root:rwx,pgrp:oinstall:r-x,other::r--\'
-type ora.drivers.acfs.type -f
6. Reboot the nodes one at a time.
7. ACFS driver should load and acfs mount come up automatically upon reboot
without manually starting them up.
==============
I have verified this to still be in place.
$ /u01/app/11.2.0.3/grid/bin/crsctl status resource ora.asm -p
NAME=ora.asm
.....
START_DEPENDENCIES=weak(ora.LISTENER.lsnr,ora.drivers.acfs)
When comparing with other environments which do not exhibit this behavior I find the dependency does not exist.
$ crsctl status resource ora.asm -p
NAME=ora.asm
.....
START_DEPENDENCIES=weak(ora.LISTENER.lsnr)
If ACFS relies on ASM for it's actual storage, it seems strange to me that ACFS is listed as a dependency for ASM.
Performing more research this morning I found some oracle documentation stating the dependency should exist, however this appears to be in contrast to our working environments.
ACFS Advanced Topics: http://docs.oracle.com/cd/E14072_01/server.112/e10500/asmfs_extra.htm#CACECCDH]
Should this dependency exist? We do currently have an SR open with oracle, but have yet to find resolution. I'm trying to perform as much work on my own as I can to speed the process along. Any insight you can provide would be appreciated.
Edited by: rjanuary on Feb 9, 2012 8:25 AM - Added OS and kernel info.