ASM instance hung on 2 node RAC cluster
AnkyJan 30 2008 — edited May 11 2008Hi,
I have a 2 node 10g RAC cluster on AIX box using ASM having shared disk on IBM ESS.
A day back my unix administrator added 100G of space partitioned in 3 disk .
But when I logged in ASM instance , I found that my instance was hung (queries hung on the prompt) , so I tried to shutdown the instance on one node but it abort with the following error :
(I got the same message on the other node , i shutdown the database before trying for ASM instance and it was clearly shutdown)
10gdb ccman_avalon:/home/10gdb > srvctl stop asm -n ccman_avalon
PRKS-1012 : Failed to stop ASM instance "+ASM1" on node "ccman_avalon", [PRKS-1012 : Failed to stop ASM instance "+ASM1" on node "ccman_avalon", [ccman_avalon:ora.ccman_avalon.ASM1.asm:timeout: killed the spawned process
ccman_avalon:ora.ccman_avalon.ASM1.asm:timeout: killed the spawned process
CRS-0216: Could not stop resource 'ora.ccman_avalon.ASM1.asm'.]]
[PRKS-1012 : Failed to stop ASM instance "+ASM1" on node "ccman_avalon", [ccman_avalon:ora.ccman_avalon.ASM1.asm:timeout: killed the spawned process
ccman_avalon:ora.ccman_avalon.ASM1.asm:timeout: killed the spawned process
CRS-0216: Could not stop resource 'ora.ccman_avalon.ASM1.asm'.]]
The instance is down now. But I can see number of connection with the ASM instance .
10gdb ccman_avalon:/home/10gdb > ps -ef|grep ASM
10gdb 675992 561274 0 11:32:45 - 0:00 oracle+ASM1 (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))
10gdb 684118 1 0 29 Jan - 0:00 oracle+ASM1 (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))
Can you guide me how to trouble shoot the issue .
Thanks,
Ankur