I am noticing some new behaviour from Oracle RAC 11.2.0.4. We just upgraded in January and now I am going through April Critical Patch Updates process. This is the normal sequence of steps we do one node at a time.
1) Stop instances on node we are updating via "srvctl stop home" -o dbhome
2) Perform yum update
3) Reboot
4) Wait for CRS stack to startup. Note: because we stopped the database instances pre-reboot CRS will leave them stopped when the node comes back up. This is perfect as we now have all updated Linux RPMs and Kernel and are ready to patch/relink against all this new code using opatch auto.
Under 11.2.0.4, the db instances stay down, but something is still trying to spawn a handful of server processes in a futile attempt to connect to database that are not up and won't be coming up.
Here is some sample PS output. You can see a couple server processes for DBA1 and DBB1 even though these databases are stopped. These processes will cause opatch auto to fail when checking the database home for open files. The quick work-around is to just kill these processes, but I'd rather know where they are coming from and stop them from the source.
Pct CPU PID PPID Virt KB Pct Mem Started Time Elapsed Time CPU Time User Process
------- ----- ----- ------- ------- ------------ -------------- ------------ -------- --------------------
0.0 5004 1 1337804 0.7 16:12 18:12 00:00:00 oracle oracle+ASM1 (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))
0.0 5018 1 1342764 0.9 16:12 18:11 00:00:00 oracle oracle+ASM1_ocr (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))
0.0 5020 1 1337428 0.5 16:12 18:11 00:00:00 oracle asm_asmb_+ASM1
0.0 5022 1 1337840 0.6 16:12 18:10 00:00:00 oracle oracle+ASM1_asmb_+asm1 (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))
0.0 5071 4622 91860 0.4 16:12 18:04 00:00:00 oracle /opt/grid/product/11.2.0/grid2/bin/evmlogger.bin -o /opt/grid/product/11.2.0/grid2/evm/log/evmlogger.info -l /opt/grid/product/11.2.0/grid2/evm/log/evmlogger.log
0.2 5106 1 652536 1.6 16:12 18:02 00:00:02 oracle /opt/grid/product/11.2.0/grid2/bin/oraagent.bin
0.0 5194 1 229812 0.4 16:12 18:02 00:00:00 oracle oracleDBB1 (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))
0.0 5178 1 1337784 0.7 16:12 18:02 00:00:00 oracle oracle+ASM1 (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))
0.0 5182 1 1337272 0.6 16:12 18:02 00:00:00 oracle oracle+ASM1 (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))
0.0 5222 1 229812 0.4 16:12 17:59 00:00:00 oracle oracleDBA1 (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))
0.0 5214 1 229812 0.4 16:12 17:59 00:00:00 oracle oracleDBB1 (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))
0.0 5218 1 229812 0.4 16:12 17:59 00:00:00 oracle oracleDBB1 (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))
0.0 5212 1 1337272 0.6 16:12 17:59 00:00:00 oracle oracle+ASM1 (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))
0.0 5239 5238 69324 0.1 16:12 38:14 00:00:02 oracle /opt/grid/product/11.2.0/grid2/opmn/bin/ons -d
0.0 5238 1 62780 0.0 16:12 38:14 00:00:00 oracle /opt/grid/product/11.2.0/grid2/opmn/bin/ons -d
0.0 5281 1 85212 0.5 16:12 38:10 00:00:00 oracle /opt/grid/product/11.2.0/grid2/bin/tnslsnr LISTENER_SCAN1 -inherit
0.0 5268 1 85052 0.4 16:12 38:10 00:00:00 oracle /opt/grid/product/11.2.0/grid2/bin/tnslsnr LISTENER -inherit