Hi,
We had a network outage in the morning at 7.22 AM. Both the DB's were connected to Virtual chassis switches which went into a reboot due to bug issue. Hence we noticed the connections switched from prddb1 to proddb2. After switch issue was resolved only only proddb2 came online and proddb1 was not.
After manual restart of proddb1 everything seems to be fine. Both nodes are online and servicing all the incoming requests.
Could you please help me in RCA !!!
Appreciate your support!!
Alertlog proddb1
2015-06-03 07:23:16.193
[cssd(6907)]CRS-1610:node proddb2 (2) at 90% heartbeat fatal, eviction in 0.008 seconds
2015-06-03 07:23:16.743
[cssd(6907)]CRS-1607:CSSD evicting node proddb2. Details in /opt/oracle/product/10.2.0/crs/log/proddb1/cssd/ocssd.log.
2015-06-03 07:23:16.839
[cssd(6907)]CRS-1601:CSSD Reconfiguration complete. Active nodes are proddb1 .
2015-06-03 07:25:06.829
[crsd(6351)]CRS-1203:Failover failed for the CRS resource ora.proddb1.vip. Details in /opt/oracle/product/10.2.0/crs/log/proddb1/crsd/crsd.log.
2015-06-03 07:25:06.917
[crsd(6351)]CRS-1203:Failover failed for the CRS resource ora.proddb2.vip. Details in /opt/oracle/product/10.2.0/crs/log/proddb1/crsd/crsd.log.
2015-06-03 07:25:06.918
[crsd(6351)]CRS-1203:Failover failed for the CRS resource ora.proddb1.gsd. Details in /opt/oracle/product/10.2.0/crs/log/proddb1/crsd/crsd.log.
2015-06-03 07:25:07.005
[crsd(6351)]CRS-1203:Failover failed for the CRS resource ora.proddb2.gsd. Details in /opt/oracle/product/10.2.0/crs/log/proddb1/crsd/crsd.log.
2015-06-03 07:25:07.006
[crsd(6351)]CRS-1203:Failover failed for the CRS resource ora.proddb1.ons. Details in /opt/oracle/product/10.2.0/crs/log/proddb1/crsd/crsd.log.
2015-06-03 07:25:07.093
[crsd(6351)]CRS-1203:Failover failed for the CRS resource ora.proddb2.ons. Details in /opt/oracle/product/10.2.0/crs/log/proddb1/crsd/crsd.log.
2015-06-03 07:25:07.093
[crsd(6351)]CRS-1203:Failover failed for the CRS resource ora.proddb1.LISTENER_PRODDB1.lsnr. Details in /opt/oracle/product/10.2.0/crs/log/proddb1/crsd/crsd.log.
2015-06-03 07:25:07.194
[crsd(6351)]CRS-1203:Failover failed for the CRS resource ora.proddb2.LISTENER_PRODDB2.lsnr. Details in /opt/oracle/product/10.2.0/crs/log/proddb1/crsd/crsd.log.
2015-06-03 07:25:07.195
[crsd(6351)]CRS-1203:Failover failed for the CRS resource ora.proddb1.ASM1.asm. Details in /opt/oracle/product/10.2.0/crs/log/proddb1/crsd/crsd.log.
2015-06-03 07:25:07.282
[crsd(6351)]CRS-1203:Failover failed for the CRS resource ora.proddb2.ASM2.asm. Details in /opt/oracle/product/10.2.0/crs/log/proddb1/crsd/crsd.log.
2015-06-03 07:25:07.283
[crsd(6351)]CRS-1203:Failover failed for the CRS resource ora.orcl.db. Details in /opt/oracle/product/10.2.0/crs/log/proddb1/crsd/crsd.log.
2015-06-03 07:25:07.284
[crsd(6351)]CRS-1203:Failover failed for the CRS resource ora.orcl.orcl1.inst. Details in /opt/oracle/product/10.2.0/crs/log/proddb1/crsd/crsd.log.
2015-06-03 07:25:07.362
[crsd(6351)]CRS-1203:Failover failed for the CRS resource ora.orcl.orcl2.inst. Details in /opt/oracle/product/10.2.0/crs/log/proddb1/crsd/crsd.log.
2015-06-03 07:25:11.250
[crsd(6351)]CRS-1204:Recovering CRS resources for node proddb2.
2015-06-03 07:29:54.319
[cssd(6907)]CRS-1601:CSSD Reconfiguration complete. Active nodes are proddb1 proddb2 .
Alertlog Proddb2
2015-06-03 07:23:16.340
[cssd(6377)]CRS-1610:node proddb1 (1) at 90% heartbeat fatal, eviction in 0.022 seconds
2015-06-03 07:23:17.151
[cssd(6377)]CRS-1607:CSSD evicting node proddb1. Details in /opt/oracle/product/10.2.0/crs/log/proddb2/cssd/ocssd.log.
2015-06-03 07:23:17.169
[cssd(6377)]CRS-1601:CSSD Reconfiguration complete. Active nodes are proddb2 .
2015-06-03 07:29:53.664
[cssd(6377)]CRS-1605:CSSD voting file is online: /opt/oracle/VMXVOTING/votingdisk. Details in /opt/oracle/product/10.2.0/crs/log/proddb2/cssd/ocssd.log.
2015-06-03 07:29:54.295
[cssd(6377)]CRS-1601:CSSD Reconfiguration complete. Active nodes are proddb1 proddb2 .
2015-06-03 07:29:54.762
[crsd(5819)]CRS-1012:The OCR service started on node proddb2.
2015-06-03 07:29:55.403
[evmd(5790)]CRS-1401:EVMD started on node proddb2.
2015-06-03 07:29:57.262
[crsd(5819)]CRS-1201:CRSD started on node proddb2.
Crsd.log [proddb1]
##################
2015-06-03 07:22:25.473: [ CRSAPP][1546561856]0CheckResource error for ora.proddb1.vip error code = 1
2015-06-03 07:22:25.545: [ CRSRES][1546561856]0In stateChanged, ora.proddb1.vip target is ONLINE
2015-06-03 07:22:25.545: [ CRSRES][1546561856]0ora.proddb1.vip on proddb1 went OFFLINE unexpectedly
2015-06-03 07:22:25.558: [ CRSRES][1546561856]0StopResource: setting CLI values
2015-06-03 07:23:16.775: [ OCRSRV][1546561856]s_update_remote_cache_int: FAILED TO RCV ACK FROM node 2 retcode 7
2015-06-03 07:25:06.734: [ CRSCOMM][1496197440]0CLEANUP: Searching for connections to failed node proddb2
2015-06-03 07:25:06.734: [ CRSEVT][1496197440]0Processing member leave for proddb2, incarnation: 2
2015-06-03 07:25:06.734: [ CRSD][1496197440]0SM: recovery in process: 8
2015-06-03 07:25:06.734: [ CRSEVT][1496197440]0Do failover for: proddb2
2015-06-03 07:25:06.783: [ CRSRES][1546561856]0Attempting to stop `ora.proddb1.vip` on member `proddb1`
2015-06-03 07:25:07.375: [ CRSRES][1496197440]0Not failing resource ora.proddb1.vip because it was locked.
2015-06-03 07:25:07.375: [ CRSRES][1496197440]0X_RES_Unavailable : Resource ora.proddb1.vip is locked
(File: rti.cpp, line: 980
2015-06-03 07:25:07.376: [ CRSRES][1496197440]0 startup = 0
2015-06-03 07:25:07.406: [ CRSRES][1496197440]0 startup = 0
2015-06-03 07:25:07.432: [ CRSRES][1496197440]0 startup = 0
2015-06-03 07:25:07.468: [ CRSRES][1496197440]0 startup = 0
2015-06-03 07:25:07.499: [ CRSRES][1496197440]0 startup = 0
2015-06-03 07:25:07.528: [ CRSRES][1496197440]0 startup = 0
2015-06-03 07:25:07.670: [ CRSRES][1542359360]0startRunnable: setting CLI values
2015-06-03 07:25:07.700: [ CRSRES][1542359360]0Attempting to start `ora.proddb2.vip` on member `proddb1`
2015-06-03 07:25:09.864: [ CRSRES][1546561856]0Stop of `ora.proddb1.vip` on member `proddb1` succeeded.
2015-06-03 07:25:09.864: [ CRSRES][1546561856]0ora.proddb1.vip RESTART_COUNT=0 RESTART_ATTEMPTS=0
2015-06-03 07:25:09.920: [ CRSRES][1546561856]0ora.proddb1.vip failed on proddb1 relocating.
2015-06-03 07:25:10.003: [ CRSRES][1546561856]0Cannot relocate ora.proddb1.vipStopping dependents
2015-06-03 07:25:10.025: [ CRSRES][1546561856]0StopResource: setting CLI values
2015-06-03 07:25:10.120: [ CRSRES][1546561856]0Attempting to stop `ora.proddb1.LISTENER_PRODDB1.lsnr` on member `proddb1`
2015-06-03 07:25:10.624: [ CRSRES][1546561856]0Stop of `ora.proddb1.LISTENER_PRODDB1.lsnr` on member `proddb1` succeeded.
2015-06-03 07:25:10.629: [ CRSRES][1546561856]0StopResource: setting CLI values
2015-06-03 07:25:10.651: [ CRSRES][1546561856]0Attempting to stop `ora.orcl.orcl1.inst` on member `proddb1`
2015-06-03 07:25:11.214: [ CRSRES][1542359360]0Start of `ora.proddb2.vip` on member `proddb1` succeeded.
2015-06-03 07:25:11.250: [ CRSEVT][1496197440]0Post recovery done evmd event for: proddb2
2015-06-03 07:25:11.250: [ CRSD][1496197440]0SM: recoveryDone: 0
2015-06-03 07:25:11.251: [ CRSEVT][1496197440]0Processing RecoveryDone
2015-06-03 07:25:14.416: [ CRSRES][1546561856]0Stop of `ora.orcl.orcl1.inst` on member `proddb1` succeeded.
2015-06-03 07:25:14.417: [ CRSRES][1546561856]0StopResource: setting CLI values
2015-06-03 07:25:14.439: [ CRSRES][1546561856]0Attempting to stop `ora.proddb1.ASM1.asm` on member `proddb1`
2015-06-03 07:25:14.543: [ CRSAPP][1496197440]0CheckResource error for ora.orcl.db error code = 1
2015-06-03 07:25:14.545: [ CRSRES][1496197440]0In stateChanged, ora.orcl.db target is ONLINE
2015-06-03 07:25:14.545: [ CRSRES][1496197440]0ora.orcl.db on proddb1 went OFFLINE unexpectedly
2015-06-03 07:25:14.545: [ CRSRES][1496197440]0StopResource: setting CLI values
2015-06-03 07:25:14.571: [ CRSRES][1496197440]0Attempting to stop `ora.orcl.db` on member `proddb1`
2015-06-03 07:25:14.684: [ CRSRES][1496197440]0Stop of `ora.orcl.db` on member `proddb1` succeeded.
2015-06-03 07:25:14.684: [ CRSRES][1496197440]0ora.orcl.db RESTART_COUNT=0 RESTART_ATTEMPTS=0
2015-06-03 07:25:14.711: [ CRSRES][1496197440]0ora.orcl.db exceeded it's failure threshold. Stopping it and its dependents!
2015-06-03 07:25:14.739: [ CRSRES][1496197440]0ora.orcl.db target set to OFFLINE before stop action
2015-06-03 07:25:14.739: [ CRSRES][1496197440]0StopResource: setting CLI values
2015-06-03 07:25:14.791: [ CRSRES][1496197440]0Target set to OFFLINE for `ora.orcl.db`
2015-06-03 07:25:18.325: [ CRSRES][1546561856]0Stop of `ora.proddb1.ASM1.asm` on member `proddb1` succeeded.
2015-06-03 07:25:18.325: [ CRSRES][1546561856]0StopResource: setting CLI values
2015-06-03 07:29:57.721: [ CRSRES][1546561856]0StopResource: setting CLI values
Crsd.log [proddb2]
##################
2015-06-03 07:23:17.180: [ OCRMAS][1231067456]th_master:13: I AM THE NEW OCR MASTER at incar 2. Node Number 2
2015-06-03 07:29:42.658: [ default][4069296880][ENTER]0
Oracle Database 10g CRS Release 10.2.0.5.0 Production Copyright 1996, 2004, Oracle. All rights reserved
2015-06-03 07:29:42.678: [ default][4069296880]0CRS Daemon Starting
2015-06-03 07:29:42.678: [ CRSMAIN][4069296880]0Checking the OCR device
2015-06-03 07:29:42.680: [ CRSMAIN][4069296880]0Connecting to the CSS Daemon
2015-06-03 07:29:42.884: [ COMMCRS][1117661504]clsc_connect: (0x1fb5c120) no listener at (ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_proddb2_))
2015-06-03 07:29:42.884: [ CSSCLNT][4069296880]clsssInitNative: connect failed, rc 9
2015-06-03 07:29:42.884: [ CRSRTI][4069296880]0CSS is not ready. Received status 3 from CSS. Waiting for good status ..
2015-06-03 07:29:44.067: [ COMMCRS][1117661504]clsc_connect: (0x1fb5c150) no listener at (ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_proddb2_))
2015-06-03 07:29:44.067: [ CSSCLNT][4069296880]clsssInitNative: connect failed, rc 9
2015-06-03 07:29:44.067: [ CRSRTI][4069296880]0CSS is not ready. Received status 3 from CSS. Waiting for good status ..
2015-06-03 07:29:45.250: [ COMMCRS][1117661504]clsc_connect: (0x1fb5c150) no listener at (ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_proddb2_))
2015-06-03 07:29:45.250: [ CSSCLNT][4069296880]clsssInitNative: connect failed, rc 9
2015-06-03 07:29:45.250: [ CRSRTI][4069296880]0CSS is not ready. Received status 3 from CSS. Waiting for good status ..
2015-06-03 07:29:46.431: [ COMMCRS][1117661504]clsc_connect: (0x1fb5c150) no listener at (ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_proddb2_))
2015-06-03 07:29:46.431: [ CSSCLNT][4069296880]clsssInitNative: connect failed, rc 9
2015-06-03 07:29:46.431: [ CRSRTI][4069296880]0CSS is not ready. Received status 3 from CSS. Waiting for good status ..
2015-06-03 07:29:47.612: [ COMMCRS][1117661504]clsc_connect: (0x1fb5c150) no listener at (ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_proddb2_))
2015-06-03 07:29:47.612: [ CSSCLNT][4069296880]clsssInitNative: connect failed, rc 9
2015-06-03 07:29:47.612: [ CRSRTI][4069296880]0CSS is not ready. Received status 3 from CSS. Waiting for good status ..
2015-06-03 07:29:48.796: [ COMMCRS][1117661504]clsc_connect: (0x1fb5c150) no listener at (ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_proddb2_))
2015-06-03 07:29:48.796: [ CSSCLNT][4069296880]clsssInitNative: connect failed, rc 9
2015-06-03 07:29:48.796: [ CRSRTI][4069296880]0CSS is not ready. Received status 3 from CSS. Waiting for good status ..
2015-06-03 07:29:49.977: [ COMMCRS][1117661504]clsc_connect: (0x1fb5c150) no listener at (ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_proddb2_))
2015-06-03 07:29:49.977: [ CSSCLNT][4069296880]clsssInitNative: connect failed, rc 9
2015-06-03 07:29:49.977: [ CRSRTI][4069296880]0CSS is not ready. Received status 3 from CSS. Waiting for good status ..
2015-06-03 07:29:51.158: [ COMMCRS][1117661504]clsc_connect: (0x1fb5c150) no listener at (ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_proddb2_))
2015-06-03 07:29:51.158: [ CSSCLNT][4069296880]clsssInitNative: connect failed, rc 9
2015-06-03 07:29:51.158: [ CRSRTI][4069296880]0CSS is not ready. Received status 3 from CSS. Waiting for good status ..
2015-06-03 07:29:52.339: [ COMMCRS][1117661504]clsc_connect: (0x1fb5c150) no listener at (ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_proddb2_))
2015-06-03 07:29:52.339: [ CSSCLNT][4069296880]clsssInitNative: connect failed, rc 9
2015-06-03 07:29:52.340: [ CRSRTI][4069296880]0CSS is not ready. Received status 3 from CSS. Waiting for good status ..
2015-06-03 07:29:53.522: [ COMMCRS][1117661504]clsc_connect: (0x1fb5c150) no listener at (ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_proddb2_))
2015-06-03 07:29:53.522: [ CSSCLNT][4069296880]clsssInitNative: connect failed, rc 9
2015-06-03 07:29:53.522: [ CRSRTI][4069296880]0CSS is not ready. Received status 3 from CSS. Waiting for good status ..
2015-06-03 07:29:54.731: [ CLSVER][4069296880]0Active Version from OCR:10.2.0.5.0
2015-06-03 07:29:54.731: [ CLSVER][4069296880]0Active Version and Software Version are same
2015-06-03 07:29:54.731: [ CRSMAIN][4069296880]0Initializing OCR
2015-06-03 07:29:54.736: [ OCRRAW][4069296880]proprioo: for disk 0 (/opt/oracle/VMXOCR/ocrdisk), id match (1), my id set (850129439,1028247821) total id sets (1), 1st set (850129439,1028247821), 2nd set (0,0) my votes (2), total votes (2)
2015-06-03 07:29:54.766: [ CRSD][4069296880]0ENV Logging level for Module: allcomp 0
2015-06-03 07:29:54.776: [ CRSD][4069296880]0ENV Logging level for Module: default 0
2015-06-03 07:29:54.785: [ CRSD][4069296880]0ENV Logging level for Module: COMMCRS 0
2015-06-03 07:29:54.794: [ CRSD][4069296880]0ENV Logging level for Module: COMMNS 0
2015-06-03 07:29:54.814: [ CRSD][4069296880]0ENV Logging level for Module: CRSUI 0
2015-06-03 07:29:54.834: [ CRSD][4069296880]0ENV Logging level for Module: CRSCOMM 0
2015-06-03 07:29:54.844: [ CRSD][4069296880]0ENV Logging level for Module: CRSRTI 0
2015-06-03 07:29:54.853: [ CRSD][4069296880]0ENV Logging level for Module: CRSMAIN 0
2015-06-03 07:29:54.869: [ CRSD][4069296880]0ENV Logging level for Module: CRSPLACE 0
2015-06-03 07:29:54.878: [ CRSD][4069296880]0ENV Logging level for Module: CRSAPP 0
2015-06-03 07:29:54.888: [ CRSD][4069296880]0ENV Logging level for Module: CRSRES 0
2015-06-03 07:29:54.897: [ CRSD][4069296880]0ENV Logging level for Module: CRSOCR 0
2015-06-03 07:29:54.906: [ CRSD][4069296880]0ENV Logging level for Module: CRSTIMER 0
2015-06-03 07:29:54.915: [ CRSD][4069296880]0ENV Logging level for Module: CRSEVT 0
2015-06-03 07:29:54.925: [ CRSD][4069296880]0ENV Logging level for Module: CRSD 0
2015-06-03 07:29:54.934: [ CRSD][4069296880]0ENV Logging level for Module: CLUCLS 0
2015-06-03 07:29:54.943: [ CRSD][4069296880]0ENV Logging level for Module: CLSVER 0
2015-06-03 07:29:54.952: [ CRSD][4069296880]0ENV Logging level for Module: OCRRAW 0
2015-06-03 07:29:54.962: [ CRSD][4069296880]0ENV Logging level for Module: OCROSD 0
2015-06-03 07:29:54.971: [ CRSD][4069296880]0ENV Logging level for Module: CSSCLNT 0
2015-06-03 07:29:54.980: [ CRSD][4069296880]0ENV Logging level for Module: OCRAPI 0
2015-06-03 07:29:54.990: [ CRSD][4069296880]0ENV Logging level for Module: OCRUTL 0
2015-06-03 07:29:54.999: [ CRSD][4069296880]0ENV Logging level for Module: OCRMSG 0
2015-06-03 07:29:55.008: [ CRSD][4069296880]0ENV Logging level for Module: OCRCLI 0
2015-06-03 07:29:55.017: [ CRSD][4069296880]0ENV Logging level for Module: OCRCAC 0
2015-06-03 07:29:55.026: [ CRSD][4069296880]0ENV Logging level for Module: OCRSRV 0
2015-06-03 07:29:55.036: [ CRSD][4069296880]0ENV Logging level for Module: OCRMAS 0
2015-06-03 07:29:55.036: [ CRSMAIN][4069296880]0Filename is /opt/oracle/product/10.2.0/crs/crs/init/proddb2.pid
[ clsdmt][1421867328]Listening to (ADDRESS=(PROTOCOL=ipc)(KEY=proddb2DBG_CRSD))
2015-06-03 07:29:55.270: [ CRSMAIN][4069296880]0Using Authorizer location: /opt/oracle/product/10.2.0/crs/crs/auth/
2015-06-03 07:29:55.286: [ CRSMAIN][4069296880]0Initializing RTI
2015-06-03 07:29:55.390: [CRSTIMER][1432357184]0Timer Thread Starting.
2015-06-03 07:29:55.390: [ CRSRES][4069296880]0Parameter SECURITY = 1, running in USER Mode
2015-06-03 07:29:55.391: [ CRSMAIN][4069296880]0Initializing EVMMgr
2015-06-03 07:29:55.571: [ COMMCRS][1442847040]clsc_connect: (0x2aaaac01b870) no listener at (ADDRESS=(PROTOCOL=ipc)(KEY=SYSTEM.evm.acceptor.auth))
2015-06-03 07:29:56.883: [ CRSMAIN][4069296880]0CRSD locked during state recovery, please wait.
2015-06-03 07:29:57.232: [ CRSMAIN][4069296880]0CRSD recovered, unlocked.
2015-06-03 07:29:57.233: [ CRSMAIN][4069296880]0QS socket on: (ADDRESS=(PROTOCOL=ipc)(KEY=ora_crsqs))
2015-06-03 07:29:57.233: [ CRSMAIN][4069296880]0QS socket on: (ADDRESS=(PROTOCOL=ipc)(KEY=ora_crsqs))
2015-06-03 07:29:57.260: [ CRSMAIN][4069296880]0CRSD UI socket on: (ADDRESS=(PROTOCOL=ipc)(KEY=CRSD_UI_SOCKET))
2015-06-03 07:29:57.262: [ CRSMAIN][4069296880]0E2E socket on: (ADDRESS=(PROTOCOL=tcp)(HOST=proddb2-priv)(PORT=49896))
2015-06-03 07:29:57.262: [ CRSMAIN][4069296880]0Starting Threads
2015-06-03 07:29:57.262: [ CRSMAIN][4069296880]0CRS Daemon Started.
2015-06-03 07:29:57.262: [ CRSMAIN][1486907712]0Starting runCommandServer for (UI = 1, E2E = 0). 0
2015-06-03 07:29:57.262: [ CRSMAIN][1489008960]0Starting runCommandServer for (UI = 1, E2E = 0). 1
2015-06-03 07:29:57.270: [ CRSRES][4069296880]0 startup = 1
2015-06-03 07:29:57.312: [ CRSRES][4069296880]0 startup = 1
2015-06-03 07:29:57.342: [ CRSRES][4069296880]0 startup = 1
2015-06-03 07:29:57.371: [ CRSRES][4069296880]0 startup = 1
2015-06-03 07:29:57.374: [ CRSRES][4069296880]0 startup = 1
2015-06-03 07:29:57.403: [ CRSRES][4069296880]0 startup = 1
2015-06-03 07:29:57.406: [ CRSRES][4069296880]0 startup = 1
2015-06-03 07:29:57.435: [ CRSRES][4069296880]0 startup = 1
2015-06-03 07:29:57.487: [ CRSRES][4069296880]0 startup = 1
2015-06-03 07:29:57.490: [ CRSRES][4069296880]0 startup = 1
2015-06-03 07:29:57.567: [ CRSRES][1495312704]0StopResource: setting CLI values
2015-06-03 07:29:57.665: [ CRSRES][1497413952]0startRunnable: setting CLI values
2015-06-03 07:29:57.670: [ CRSRES][1497413952]0Attempting to start `ora.proddb1.vip` on member `proddb2`
2015-06-03 07:29:57.699: [ CRSRES][1495312704]0Attempting to stop `ora.proddb2.vip` on member `proddb1`
2015-06-03 07:29:58.000: [ CRSRES][1495312704]0Stop of `ora.proddb2.vip` on member `proddb1` succeeded.
2015-06-03 07:29:58.024: [ CRSRES][1495312704]0startRunnable: setting CLI values
2015-06-03 07:29:58.025: [ CRSRES][1495312704]0Attempting to start `ora.proddb2.vip` on member `proddb2`
2015-06-03 07:29:58.842: [ CRSRES][1497413952]0Start of `ora.proddb1.vip` on member `proddb2` succeeded.
2015-06-03 07:30:01.484: [ CRSRES][1495312704]0Start of `ora.proddb2.vip` on member `proddb2` succeeded.
2015-06-03 07:30:01.632: [ CRSRES][1495312704]0startRunnable: setting CLI values
2015-06-03 07:30:01.636: [ CRSRES][1495312704]0Attempting to start `ora.proddb2.ASM2.asm` on member `proddb2`
2015-06-03 07:30:01.859: [ CRSRES][1533086016]0startRunnable: setting CLI values
2015-06-03 07:30:01.888: [ CRSRES][1533086016]0Attempting to start `ora.proddb2.LISTENER_PRODDB2.lsnr` on member `proddb2`
2015-06-03 07:30:05.716: [ CRSRES][1533086016]0Start of `ora.proddb2.LISTENER_PRODDB2.lsnr` on member `proddb2` succeeded.
2015-06-03 07:30:21.197: [ CRSRES][1533086016]0startRunnable: setting CLI values
2015-06-03 07:30:21.223: [ CRSRES][1533086016]0Attempting to start `ora.proddb2.ons` on member `proddb2`
2015-06-03 07:30:22.780: [ CRSRES][1533086016]0Start of `ora.proddb2.ons` on member `proddb2` succeeded.
2015-06-03 07:30:22.888: [ CRSRES][1495312704]0Start of `ora.proddb2.ASM2.asm` on member `proddb2` succeeded.
2015-06-03 07:30:23.032: [ CRSRES][1495312704]0startRunnable: setting CLI values
2015-06-03 07:30:23.034: [ CRSRES][1495312704]0Attempting to start `ora.orcl.orcl2.inst` on member `proddb2`
2015-06-03 07:31:10.138: [ CRSRES][1495312704]0Start of `ora.orcl.orcl2.inst` on member `proddb2` succeeded.
2015-06-03 07:31:10.140: [ CRSRES][1495312704]0Skip online resource: ora.proddb2.LISTENER_PRODDB2.lsnr
2015-06-03 07:31:10.140: [ CRSRES][1530984768]0Skip online resource: ora.proddb2.ons
2015-06-03 07:31:13.891: [ CRSRES][1497413952]0startRunnable: setting CLI values
2015-06-03 07:31:13.892: [ CRSRES][1495312704]0startRunnable: setting CLI values
2015-06-03 07:31:13.971: [ CRSRES][1497413952]0Attempting to start `ora.proddb2.gsd` on member `proddb2`
2015-06-03 07:31:13.972: [ CRSRES][1495312704]0Attempting to start `ora.orcl.db` on member `proddb2`
2015-06-03 07:31:13.976: [ OCRUTL][1390397760]u_freem: mem passed is null
2015-06-03 07:31:14.502: [ CRSRES][1495312704]0Start of `ora.orcl.db` on member `proddb2` succeeded.
2015-06-03 07:31:14.762: [ CRSRES][1497413952]0Start of `ora.proddb2.gsd` on member `proddb2` succeeded.
2015-06-03 07:31:14.981: [ CRSRES][1495312704]0CRS-1002: Resource 'ora.orcl.db' is already running on member 'proddb2'