Hi All,
Setup Details:
Oracle 12.1.0.2 Two node RAC with Dataguard.
Node-1 is working fine currently
Before 15 days node-2 was evicted due to network time out and it was rebooted automatically. But after reboot instance didnt started in node2.
Today I have again tried to reboot node twice and my observation are :
Node-2 has tried to start cssd service for more than 10 mins then it has terminated service by it self.
Below messages are reported in crs alert logs:
2017-08-26 17:15:39.229 [OCSSD(15937)]CRS-8500: Oracle Clusterware OCSSD process is starting with operating system process ID 15937
2017-08-26 17:15:40.280 [OCSSD(15937)]CRS-1713: CSSD daemon is started in hub mode
2017-08-26 17:15:55.935 [OCSSD(15937)]CRS-1707: Lease acquisition for node racdbnod2 number 2 completed
2017-08-26 17:15:57.044 [OCSSD(15937)]CRS-1605: CSSD voting file is online: /dev/mapper/mpathl; details in /opt/u01/app/oracle/diag/crs/racdbnod2/crs/trace/ocssd.trc.
2017-08-26 17:15:57.048 [OCSSD(15937)]CRS-1605: CSSD voting file is online: /dev/mapper/mpathi; details in /opt/u01/app/oracle/diag/crs/racdbnod2/crs/trace/ocssd.trc.
2017-08-26 17:15:57.052 [OCSSD(15937)]CRS-1605: CSSD voting file is online: /dev/mapper/mpathr; details in /opt/u01/app/oracle/diag/crs/racdbnod2/crs/trace/ocssd.trc.
2017-08-26 17:25:38.283 [CSSDAGENT(15926)]CRS-5818: Aborted command 'start' for resource 'ora.cssd'. Details at (:CRSAGF00113:) {0:9:3} in /opt/u01/app/oracle/diag/crs/racdbnod2/crs/trace/ohasd_cssdagent_root.trc.
2017-08-26 17:25:38.284 [OCSSD(15937)]CRS-1656: The CSS daemon is terminating due to a fatal error; Details at (:CSSSC00012:) in /opt/u01/app/oracle/diag/crs/racdbnod2/crs/trace/ocssd.trc
2017-08-26 17:25:38.284 [OCSSD(15937)]CRS-1603: CSSD on node racdbnod2 shutdown by user.
Sat Aug 26 17:25:43 2017
Errors in file /opt/u01/app/oracle/diag/crs/racdbnod2/crs/trace/ocssd.trc (incident=761):
CRS-8503 [] [] [] [] [] [] [] [] [] [] [] []
Incident details in: /opt/u01/app/oracle/diag/crs/racdbnod2/crs/incident/incdir_761/ocssd_i761.trc
2017-08-26 17:25:43.447 [OCSSD(15937)]CRS-8503: Oracle Clusterware OCSSD process with operating system process ID 15937 experienced fatal signal or exception code 6
Sweep [inc][761]: completed
2017-08-26 17:25:43.703 [CSSDMONITOR(31194)]CRS-8500: Oracle Clusterware CSSDMONITOR process is starting with operating system process ID 31194
2017-08-26 17:25:49.286 [OHASD(15281)]CRS-2757: Command 'Start' timed out waiting for response from the resource 'ora.cssd'. Details at (:CRSPE00163:) {0:9:3} in /opt/u01/app/oracle/diag/crs/racdbnod2/crs/trace/ohasd.trc.
2017-08-26 17:25:49.389 [CSSDAGENT(31672)]CRS-8500: Oracle Clusterware CSSDAGENT process is starting with operating system process ID 31672
2017-08-26 17:49:01.840 [CLSECHO(46397)]CRS-10001: 26-Aug-17 17:49 AFD-9204: false
Please provide your expert comments on this issue.
Thanks in advance.