Hello!
I have a two-node RAC with Oracle 12.2 and Oracle Linux 7.3
My RAC was working for one year but for now it has stop working.
I can not start it with 'crsctl start crs': the command is successed, but the ora.evmd service remains in intermediate state:
[root@dbperf11 mapper]# /oracle-grid/product/grid/12.2.0.1/bin/crsctl status res -t -init
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.asm
1 ONLINE OFFLINE STABLE
ora.cluster_interconnect.haip
1 ONLINE OFFLINE STABLE
ora.crf
1 ONLINE OFFLINE STABLE
ora.crsd
1 ONLINE OFFLINE STABLE
ora.cssd
1 ONLINE OFFLINE STABLE
ora.cssdmonitor
1 ONLINE ONLINE dbperf11 STABLE
ora.ctssd
1 ONLINE OFFLINE STABLE
ora.diskmon
1 OFFLINE OFFLINE STABLE
ora.driver.afd
1 ONLINE ONLINE dbperf11 STABLE
ora.drivers.acfs
1 ONLINE ONLINE dbperf11 STABLE
ora.evmd
1 ONLINE INTERMEDIATE dbperf11 STABLE
ora.gipcd
1 ONLINE ONLINE dbperf11 STABLE
ora.gpnpd
1 ONLINE ONLINE dbperf11 STABLE
ora.mdnsd
1 ONLINE ONLINE dbperf11 STABLE
ora.storage
1 ONLINE OFFLINE STABLE
--------------------------------------------------------------------------------
There are no any errors in crs/trace/alert.log.
After some discovering I found that the issue is with ora.cssd resource: it cann't start. I can see in $GI_HOME/log/$NODE/cssd/ocssd.log:
| 2018-06-05 11:45:47.186: [ | CSSD][349184064]clssscGetParameterProfile: buffer passed for parameter ASM discovery (3) is too short, required 23, passed 20 |
| 2018-06-05 11:45:47.186: [ | CSSD][349184064]clssnmReadDiscoveryProfile: voting file discovery string(/dev/mapper/ora*,AFD:*) |
| 2018-06-05 11:45:47.186: [ | CSSD][349184064]clssnkInit: NK generic layer initializing. |
| 2018-06-05 11:45:47.187: [ | CSSD][140302080]clssscthrdmain: Starting thread clssnmvDDiscThread |
| 2018-06-05 11:45:47.187: [ | GPNP][140302080]clsgpnp_Init: [at clsgpnp0.c:691] '/oracle-grid/product/grid/12.2.0.1' in effect as GPnP home base. |
| 2018-06-05 11:45:47.187: [ | GPNP][140302080]clsgpnp_Init: [at clsgpnp0.c:757] GPnP pid=62169, cli=clsuGpnpg GPNP comp tracelevel=1, depcomp tracelevel=0, tlsrc:init, apitl:0, tstenv:0, devenv:0, envopt:0, flags=2003 |
| 2018-06-05 11:45:47.190: [ | GPNP][140302080]clsgpnpkwf_initwfloc: [at clsgpnpkwf.c:403] Using FS Wallet Location : /oracle-grid/product/grid/12.2.0.1/gpnp/dbperf11/wallets/peer/ |
| 2018-06-05 11:45:47.190: [ | GPNP][140302080]clsgpnpkwf_initwfloc: [at clsgpnpkwf.c:415] Wallet readable. Path: /oracle-grid/product/grid/12.2.0.1/gpnp/dbperf11/wallets/peer/ |
[ default][140302080]Oracle Clusterware infrastructure error in OCSSD (OS PID 62169): Fatal signal 6 has occurred in program ocssd thread 140302080; nested signal count is 1
[ default][140302080]clsssc_sighandler_common: CSSD handling internal error CRS-8503 (signal 6)
[ default][140302080]clsssc_sighandler_common: Dumping CSSD state and exiting
Where can I found and how can I recover this issue?