Hello
I have installed Oracle 19c RAC on Oracle Linux 7.8 (2 node RAC)
We use SAN Storage , Multipath Disks as Shared Storage
My problem is that sometimes ASM instance and CRS in the first node go down (and sometimes it starts again by itself )
Please If according to the following description you can help me , I'd appreciate it
and If you want more information from me I will welcome
I need to say for disks permission I used “Linux Udev” and the config is :
in file /etc/udev/rules.d/12-oracle-asm-permissions.rules:
ENV{DM_NAME}=="mpatha1", OWNER:="oracle", GROUP:="asmadmin", MODE:="660", SYMLINK+="ASM/oraasm-$env{DM_NAME}"
ENV{DM_NAME}=="mpathb1", OWNER:="oracle", GROUP:="asmadmin", MODE:="660", SYMLINK+="ASM/oraasm-$env{DM_NAME}"
ls –l /dev/ASM/
total 0
lrwxrwxrwx 1 root root 7 Aug 10 18:40 oraasm-mpatha1 -> ../dm-7
lrwxrwxrwx 1 root root 7 Aug 10 18:39 oraasm-mpathb1 -> ../dm-8
ls -l /dev/dm-*
brw-rw---- 1 root disk 252, 0 Aug 9 10:05 /dev/dm-0
brw-rw---- 1 root disk 252, 1 Aug 9 10:05 /dev/dm-1
brw-rw---- 1 root disk 252, 2 Aug 9 10:05 /dev/dm-2
brw-rw---- 1 root disk 252, 3 Aug 9 10:05 /dev/dm-3
brw-rw---- 1 root disk 252, 4 Aug 9 10:06 /dev/dm-4
brw-rw---- 1 root disk 252, 5 Aug 9 10:06 /dev/dm-5
brw-rw---- 1 root disk 252, 6 Aug 9 10:06 /dev/dm-6
brw-rw---- 1 oracle asmadmin 252, 7 Aug 10 18:41 /dev/dm-7
brw-rw---- 1 oracle asmadmin 252, 8 Aug 10 18:41 /dev/dm-8
brw-rw---- 1 root disk 252, 9 Aug 9 10:05 /dev/dm-9
When the problem occurred I ran these commands :
srvctl config database
PRCR-1119 : Failed to look up CRS resources of database type
PRCR-1115 : Failed to find entities of type resource that match filters (TYPE == ora.database.type) and contain attributes DB_UNIQUE_NAME,ORACLE_HOME,VERSION
CRS-0184 : Cannot communicate with the CRS daemon.
[oracle@nscdbnod1 ~]$ crsctl check cluster -all
**************************************************************
nscdbnod1:
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
nscdbnod2:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
[oracle@nscdbnod1 ~]$ crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
In the /Oracle/oracle_base/diag/crs/nscdbnod1/crs/trace/alert.log file I found these logs :
2021-08-10 15:17:29.428 [ORAAGENT(48246)]CRS-5818: Aborted command 'check' for resource 'ora.asm'. Details at (:CRSAGF00113:) {0:5:2} in /Oracle/oracle_base/diag/crs/nscdbnod1/crs/trace/crsd_oraagent_oracle.trc.
2021-08-10 15:17:46.431 [ORAAGENT(48246)]CRS-5818: Aborted command 'check' for resource 'ora.DATA.dg'. Details at (:CRSAGF00113:) {0:3:14} in /Oracle/oracle_base/diag/crs/nscdbnod1/crs/trace/crsd_oraagent_oracle.trc.
2021-08-10 15:17:56.405 [ORAAGENT(48246)]CRS-5818: Aborted command 'check' for resource 'ora.ASMNET1LSNR_ASM.lsnr'. Details at (:CRSAGF00113:) {0:5:2} in /Oracle/oracle_base/diag/crs/nscdbnod1/crs/trace/crsd_oraagent_oracle.trc.
2021-08-10 15:18:01.428 [ORAAGENT(48246)]CRS-5818: Aborted command 'check' for resource 'ora.asm'. Details at (:CRSAGF00113:) {0:5:2} in /Oracle/oracle_base/diag/crs/nscdbnod1/crs/trace/crsd_oraagent_oracle.trc.
2021-08-10 15:18:06.631 [ORAROOTAGENT(27356)]CRS-5822: Agent '/Oracle/oracle_product/19c/grid/bin/orarootagent_root' disconnected from server. Details at (:CRSAGF00117:) {0:2:4} in /Oracle/oracle_base/diag/crs/nscdbnod1/crs/trace/crsd_orarootagent_root.trc.
2021-08-10 15:18:06.632 [ORAAGENT(48246)]CRS-5822: Agent '/Oracle/oracle_product/19c/grid/bin/oraagent_oracle' disconnected from server. Details at (:CRSAGF00117:) {0:5:19} in /Oracle/oracle_base/diag/crs/nscdbnod1/crs/trace/crsd_oraagent_oracle.trc.
**********************************************************************************
And in /Oracle/oracle_base/diag/crs/nscdbnod1/crs/trace/crsd_oraagent_oracle.trc file around the problem time I found :
2021-08-10 15:17:56.405 :CLSDYNAM:3799987968: [ora.ASMNET1LSNR_ASM.lsnr]{0:5:2} [check] abort command: check
2021-08-10 15:17:56.405 :CLSDYNAM:3799987968: [ora.ASMNET1LSNR_ASM.lsnr]{0:5:2} [check] tryActionLock {
2021-08-10 15:17:58.435 :CLSDYNAM:827778816: [ora.DATA.dg]{0:3:14} [check] DgpAgent::runCheck 210 sleep
2021-08-10 15:18:01.429 : AGENT:834082560: [ NONE] {0:5:2} {0:5:2} Created alert : (:CRSAGF00113:) : Aborting the command: check for resource: ora.asm 1 1
2021-08-10 15:18:01.429 :CLSDYNAM:834082560: [ ora.asm]{0:5:2} [check] (:CLSN00110:) clsn_agent::abort {
2021-08-10 15:18:01.429 :CLSDYNAM:834082560: [ ora.asm]{0:5:2} [check] abort {
2021-08-10 15:18:01.429 :CLSDYNAM:834082560: [ ora.asm]{0:5:2} [check] Agent::doStateDump Default Agent Dump
2021-08-10 15:18:01.429 :CLSDYNAM:834082560: [ ora.asm]{0:5:2} [check] Agent::doStateDump last call info:
2021-08-10 15:18:01.429 :CLSDYNAM:834082560: [ ora.asm]{0:5:2} [check] Time:08/10/2021 15:17:31.431 Tint:{0:5:2} action:104 resname:ora.asm lastCall:AsmAgent::gimhChecks 240 index[7]:0 state:3 stateStrs:
********************************************************************************
And in file /Oracle/oracle_base/diag/crs/nscdbnod1/crs/trace/crsd_orarootagent_root.trc I found :
2021-08-10 15:18:06.631 : CRSCOMM:208418560: [ INFO] IpcC: IPC client connection 493 to member 0 has been removed
2021-08-10 15:18:06.631 :CLSFRAME:208418560: [ INFO] Removing IPC Member:{Relative|Node:0|Process:0|Type:1}
2021-08-10 15:18:06.631 :CLSFRAME:208418560: [ INFO] Disconnected from CRSD:nscdbnod1 process: {Relative|Node:0|Process:0|Type:1}
2021-08-10 15:18:06.631 : AGENT:4156544768: [ NONE] {0:2:4} {0:2:4} Created alert : (:CRSAGF00117:) : Disconnected from server, Agent is shutting down.
2021-08-10 15:18:06.631 : AGENT:4156544768: [ INFO] {0:2:4} Agfw calling user exitCB, will exit on return
2021-08-10 15:18:06.631 : AGENT:4156544768: [ INFO] {0:2:4} returned from user exitCB, exiting
2021-08-10 15:18:06.631 : AGENT:4156544768: [ INFO] {0:2:4} Agent is exiting with exit code: 1
Trace file /Oracle/oracle_base/diag/crs/nscdbnod1/crs/trace/crsd_orarootagent_root.trc
Oracle Database 19c Clusterware Release 19.0.0.0.0 - Production
Version 19.3.0.0.0 Copyright 1996, 2019 Oracle. All rights reserved.
default:3339399488: 1: clskec:has:CLSU:910 4 args[clsdAdr_CLSK_err][mod=clsdadr.c][loc=(:CLSD00302:)][msg=2021-08-10 15:18:11.819 (:CLSD00302:) Trace file size and number of segments fetched from environment variable: ORA_DAEMON_TRACE_FILE_OPTIONS filesize=26214400,numsegments=10 Detected in function clsdAdrGetEnvVar_TFO at line number 6819. ]
CLSB:3339399488: [ INFO] Argument count (argc) for this daemon is 1
CLSB:3339399488: [ INFO] Argument 0 is: /Oracle/oracle_product/19c/grid/bin/orarootagent.bin
2021-08-10 15:18:12.072 : AGENT:3339399488: [ INFO] Logging level for Module: clsdadr 2
2021-08-10 15:18:12.072 : AGENT:3339399488: [ INFO] Logging level for Module: clsdstat 2
Thank you