Hello All,
My database details are as follows -
1) OEL - 6.5
2) Database - 12.1.0.1
Cluster is showing error. Ping (pub & priv ip's from one node to another is okay). Only problem is with cluster.
[grid@node1 ~]$ crsctl stat res -t
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4000: Command Status failed, or completed with errors.
[grid@node1 ~]$ crsctl stat res -t -init
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.asm
1 ONLINE OFFLINE Instance Shutdown,ST
ABLE
ora.cluster_interconnect.haip
1 ONLINE ONLINE node1 STABLE
ora.crf
1 ONLINE ONLINE node1 STABLE
ora.crsd
1 ONLINE OFFLINE STABLE
ora.cssd
1 ONLINE ONLINE node1 STABLE
ora.cssdmonitor
1 ONLINE ONLINE node1 STABLE
ora.ctssd
1 ONLINE ONLINE node1 OBSERVER,STABLE
ora.diskmon
1 OFFLINE OFFLINE STABLE
ora.drivers.acfs
1 ONLINE ONLINE node1 STABLE
ora.evmd
1 ONLINE ONLINE node1 STABLE
ora.gipcd
1 ONLINE ONLINE node1 STABLE
ora.gpnpd
1 ONLINE ONLINE node1 STABLE
ora.mdnsd
1 ONLINE ONLINE node1 STABLE
ora.storage
1 ONLINE OFFLINE STABLE
--------------------------------------------------------------------------------
[grid@node1 ~]$
Log file of Crsd is as belows. Please suggest how to resolve this error.
019-10-20 08:41:03.274023 :CLSDYNAM:215582464: [ ora.asm]{0:11:46} [clean] InstAgent::stop db/asm
2019-10-20 08:41:03.274035 :CLSDYNAM:215582464: [ ora.asm]{0:11:46} [clean] ConnectionPool::stopConnection
2019-10-20 08:41:03.274058 :CLSDYNAM:215582464: [ ora.asm]{0:11:46} [clean] ConnectionPool::removeConnection connection count 0
2019-10-20 08:41:03.274070 :CLSDYNAM:215582464: [ ora.asm]{0:11:46} [clean] ConnectionPool::removeConnection freed 0
2019-10-20 08:41:03.274082 :CLSDYNAM:215582464: [ ora.asm]{0:11:46} [clean] ConnectionPool::stopConnection sid +ASM1 status 1
2019-10-20 08:41:03.275866 :CLSDYNAM:215582464: [ ora.asm]{0:11:46} [clean] getResAttrib: attrib name USR_ORA_OPI value false len 5
2019-10-20 08:41:03.276080 :CLSDYNAM:215582464: [ ora.asm]{0:11:46} [clean] InstAgent::stop Container->stop, isShutdown = 0
2019-10-20 08:41:03.276266 :CLSDYNAM:215582464: [ ora.asm]{0:11:46} [clean] InstAgent::stop calling initCssData
2019-10-20 08:41:03.276311 :CLSDYNAM:215582464: [ ora.asm]{0:11:46} [clean] CssData Constructor groupName:HB+ASM cssRegistrationType:0 memno:-1
2019-10-20 08:41:03.276328 : USRTHRD:215582464: {0:11:46} CssData::initMutex &m_cssDataLock:01a92928
2019-10-20 08:41:03.287412 :CLSDYNAM:215582464: [ ora.asm]{0:11:46} [clean] CssGroup::regis registration of groupName:HB+ASM mbrid:-1 m_grpNum:-1 regType:0 succeeded
2019-10-20 08:41:03.287445 :CLSDYNAM:215582464: [ ora.asm]{0:11:46} [clean] CssData::Constructor regis succeeded groupName:HB+ASM memno:-1
2019-10-20 08:41:03.287460 :CLSDYNAM:215582464: [ ora.asm]{0:11:46} [clean] CssData::Constructor m_pCssGroup:01ac99c0 group HB+ASM
2019-10-20 08:41:03.287475 :CLSDYNAM:215582464: [ ora.asm]{0:11:46} [clean] InstAgent::init m_groupName:HB+ASM m_pCssData 01a90860
2019-10-20 08:41:03.287491 : USRTHRD:215582464: {0:11:46} CssData Destructor groupName:HB+ASM
2019-10-20 08:41:03.287552 : USRTHRD:215582464: {0:11:46} CssGroup::dereg default regType:0
2019-10-20 08:41:03.291234 : USRTHRD:215582464: {0:11:46} CssData::destroyMutex &m_cssDataLock:01a92928
2019-10-20 08:41:03.291261 : USRTHRD:215582464: {0:11:46} CssData::Destructor
2019-10-20 08:41:03.291288 :CLSDYNAM:215582464: [ ora.asm]{0:11:46} [clean] InstAgent::stop prev clsagfw_res_status 2 current clsagfw_res_status 2
2019-10-20 08:41:03.291304 :CLSDYNAM:215582464: [ ora.asm]{0:11:46} [clean] InstAgent::stop m_prevState:2 }
2019-10-20 08:41:03.291319 :CLSDYNAM:215582464: [ ora.asm]{0:11:46} [clean] clean }
2019-10-20 08:41:03.291397 :CLSDYNAM:215582464: [ ora.asm]{0:11:46} [clean] (:CLSN00106:) clsn_agent::clean }
2019-10-20 08:41:03.291417 : AGFW:215582464: {0:11:46} Command: clean for resource: ora.asm 1 1 completed with status: SUCCESS
2019-10-20 08:41:03.293572 : AGFW:233195264: {0:11:46} Agent sending reply for: RESOURCE_CLEAN[ora.asm 1 1] ID 4100:18755
2019-10-20 08:41:03.297221 :CLSDYNAM:213481216: [ ora.asm]{0:11:46} [check] InstAgent::checkState gimh oracleSid:+ASM1
2019-10-20 08:41:03.297244 :CLSDYNAM:213481216: [ ora.asm]{0:11:46} [check] Gimh::constructor ohome:/u01/app/12.1.0/grid sid:+ASM1
2019-10-20 08:41:03.297340 :CLSDYNAM:213481216: [ ora.asm]{0:11:46} [check] ConnectionPool::resetConnection s_statusOfConnectionMap 00f3aed8
2019-10-20 08:41:03.297363 :CLSDYNAM:213481216: [ ora.asm]{0:11:46} [check] ConnectionPool::resetConnection sid +ASM1 status 2
2019-10-20 08:41:03.297470 :CLSDYNAM:213481216: [ ora.asm]{0:11:46} [check] Gimh::check OH /u01/app/12.1.0/grid SID +ASM1
2019-10-20 08:41:03.297576 :CLSDYNAM:213481216: [ ora.asm]{0:11:46} [check] Gimh::check condition changes to (GIMH_NEXT_NUM) 0(Abnormal Termination),1(Instance Shutdown) exists
2019-10-20 08:41:03.297592 :CLSDYNAM:213481216: [ ora.asm]{0:11:46} [check] (:CLSN00006:)AsmAgent::gimhChecks failed gimh state 0
2019-10-20 08:41:03.309728 :CLSDYNAM:213481216: [ ora.asm]{0:11:46} [check] checkCrsStat 2 CLSCRS_STAT ret: 184
2019-10-20 08:41:03.309783 :CLSDYNAM:213481216: [ ora.asm]{0:11:46} [check] clsnUtils::error Exception type=2 string=
2019-10-20 08:41:03.309826 :CLSDYNAM:213481216: [ ora.asm]{0:11:46} [check] checkCrsStat 2 status:0 ret:184 usererror:184
2019-10-20 08:41:03.311117 :CLSDYNAM:213481216: [ ora.asm]{0:11:46} [check] AsmAgent::checkCbk: Exception UserErrorException
2019-10-20 08:41:03.311135 :CLSDYNAM:213481216: [ ora.asm]{0:11:46} [check]
2019-10-20 08:41:03.311219 : USRTHRD:213481216: {0:11:46} Gimh::destructor gimh_dest_query_ctx rc=0
2019-10-20 08:41:03.311248 : USRTHRD:213481216: {0:11:46} Gimh::destructor gimh_dest_inst_ctx rc=0
2019-10-20 08:41:03.311309 :CLSDYNAM:213481216: [ ora.asm]{0:11:46} [check] ConnectionPool::stopConnection
2019-10-20 08:41:03.311467 :CLSDYNAM:213481216: [ ora.asm]{0:11:46} [check] ConnectionPool::removeConnection connection count 0
2019-10-20 08:41:03.311477 :CLSDYNAM:213481216: [ ora.asm]{0:11:46} [check] ConnectionPool::removeConnection freed 0
2019-10-20 08:41:03.311485 :CLSDYNAM:213481216: [ ora.asm]{0:11:46} [check] ConnectionPool::stopConnection sid +ASM1 status 1
2019-10-20 08:41:03.311830 : AGFW:233195264: {0:11:46} ora.asm 1 1 state changed from: CLEANING to: PLANNED_OFFLINE
2019-10-20 08:41:03.311902 : AGFW:233195264: {0:11:46} Agent sending last reply for: RESOURCE_CLEAN[ora.asm 1 1] ID 4100:18755
2019-10-20 08:41:03.328083 : AGFW:233195264: {0:0:4140} Agent received the message: RESOURCE_START[ora.asm 1 1] ID 4098:18762
2019-10-20 08:41:03.328105 : AGFW:233195264: {0:0:4140} Preparing START command for: ora.asm 1 1
2019-10-20 08:41:03.328113 : AGFW:233195264: {0:0:4140} ora.asm 1 1 state changed from: PLANNED_OFFLINE to: STARTING
2019-10-20 08:41:03.328951 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] (:CLSN00107:) clsn_agent::start {
2019-10-20 08:41:03.333151 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] InstAgent::start {
2019-10-20 08:41:03.333199 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] ConnectionPool::stopConnection
2019-10-20 08:41:03.333218 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] ConnectionPool::removeConnection connection count 0
2019-10-20 08:41:03.333647 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] ConnectionPool::removeConnection freed 0
2019-10-20 08:41:03.333745 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] ConnectionPool::stopConnection sid +ASM1 status 1
2019-10-20 08:41:03.333761 : USRTHRD:230291200: {0:0:4140} ConnectionPool::~ConnectionPool m_oracleHome:/u01/app/12.1.0/grid, m_oracleSid:+ASM1, m_usrOraEnv:
2019-10-20 08:41:03.333803 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] AsmAgent::refresh
2019-10-20 08:41:03.333817 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] AsmAgent::refresh ORACLE_HOME = /u01/app/12.1.0/grid
2019-10-20 08:41:03.333826 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] clsnUtils::cmdIdIsStart CmdId:257
2019-10-20 08:41:03.333851 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] clsnUtils::cmdIdIsStart CmdId:257
2019-10-20 08:41:03.333875 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] AsmAgent:getOracleSidAttrib 2 getResAttrib USR_ORA_INST_NAME oracleSid:
2019-10-20 08:41:03.333885 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] AsmAgent:getOracleSidAttrib 4 getEnvVar USR_ORA_INST_NAME oracle_sid:
2019-10-20 08:41:03.350949 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] clsnUtils::getASMSIDSuffix Node 0 doesn't exist
2019-10-20 08:41:03.350967 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] clsnUtils::getASMSIDSuffix getCssNodeName: Exception CssException
2019-10-20 08:41:03.350978 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] clssnsqlnum ret = 12
2019-10-20 08:41:03.351021 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] clsnUtils::getASMSIDSuffix CSS based nodeNum 1
2019-10-20 08:41:03.351038 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] AsmAgent:getOracleSidAttrib 5 generated oracleSid:+ASM1
2019-10-20 08:41:03.351046 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] AsmAgent:getOracleSidAttrib oracleSid:+ASM1
2019-10-20 08:41:03.351055 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] AsmAgent::refresh ORACLE_SID = +ASM1
2019-10-20 08:41:03.351077 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] Utils::getOracleHomeAttrib getEnvVar oracle_home:/u01/app/12.1.0/grid
2019-10-20 08:41:03.351085 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] Utils::getOracleHomeAttrib oracle_home:/u01/app/12.1.0/grid
2019-10-20 08:41:03.351940 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] ConnectionPool::ConnectionPool 2 m_oracleHome:/u01/app/12.1.0/grid, m_oracleSid:+ASM1, m_usrOraEnv: m_instanceType:2 m_instanceVersion:12.1.0.1.0
2019-10-20 08:41:03.351989 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] AsmAgent::setOracleSidAttrib updating GEN_USR_ORA_INST_NAME to +ASM1
2019-10-20 08:41:03.352013 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] clsnUtils::setResAttrib nonPerX current value GEN_USR_ORA_INST_NAME value +ASM1
2019-10-20 08:41:03.352022 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] setResAttrib clsagfw_modify_attribute attr GEN_USR_ORA_INST_NAME value +ASM1 retCode 0
2019-10-20 08:41:03.352033 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] getResAttrib: attrib name USR_ORA_OPI value false len 5
2019-10-20 08:41:03.352057 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] sclsnInstAgent::sGetStartupComment entry
2019-10-20 08:41:03.352066 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] ConfigFile::getAltEntry altEntry:+ASM1
2019-10-20 08:41:03.352075 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] sclsnInstAgent::sGetStartupComment asmEntry:+ASM1 m_dbName:+ASM1
2019-10-20 08:41:03.352091 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] ConfigFile::getComment name:+asm1 comment:line added by Agent StartCompleted
2019-10-20 08:41:03.352100 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] sclsnInstAgent::sGetStartupComment dbName comment:line added by Agent StartCompleted
2019-10-20 08:41:03.352124 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] InstAgent::checkState gimh oracleSid:+ASM1
2019-10-20 08:41:03.352159 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] Gimh::constructor ohome:/u01/app/12.1.0/grid sid:+ASM1
2019-10-20 08:41:03.352210 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] ConnectionPool::resetConnection s_statusOfConnectionMap 00f3aed8
2019-10-20 08:41:03.352218 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] ConnectionPool::resetConnection sid +ASM1 status 2
2019-10-20 08:41:03.352268 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] Gimh::check OH /u01/app/12.1.0/grid SID +ASM1
2019-10-20 08:41:03.352333 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] Gimh::check condition changes to (GIMH_NEXT_NUM) 0(Abnormal Termination),1(Instance Shutdown) exists
2019-10-20 08:41:03.352342 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] (:CLSN00006:)AsmAgent::gimhChecks failed gimh state 0
2019-10-20 08:41:03.359632 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] checkCrsStat 2 CLSCRS_STAT ret: 184
2019-10-20 08:41:03.359648 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] clsnUtils::error Exception type=2 string=
2019-10-20 08:41:03.359657 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] checkCrsStat 2 status:0 ret:184 usererror:184
2019-10-20 08:41:03.363766 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] AsmAgent::checkCbk: Exception UserErrorException
2019-10-20 08:41:03.363824 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start]
2019-10-20 08:41:03.363954 : USRTHRD:230291200: {0:0:4140} Gimh::destructor gimh_dest_query_ctx rc=0
2019-10-20 08:41:03.363989 : USRTHRD:230291200: {0:0:4140} Gimh::destructor gimh_dest_inst_ctx rc=0
2019-10-20 08:41:03.364017 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] ConnectionPool::stopConnection
2019-10-20 08:41:03.364036 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] ConnectionPool::removeConnection connection count 0
2019-10-20 08:41:03.364046 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] ConnectionPool::removeConnection freed 0
2019-10-20 08:41:03.364053 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] ConnectionPool::stopConnection sid +ASM1 status 1
2019-10-20 08:41:03.364066 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] InstAgent::checkState 1 prev clsagfw_res_status 3 current clsagfw_res_status 2
2019-10-20 08:41:03.364117 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] InstAgent::start 1 writing configFile startOption:StartOption[1]
2019-10-20 08:41:03.364126 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] sModifyConfig for +ASM
2019-10-20 08:41:03.364153 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] sclsnInstAgent::sUpdateFile entry { path:/etc file:oratab pConfigF:0x7f7efc19c498
2019-10-20 08:41:03.364162 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] ConfigFile::getAltName this:0x7f7efc173c90
2019-10-20 08:41:03.364170 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] ConfigFile:getAltName altName:+asm
2019-10-20 08:41:03.364178 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] sclsnInstAgent::sUpdateFile dbname:+ASM1 altName:+asm
2019-10-20 08:41:03.364187 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] ConfigFile::getAltEntry altEntry:+ASM1
2019-10-20 08:41:03.364195 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] sclsnInstAgent::sUpdateFile altEntry:+ASM1 dbName:+ASM1
2019-10-20 08:41:03.364202 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] sclsnInstAgent::sUpdateFile remove ASM dbName:+ASM1
2019-10-20 08:41:03.364318 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] ConfigFile::parse mmap name:-mgmtdb nameWithCase:-MGMTDB value:/u01/app/12.1.0/grid:N comment:line added by Agent
2019-10-20 08:41:03.364366 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] ConfigFile::parse mmap name:prod nameWithCase:prod value:/u01/app/oracle/product/12.1.0/dbhome_1:N comment:line added by Agent
2019-10-20 08:41:03.364380 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] ConfigFile::getValue name +asm1 value
2019-10-20 08:41:03.374428 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] at the end Insert newstr +ASM1:/u01/app/12.1.0/grid:N # line added by Agent
2019-10-20 08:41:03.374493 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] ConfigFile::parse mmap name:-mgmtdb nameWithCase:-MGMTDB value:/u01/app/12.1.0/grid:N comment:line added by Agent
2019-10-20 08:41:03.374521 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] ConfigFile::parse mmap name:prod nameWithCase:prod value:/u01/app/oracle/product/12.1.0/dbhome_1:N comment:line added by Agent
2019-10-20 08:41:03.374545 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] ConfigFile::parse mmap name:+asm1 nameWithCase:+ASM1 value:/u01/app/12.1.0/grid:N comment:line added by Agent
2019-10-20 08:41:03.376134 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] ConfigFile::parse mmap name:-mgmtdb nameWithCase:-MGMTDB value:/u01/app/12.1.0/grid:N comment:line added by Agent
2019-10-20 08:41:03.376187 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] ConfigFile::parse mmap name:prod nameWithCase:prod value:/u01/app/oracle/product/12.1.0/dbhome_1:N comment:line added by Agent
2019-10-20 08:41:03.376214 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] ConfigFile::parse mmap name:+asm1 nameWithCase:+ASM1 value:/u01/app/12.1.0/grid:N comment:line added by Agent
2019-10-20 08:41:03.376225 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] ConfigFile::updateInPlace file /etc/oratab is updated
2019-10-20 08:41:03.376235 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] sclsnInstAgent::sUpdateFile file updated with dbName +ASM1 m_configValue /u01/app/12.1.0/grid:N
2019-10-20 08:41:03.382575 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] sclsnInstAgent::sUpdateFile entry { path: file:orastart.node1 pConfigF:0x7f7efc19c4a0
2019-10-20 08:41:03.382591 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] ConfigFile::getAltName this:0x7f7efc237040
2019-10-20 08:41:03.382600 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] ConfigFile:getAltName altName:+asm
2019-10-20 08:41:03.382608 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] sclsnInstAgent::sUpdateFile dbname:+ASM1 altName:+asm
2019-10-20 08:41:03.382618 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] ConfigFile::getAltEntry altEntry:+ASM1
2019-10-20 08:41:03.382626 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] sclsnInstAgent::sUpdateFile altEntry:+ASM1 dbName:+ASM1
2019-10-20 08:41:03.382633 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] sclsnInstAgent::sUpdateFile remove ASM dbName:+ASM1
2019-10-20 08:41:03.382699 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] ConfigFile::parse mmap name:-mgmtdb nameWithCase:-MGMTDB value:/u01/app/12.1.0/grid:N comment:line added by Agent StartCompleted
2019-10-20 08:41:03.382710 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] ConfigFile::getValue name +asm1 value
2019-10-20 08:41:03.390866 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] at the end Insert newstr +ASM1:/u01/app/12.1.0/grid:N # line added by AgentStartOption[1]
2019-10-20 08:41:03.390966 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] ConfigFile::parse mmap name:-mgmtdb nameWithCase:-MGMTDB value:/u01/app/12.1.0/grid:N comment:line added by Agent StartCompleted
2019-10-20 08:41:03.390992 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] ConfigFile::parse mmap name:+asm1 nameWithCase:+ASM1 value:/u01/app/12.1.0/grid:N comment:line added by AgentStartOption[1]
2019-10-20 08:41:03.392113 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] ConfigFile::parse mmap name:-mgmtdb nameWithCase:-MGMTDB value:/u01/app/12.1.0/grid:N comment:line added by Agent StartCompleted
2019-10-20 08:41:03.392148 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] ConfigFile::parse mmap name:+asm1 nameWithCase:+ASM1 value:/u01/app/12.1.0/grid:N comment:line added by AgentStartOption[1]
2019-10-20 08:41:03.392158 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] ConfigFile::updateInPlace file /u01/app/12.1.0/grid/srvm/admin/orastart.node1 is updated
2019-10-20 08:41:03.392169 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] sclsnInstAgent::sUpdateFile file updated with dbName +ASM1 m_configValue /u01/app/12.1.0/grid:N
2019-10-20 08:41:03.399795 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] InstConnection:InstConnection: init:e8098c50 oracleHome:/u01/app/12.1.0/grid oracleSid:+ASM1 instanceType:2 instanceVersion:12.1.0.1.0
2019-10-20 08:41:03.399861 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] clsnInstConnection::makeConnectStr UsrOraEnv m_oracleHome /u01/app/12.1.0/grid Crshome /u01/app/12.1.0/grid
2019-10-20 08:41:03.399956 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] makeConnectStr = (DESCRIPTION=(ADDRESS=(PROTOCOL=beq)(PROGRAM=/u01/app/12.1.0/grid/bin/oracle)(ARGV0=oracle+ASM1)(ENVS='ORACLE_HOME=/u01/app/12.1.0/grid,ORACLE_SID=+ASM1')(ARGS='(DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))')(CONNECT_DATA=(SID=+ASM1))))
2019-10-20 08:41:03.400776 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] Container:start oracle home /u01/app/12.1.0/grid
2019-10-20 08:41:03.400939 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] InstConnection::connectInt: server not attached
2019-10-20 08:41:04.435306 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] InstConnection:connectInt connected
2019-10-20 08:41:04.435306 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] InstAgent::startup
2019-10-20 08:41:19.571531 : AGFW:250087008: Recvd request to shed the threads
2019-10-20 08:41:19.571551 :CLSFRAME:250087008: TM [MultiThread] is changing desired thread # to 6. Current # is 7
2019-10-20 08:41:19.579005 :CLSFRAME:211379968: {0:21:47} Worker thread is exiting in TM [MultiThread] to meet the desired count of 6. New count is 6
2019-10-20 08:41:19.579320 :CLSDYNAM:217683712: [ora.gipcd]{0:21:2} [check] ClsdmClient::sendMessage clsdmc_respget return: status=0, ecode=0
2019-10-20 08:41:19.579716 : CLSDMC:215582464: command 0 failed with status 1
2019-10-20 08:41:19.579746 :CLSDYNAM:215582464: [ora.evmd]{0:21:2} [check] ClsdmClient::sendMessage clsdmc_respget return: status=0, ecode=1
2019-10-20 08:41:19.579762 :CLSDYNAM:215582464: [ora.evmd]{0:21:2} [check] DaemonAgent::check returned 0
2019-10-20 08:41:19.579774 :CLSDYNAM:215582464: [ora.evmd]{0:21:2} [check] Deep check returned 1
2019-10-20 08:41:19.579870 :CLSDYNAM:235296512: [ora.gpnpd]{0:21:2} [check] ClsdmClient::sendMessage clsdmc_respget return: status=0, ecode=0
2019-10-20 08:41:19.587013 :CLSDYNAM:213481216: [ora.mdnsd]{0:21:2} [check] ClsdmClient::sendMessage clsdmc_respget return: status=0, ecode=0
2019-10-20 08:41:49.430435 : AGFW:233195264: {0:0:4016} Agent received the message: AGENT_HB[Engine] ID 12293:18800
2019-10-20 08:41:49.574757 :CLSDYNAM:213481216: [ora.gpnpd]{0:21:2} [check] ClsdmClient::sendMessage clsdmc_respget return: status=0, ecode=0
2019-10-20 08:41:49.575017 :CLSDYNAM:235296512: [ora.gipcd]{0:21:2} [check] ClsdmClient::sendMessage clsdmc_respget return: status=0, ecode=0
2019-10-20 08:41:49.576239 : CLSDMC:217683712: command 0 failed with status 1
2019-10-20 08:41:49.576265 :CLSDYNAM:217683712: [ora.evmd]{0:21:2} [check] ClsdmClient::sendMessage clsdmc_respget return: status=0, ecode=1
2019-10-20 08:41:49.576278 :CLSDYNAM:217683712: [ora.evmd]{0:21:2} [check] DaemonAgent::check returned 0
2019-10-20 08:41:49.576288 :CLSDYNAM:217683712: [ora.evmd]{0:21:2} [check] Deep check returned 1
2019-10-20 08:41:49.576464 :CLSFRAME:250087008: TM [MultiThread] is changing desired thread # to 7. Current # is 6
2019-10-20 08:41:49.576522 :CLSFRAME:250087008: TM [MultiThread] is changing desired thread # to 8. Current # is 7
2019-10-20 08:41:49.576962 :CLSFRAME:250087008: TM [MultiThread] is changing desired thread # to 9. Current # is 8
2019-10-20 08:41:49.577016 :CLSFRAME:250087008: TM [MultiThread] is changing desired thread # to 10. Current # is 9
2019-10-20 08:41:49.589477 :CLSDYNAM:215582464: [ora.mdnsd]{0:21:2} [check] ClsdmClient::sendMessage clsdmc_respget return: status=0, ecode=0
2019-10-20 08:42:19.429163 : AGFW:233195264: {0:0:4016} Agent received the message: AGENT_HB[Engine] ID 12293:18830
2019-10-20 08:42:19.572608 :CLSDYNAM:215582464: [ora.gpnpd]{0:21:2} [check] ClsdmClient::sendMessage clsdmc_respget return: status=0, ecode=0
2019-10-20 08:42:19.576148 :CLSDYNAM:213481216: [ora.gipcd]{0:21:2} [check] ClsdmClient::sendMessage clsdmc_respget return: status=0, ecode=0
2019-10-20 08:42:19.576605 : CLSDMC:4026529536: command 0 failed with status 1
2019-10-20 08:42:19.576630 :CLSDYNAM:4026529536: [ora.evmd]{0:21:2} [check] ClsdmClient::sendMessage clsdmc_respget return: status=0, ecode=1
2019-10-20 08:42:19.576644 :CLSDYNAM:4026529536: [ora.evmd]{0:21:2} [check] DaemonAgent::check returned 0
2019-10-20 08:42:19.576653 :CLSDYNAM:4026529536: [ora.evmd]{0:21:2} [check] Deep check returned 1
2019-10-20 08:42:19.589187 : AGFW:250087008: Recvd request to shed the threads
2019-10-20 08:42:19.589187 :CLSFRAME:250087008: TM [MultiThread] is changing desired thread # to 8. Current # is 10
2019-10-20 08:42:19.589435 :CLSFRAME:4026529536: {0:21:50} Worker thread is exiting in TM [MultiThread] to meet the desired count of 8. New count is 9
2019-10-20 08:42:19.589593 :CLSFRAME:215582464: {0:21:51} Worker thread is exiting in TM [MultiThread] to meet the desired count of 8. New count is 8
2019-10-20 08:42:19.591605 :CLSDYNAM:235296512: [ora.mdnsd]{0:21:2} [check] ClsdmClient::sendMessage clsdmc_respget return: status=0, ecode=0
2019-10-20 08:42:49.424813 : AGFW:233195264: {0:0:4016} Agent received the message: AGENT_HB[Engine] ID 12293:18858
2019-10-20 08:42:49.569315 : CLSDMC:217683712: command 0 failed with status 1
2019-10-20 08:42:49.569342 :CLSDYNAM:217683712: [ora.evmd]{0:21:2} [check] ClsdmClient::sendMessage clsdmc_respget return: status=0, ecode=1
2019-10-20 08:42:49.569354 :CLSDYNAM:217683712: [ora.evmd]{0:21:2} [check] DaemonAgent::check returned 0
2019-10-20 08:42:49.569361 :CLSDYNAM:217683712: [ora.evmd]{0:21:2} [check] Deep check returned 1
2019-10-20 08:42:49.569609 :CLSDYNAM:235296512: [ora.gpnpd]{0:21:2} [check] ClsdmClient::sendMessage clsdmc_respget return: status=0, ecode=0
2019-10-20 08:42:49.569936 :CLSDYNAM:241600256: [ora.gipcd]{0:21:2} [check] ClsdmClient::sendMessage clsdmc_respget return: status=0, ecode=0
2019-10-20 08:42:49.590266 : AGFW:250087008: Recvd request to shed the threads
2019-10-20 08:42:49.590278 :CLSFRAME:250087008: TM [MultiThread] is changing desired thread # to 7. Current # is 8
2019-10-20 08:42:49.590488 :CLSFRAME:241600256: {0:21:53} Worker thread is exiting in TM [MultiThread] to meet the desired count of 7. New count is 7
2019-10-20 08:42:49.590986 :CLSDYNAM:235296512: [ora.mdnsd]{0:21:2} [check] ClsdmClient::sendMessage clsdmc_respget return: status=0, ecode=0
2019-10-20 08:43:19.426904 : AGFW:233195264: {0:0:4016} Agent received the message: AGENT_HB[Engine] ID 12293:18888
2019-10-20 08:43:19.570782 :CLSDYNAM:235296512: [ora.gpnpd]{0:21:2} [check] ClsdmClient::sendMessage clsdmc_respget return: status=0, ecode=0
2019-10-20 08:43:19.571204 : CLSDMC:217683712: command 0 failed with status 1
2019-10-20 08:43:19.571234 :CLSDYNAM:217683712: [ora.evmd]{0:21:2} [check] ClsdmClient::sendMessage clsdmc_respget return: status=0, ecode=1
2019-10-20 08:43:19.571244 :CLSDYNAM:217683712: [ora.evmd]{0:21:2} [check] DaemonAgent::check returned 0
2019-10-20 08:43:19.571251 :CLSDYNAM:217683712: [ora.evmd]{0:21:2} [check] Deep check returned 1
2019-10-20 08:43:19.571685 :CLSDYNAM:228189952: [ora.gipcd]{0:21:2} [check] ClsdmClient::sendMessage clsdmc_respget return: status=0, ecode=0
2019-10-20 08:43:19.593009 : AGFW:250087008: Recvd request to shed the threads
2019-10-20 08:43:19.593022 :CLSFRAME:250087008: TM [MultiThread] is changing desired thread # to 6. Current # is 7
2019-10-20 08:43:19.593050 :CLSFRAME:217683712: {0:21:55} Worker thread is exiting in TM [MultiThread] to meet the desired count of 6. New count is 6
2019-10-20 08:43:19.594164 :CLSDYNAM:235296512: [ora.mdnsd]{0:21:2} [check] ClsdmClient::sendMessage clsdmc_respget return: status=0, ecode=0
2019-10-20 08:43:49.426579 : AGFW:233195264: {0:0:4016} Agent received the message: AGENT_HB[Engine] ID 12293:18918
2019-10-20 08:43:49.569336 :CLSDYNAM:213481216: [ora.gipcd]{0:21:2} [check] ClsdmClient::sendMessage clsdmc_respget return: status=0, ecode=0
2019-10-20 08:43:49.569490 :CLSDYNAM:235296512: [ora.gpnpd]{0:21:2} [check] ClsdmClient::sendMessage clsdmc_respget return: status=0, ecode=0
2019-10-20 08:43:49.569638 :CLSFRAME:250087008: TM [MultiThread] is changing desired thread # to 7. Current # is 6
2019-10-20 08:43:49.570868 : CLSDMC:211379968: command 0 failed with status 1
2019-10-20 08:43:49.570922 :CLSDYNAM:211379968: [ora.evmd]{0:21:2} [check] ClsdmClient::sendMessage clsdmc_respget return: status=0, ecode=1
2019-10-20 08:43:49.570931 :CLSDYNAM:211379968: [ora.evmd]{0:21:2} [check] DaemonAgent::check returned 0
2019-10-20 08:43:49.570936 :CLSDYNAM:211379968: [ora.evmd]{0:21:2} [check] Deep check returned 1
2019-10-20 08:43:49.592201 : AGFW:250087008: Recvd request to shed the threads
2019-10-20 08:43:49.592214 :CLSFRAME:250087008: TM [MultiThread] is changing desired thread # to 6. Current # is 7
2019-10-20 08:43:49.592247 :CLSFRAME:211379968: {0:21:57} Worker thread is exiting in TM [MultiThread] to meet the desired count of 6. New count is 6
2019-10-20 08:43:49.593275 :CLSDYNAM:213481216: [ora.mdnsd]{0:21:2} [check] ClsdmClient::sendMessage clsdmc_respget return: status=0, ecode=0
2019-10-20 08:44:04.391305 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] ORA-03113: end-of-file on communication channel
Process ID: 0
Session ID: 0 Serial number: 0
2019-10-20 08:44:04.391353 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] InstAgent::start: 1 errcode 3113
2019-10-20 08:44:04.391363 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] InstAgent::start: untrapped error
2019-10-20 08:44:04.391381 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] InstAgent::start: 2 errcode 3113
2019-10-20 08:44:04.391427 : USRTHRD:230291200: {0:0:4140} InstConnection:~InstConnection: this e8098c50
2019-10-20 08:44:04.391566 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] clean {
2019-10-20 08:44:04.391582 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] InstAgent::stop_option stop mode immediate option 1
2019-10-20 08:44:04.391590 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] InstAgent::stop {
2019-10-20 08:44:04.391602 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] getResAttrib: attrib name USR_ORA_OPI value false len 5
2019-10-20 08:44:04.391625 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] InstAgent::stop original reason system do shutdown abort
2019-10-20 08:44:04.391690 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] InstAgent::checkState gimh oracleSid:+ASM1
2019-10-20 08:44:04.391698 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] Gimh::constructor ohome:/u01/app/12.1.0/grid sid:+ASM1
2019-10-20 08:44:04.391741 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] ConnectionPool::resetConnection s_statusOfConnectionMap 00f3aed8
2019-10-20 08:44:04.391748 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] ConnectionPool::resetConnection sid +ASM1 status 2
2019-10-20 08:44:04.391793 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] Gimh::check OH /u01/app/12.1.0/grid SID +ASM1
2019-10-20 08:44:04.391851 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] Gimh::check condition changes to (GIMH_NEXT_NUM) 0(Abnormal Termination) exists
2019-10-20 08:44:04.391860 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] (:CLSN00006:)AsmAgent::gimhChecks failed gimh state 0
2019-10-20 08:44:04.398239 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] checkCrsStat 2 CLSCRS_STAT ret: 184
2019-10-20 08:44:04.398254 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] clsnUtils::error Exception type=2 string=
2019-10-20 08:44:04.398263 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] checkCrsStat 2 status:0 ret:184 usererror:184
2019-10-20 08:44:04.399681 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] AsmAgent::checkCbk: Exception UserErrorException
2019-10-20 08:44:04.399697 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start]
2019-10-20 08:44:04.399780 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] (:CLSN00006:)InstAgent::checkState return unplanned offline
2019-10-20 08:44:04.399817 : USRTHRD:230291200: {0:0:4140} Gimh::destructor gimh_dest_query_ctx rc=0
2019-10-20 08:44:04.399847 : USRTHRD:230291200: {0:0:4140} Gimh::destructor gimh_dest_inst_ctx rc=0
2019-10-20 08:44:04.399873 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] ConnectionPool::stopConnection
2019-10-20 08:44:04.399891 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] ConnectionPool::removeConnection connection count 0
2019-10-20 08:44:04.399905 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] ConnectionPool::removeConnection freed 0
2019-10-20 08:44:04.399912 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] ConnectionPool::stopConnection sid +ASM1 status 1
2019-10-20 08:44:04.400083 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] InstAgent::checkState 1 prev clsagfw_res_status 2 current clsagfw_res_status 1
2019-10-20 08:44:04.400155 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] InstAgent::stop shutdown mode: 4
2019-10-20 08:44:04.400193 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] ConnectionPool::getConnection 160 ERROR sid +ASM1 pool status 1
2019-10-20 08:44:04.400914 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] InstAgent::stop pool pConnxn 00000000
2019-10-20 08:44:04.400950 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] InstConnection:InstConnection: init:e809be50 oracleHome:/u01/app/12.1.0/grid oracleSid:+ASM1 instanceType:2 instanceVersion:12.1.0.1.0
2019-10-20 08:44:04.401154 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] clsnInstConnection::makeConnectStr UsrOraEnv m_oracleHome /u01/app/12.1.0/grid Crshome /u01/app/12.1.0/grid
2019-10-20 08:44:04.401194 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] makeConnectStr = (DESCRIPTION=(ADDRESS=(PROTOCOL=beq)(PROGRAM=/u01/app/12.1.0/grid/bin/oracle)(ARGV0=oracle+ASM1)(ENVS='ORACLE_HOME=/u01/app/12.1.0/grid,ORACLE_SID=+ASM1')(ARGS='(DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))')(CONNECT_DATA=(SID=+ASM1))))
2019-10-20 08:44:04.401948 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] InstAgent::stop non pool pConnxn 1 e809be50
2019-10-20 08:44:04.402001 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] InstConnection::connectInt: server not attached
2019-10-20 08:44:05.431049 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] InstConnection:connectInt connected
2019-10-20 08:44:05.431866 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] InstConnection::shutdown mode 4
2019-10-20 08:44:06.648102 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] ConnectionPool::removeConnection connection count 1
2019-10-20 08:44:06.648123 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] ConnectionPool::removeConnection sid +ASM1, InstConnection e809be50
2019-10-20 08:44:06.648136 : USRTHRD:230291200: {0:0:4140} InstConnection::breakCall pConnxn:e809be50 DetachLock:00f31fe8 m_pSrvH:e81df6f0, m_pSvcH:e8206868
2019-10-20 08:44:06.648168 : USRTHRD:230291200: {0:0:4140} InstConnection:~InstConnection: this e809be50
2019-10-20 08:44:06.648721 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] ConnectionPool::removeConnection delete InstConnection e809be50
2019-10-20 08:44:06.648741 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] ConnectionPool::removeConnection freed 1
2019-10-20 08:44:06.648759 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] ConnectionPool::stopConnection
2019-10-20 08:44:06.648779 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] ConnectionPool::removeConnection connection count 0
2019-10-20 08:44:06.648789 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] ConnectionPool::removeConnection freed 0
2019-10-20 08:44:06.648797 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] ConnectionPool::stopConnection sid +ASM1 status 1
2019-10-20 08:44:06.648813 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] InstAgent::stop db/asm
2019-10-20 08:44:06.648823 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] ConnectionPool::stopConnection
2019-10-20 08:44:06.648838 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] ConnectionPool::removeConnection connection count 0
2019-10-20 08:44:06.648861 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] ConnectionPool::removeConnection freed 0
2019-10-20 08:44:06.648875 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] ConnectionPool::stopConnection sid +ASM1 status 1
2019-10-20 08:44:06.648948 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] getResAttrib: attrib name USR_ORA_OPI value false len 5
2019-10-20 08:44:06.648976 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] InstAgent::stop Container->stop, isShutdown = 0
2019-10-20 08:44:06.649023 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] InstAgent::stop calling initCssData
2019-10-20 08:44:06.649047 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] CssData Constructor groupName:HB+ASM cssRegistrationType:0 memno:-1
2019-10-20 08:44:06.649058 : USRTHRD:230291200: {0:0:4140} CssData::initMutex &m_cssDataLock:e803f388
2019-10-20 08:44:06.654728 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] CssGroup::regis registration of groupName:HB+ASM mbrid:-1 m_grpNum:-1 regType:0 succeeded
2019-10-20 08:44:06.654749 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] CssData::Constructor regis succeeded groupName:HB+ASM memno:-1
2019-10-20 08:44:06.654759 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] CssData::Constructor m_pCssGroup:e81d13a0 group HB+ASM
2019-10-20 08:44:06.654769 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] InstAgent::init m_groupName:HB+ASM m_pCssData e803d2c0
2019-10-20 08:44:06.654779 : USRTHRD:230291200: {0:0:4140} CssData Destructor groupName:HB+ASM
2019-10-20 08:44:06.654791 : USRTHRD:230291200: {0:0:4140} CssGroup::dereg default regType:0
2019-10-20 08:44:06.657219 : USRTHRD:230291200: {0:0:4140} CssData::destroyMutex &m_cssDataLock:e803f388
2019-10-20 08:44:06.657240 : USRTHRD:230291200: {0:0:4140} CssData::Destructor
2019-10-20 08:44:06.657260 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] InstAgent::stop prev clsagfw_res_status 1 current clsagfw_res_status 2
2019-10-20 08:44:06.657270 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] InstAgent::stop m_prevState:2 }
2019-10-20 08:44:06.657279 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] clean }
2019-10-20 08:44:06.657287 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] InstAgent::start exception }
2019-10-20 08:44:06.657299 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] InstAgent::start 2.1 writing configFile startOption: StartCompleted
2019-10-20 08:44:06.657311 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] sModifyConfig for +ASM
2019-10-20 08:44:06.657339 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] sclsnInstAgent::sUpdateFile entry { path:/etc file:oratab pConfigF:0x7f7efc19c498
2019-10-20 08:44:06.657380 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] ConfigFile::getAltName this:0x7f7efc173c90
2019-10-20 08:44:06.657391 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] ConfigFile:getAltName altName:+asm
2019-10-20 08:44:06.657400 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] sclsnInstAgent::sUpdateFile dbname:+ASM1 altName:+asm
2019-10-20 08:44:06.657411 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] ConfigFile::getAltEntry altEntry:+ASM1
2019-10-20 08:44:06.657622 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] sclsnInstAgent::sUpdateFile altEntry:+ASM1 dbName:+ASM1
2019-10-20 08:44:06.657633 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] sclsnInstAgent::sUpdateFile remove ASM dbName:+ASM1
2019-10-20 08:44:06.657765 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] ConfigFile::parse mmap name:-mgmtdb nameWithCase:-MGMTDB value:/u01/app/12.1.0/grid:N comment:line added by Agent
2019-10-20 08:44:06.657788 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] ConfigFile::parse mmap name:prod nameWithCase:prod value:/u01/app/oracle/product/12.1.0/dbhome_1:N comment:line added by Agent
2019-10-20 08:44:06.657800 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] ConfigFile::getValue name +asm1 value
2019-10-20 08:44:06.667601 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] at the end Insert newstr +ASM1:/u01/app/12.1.0/grid:N # line added by Agent
2019-10-20 08:44:06.667669 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] ConfigFile::parse mmap name:-mgmtdb nameWithCase:-MGMTDB value:/u01/app/12.1.0/grid:N comment:line added by Agent
2019-10-20 08:44:06.667944 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] ConfigFile::parse mmap name:prod nameWithCase:prod value:/u01/app/oracle/product/12.1.0/dbhome_1:N comment:line added by Agent
2019-10-20 08:44:06.667974 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] ConfigFile::parse mmap name:+asm1 nameWithCase:+ASM1 value:/u01/app/12.1.0/grid:N comment:line added by Agent
2019-10-20 08:44:06.670583 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] ConfigFile::parse mmap name:-mgmtdb nameWithCase:-MGMTDB value:/u01/app/12.1.0/grid:N comment:line added by Agent
2019-10-20 08:44:06.670617 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] ConfigFile::parse mmap name:prod nameWithCase:prod value:/u01/app/oracle/product/12.1.0/dbhome_1:N comment:line added by Agent
2019-10-20 08:44:06.670645 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] ConfigFile::parse mmap name:+asm1 nameWithCase:+ASM1 value:/u01/app/12.1.0/grid:N comment:line added by Agent
2019-10-20 08:44:06.670657 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] ConfigFile::updateInPlace file /etc/oratab is updated
2019-10-20 08:44:06.670672 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] sclsnInstAgent::sUpdateFile file updated with dbName +ASM1 m_configValue /u01/app/12.1.0/grid:N
2019-10-20 08:44:06.676608 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] sclsnInstAgent::sUpdateFile entry { path: file:orastart.node1 pConfigF:0x7f7efc19c4a0
2019-10-20 08:44:06.676627 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] ConfigFile::getAltName this:0x7f7efc237040
2019-10-20 08:44:06.676637 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] ConfigFile:getAltName altName:+asm
2019-10-20 08:44:06.676646 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] sclsnInstAgent::sUpdateFile dbname:+ASM1 altName:+asm
2019-10-20 08:44:06.676657 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] ConfigFile::getAltEntry altEntry:+ASM1
2019-10-20 08:44:06.676665 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] sclsnInstAgent::sUpdateFile altEntry:+ASM1 dbName:+ASM1
2019-10-20 08:44:06.677648 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] sclsnInstAgent::sUpdateFile remove ASM dbName:+ASM1
2019-10-20 08:44:06.677751 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] ConfigFile::parse mmap name:-mgmtdb nameWithCase:-MGMTDB value:/u01/app/12.1.0/grid:N comment:line added by Agent StartCompleted
2019-10-20 08:44:06.677776 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] ConfigFile::getValue name +asm1 value
2019-10-20 08:44:06.684743 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] at the end Insert newstr +ASM1:/u01/app/12.1.0/grid:N # line added by Agent StartCompleted
2019-10-20 08:44:06.684951 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] ConfigFile::parse mmap name:-mgmtdb nameWithCase:-MGMTDB value:/u01/app/12.1.0/grid:N comment:line added by Agent StartCompleted
2019-10-20 08:44:06.684970 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] ConfigFile::parse mmap name:+asm1 nameWithCase:+ASM1 value:/u01/app/12.1.0/grid:N comment:line added by Agent StartCompleted
2019-10-20 08:44:06.685331 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] ConfigFile::parse mmap name:-mgmtdb nameWithCase:-MGMTDB value:/u01/app/12.1.0/grid:N comment:line added by Agent StartCompleted
2019-10-20 08:44:06.685352 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] ConfigFile::parse mmap name:+asm1 nameWithCase:+ASM1 value:/u01/app/12.1.0/grid:N comment:line added by Agent StartCompleted
2019-10-20 08:44:06.685359 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] ConfigFile::updateInPlace file /u01/app/12.1.0/grid/srvm/admin/orastart.node1 is updated
2019-10-20 08:44:06.685366 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] sclsnInstAgent::sUpdateFile file updated with dbName +ASM1 m_configValue /u01/app/12.1.0/grid:N
2019-10-20 08:44:06.689546 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] Agent::commonStart Exception OCIException
2019-10-20 08:44:06.690747 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] clsnUtils::error Exception type=2 string=
CRS-5017: The resource action "ora.asm start" encountered the following error:
ORA-03113: end-of-file on communication channel
Process ID: 0
Session ID: 0 Serial number: 0
. For details refer to "(:CLSN00107:)" in "/u01/app/grid/diag/crs/node1/crs/trace/ohasd_oraagent_grid.trc".
2019-10-20 08:44:06.690749 : AGFW:230291200: {0:0:4140} sending status msg [CRS-5017: The resource action "ora.asm start" encountered the following error:
ORA-03113: end-of-file on communication channel
Process ID: 0
Session ID: 0 Serial number: 0
. For details refer to "(:CLSN00107:)" in "/u01/app/grid/diag/crs/node1/crs/trace/ohasd_oraagent_grid.trc".
] for start for resource: ora.asm 1 1
2019-10-20 08:44:06.691065 : AGFW:233195264: {0:0:4140} Agent sending reply for: RESOURCE_START[ora.asm 1 1] ID 4098:18762
2019-10-20 08:44:06.693342 :CLSDYNAM:230291200: [ ora.asm]{0:0:4140} [start] (:CLSN00107:) clsn_agent::start }
2019-10-20 08:44:06.693352 : AGFW:230291200: {0:0:4140} Command: start for resource: ora.asm 1 1 completed with status: FAIL
2019-10-20 08:44:06.693712 : AGFW:233195264: {0:0:4140} Agent sending reply for: RESOURCE_START[ora.asm 1 1] ID 4098:18762
2019-10-20 08:44:06.695648 :CLSDYNAM:228189952: [ ora.asm]{0:0:4140} [check] InstAgent::checkState gimh oracleSid:+ASM1
2019-10-20 08:44:06.695681 :CLSDYNAM:228189952: [ ora.asm]{0:0:4140} [check] Gimh::constructor ohome:/u01/app/12.1.0/grid sid:+ASM1
2019-10-20 08:44:06.695814 :CLSDYNAM:228189952: [ ora.asm]{0:0:4140} [check] ConnectionPool::resetConnection s_statusOfConnectionMap 00f3aed8
2019-10-20 08:44:06.695837 :CLSDYNAM:228189952: [ ora.asm]{0:0:4140} [check] ConnectionPool::resetConnection sid +ASM1 status 2
2019-10-20 08:44:06.695928 :CLSDYNAM:228189952: [ ora.asm]{0:0:4140} [check] Gimh::check OH /u01/app/12.1.0/grid SID +ASM1
2019-10-20 08:44:06.695985 :CLSDYNAM:228189952: [ ora.asm]{0:0:4140} [check] Gimh::check condition changes to (GIMH_NEXT_NUM) 0(Abnormal Termination),1(Instance Shutdown) exists
2019-10-20 08:44:06.695991 :CLSDYNAM:228189952: [ ora.asm]{0:0:4140} [check] (:CLSN00006:)AsmAgent::gimhChecks failed gimh state 0
2019-10-20 08:44:06.700803 :CLSDYNAM:228189952: [ ora.asm]{0:0:4140} [check] checkCrsStat 2 CLSCRS_STAT ret: 184
2019-10-20 08:44:06.701579 :CLSDYNAM:228189952: [ ora.asm]{0:0:4140} [check] clsnUtils::error Exception type=2 string=
2019-10-20 08:44:06.701609 :CLSDYNAM:228189952: [ ora.asm]{0:0:4140} [check] checkCrsStat 2 status:0 ret:184 usererror:184
2019-10-20 08:44:06.703250 :CLSDYNAM:228189952: [ ora.asm]{0:0:4140} [check] AsmAgent::checkCbk: Exception UserErrorException
2019-10-20 08:44:06.703292 :CLSDYNAM:228189952: [ ora.asm]{0:0:4140} [check]
2019-10-20 08:44:06.703365 : USRTHRD:228189952: {0:0:4140} Gimh::destructor gimh_dest_query_ctx rc=0
2019-10-20 08:44:06.703886 : USRTHRD:228189952: {0:0:4140} Gimh::destructor gimh_dest_inst_ctx rc=0
2019-10-20 08:44:06.703914 :CLSDYNAM:228189952: [ ora.asm]{0:0:4140} [check] ConnectionPool::stopConnection
2019-10-20 08:44:06.703927 :CLSDYNAM:228189952: [ ora.asm]{0:0:4140} [check] ConnectionPool::removeConnection connection count 0
2019-10-20 08:44:06.703933 :CLSDYNAM:228189952: [ ora.asm]{0:0:4140} [check] ConnectionPool::removeConnection freed 0
2019-10-20 08:44:06.703938 :CLSDYNAM:228189952: [ ora.asm]{0:0:4140} [check] ConnectionPool::stopConnection sid +ASM1 status 1
2019-10-20 08:44:06.703947 :CLSDYNAM:228189952: [ ora.asm]{0:0:4140} [check] InstAgent::checkState 1 prev clsagfw_res_status 2 current clsagfw_res_status 1
2019-10-20 08:44:06.704218 : AGFW:233195264: {0:0:4140} ora.asm 1 1 state changed from: STARTING to: OFFLINE
2019-10-20 08:44:06.704313 : AGFW:233195264: {0:0:4140} Agent sending last reply for: RESOURCE_START[ora.asm 1 1] ID 4098:18762
2019-10-20 08:44:06.708220 : AGFW:233195264: {0:0:4140} Agent received the message: RESOURCE_CLEAN[ora.asm 1 1] ID 4100:18937
2019-10-20 08:44:06.708239 : AGFW:233195264: {0:0:4140} Preparing CLEAN command for: ora.asm 1 1
2019-10-20 08:44:06.708245 : AGFW:233195264: {0:0:4140} ora.asm 1 1 state changed from: OFFLINE to: CLEANING
2019-10-20 08:44:06.709306 :CLSDYNAM:213481216: [ ora.asm]{0:0:4140} [clean] (:CLSN00106:) clsn_agent::clean {
2019-10-20 08:44:06.709365 :CLSDYNAM:213481216: [ ora.asm]{0:0:4140} [clean] clean {
2019-10-20 08:44:06.709398 :CLSDYNAM:213481216: [ ora.asm]{0:0:4140} [clean] InstAgent::stop_option stop mode immediate option 1
2019-10-20 08:44:06.709424 :CLSDYNAM:213481216: [ ora.asm]{0:0:4140} [clean] InstAgent::stop {
2019-10-20 08:44:06.709451 :CLSDYNAM:213481216: [ ora.asm]{0:0:4140} [clean] getResAttrib: attrib name USR_ORA_OPI value false len 5
2019-10-20 08:44:06.709485 :CLSDYNAM:213481216: [ ora.asm]{0:0:4140} [clean] InstAgent::stop original reason system do shutdown abort
2019-10-20 08:44:06.709584 :CLSDYNAM:213481216: [ ora.asm]{0:0:4140} [clean] InstAgent::checkState gimh oracleSid:+ASM1
2019-10-20 08:44:06.709602 :CLSDYNAM:213481216: [ ora.asm]{0:0:4140} [clean] Gimh::constructor ohome:/u01/app/12.1.0/grid sid:+ASM1
2019-10-20 08:44:06.709653 :CLSDYNAM:213481216: [ ora.asm]{0:0:4140} [clean] ConnectionPool::resetConnection s_statusOfConnectionMap 00f3aed8
2019-10-20 08:44:06.709667 :CLSDYNAM:213481216: [ ora.asm]{0:0:4140} [clean] ConnectionPool::resetConnection sid +ASM1 status 2
2019-10-20 08:44:06.709727 :CLSDYNAM:213481216: [ ora.asm]{0:0:4140} [clean] Gimh::check OH /u01/app/12.1.0/grid SID +ASM1
2019-10-20 08:44:06.709786 :CLSDYNAM:213481216: [ ora.asm]{0:0:4140} [clean] Gimh::check condition changes to (GIMH_NEXT_NUM) 0(Abnormal Termination),1(Instance Shutdown) exists
2019-10-20 08:44:06.709799 :CLSDYNAM:213481216: [ ora.asm]{0:0:4140} [clean] (:CLSN00006:)AsmAgent::gimhChecks failed gimh state 0
2019-10-20 08:44:06.714869 :CLSDYNAM:213481216: [ ora.asm]{0:0:4140} [clean] checkCrsStat 2 CLSCRS_STAT ret: 184
2019-10-20 08:44:06.714985 :CLSDYNAM:213481216: [ ora.asm]{0:0:4140} [clean] clsnUtils::error Exception type=2 string=
2019-10-20 08:44:06.714992 :CLSDYNAM:213481216: [ ora.asm]{0:0:4140} [clean] checkCrsStat 2 status:0 ret:184 usererror:184
2019-10-20 08:44:06.720156 :CLSDYNAM:213481216: [ ora.asm]{0:0:4140} [clean] AsmAgent::checkCbk: Exception UserErrorException
2019-10-20 08:44:06.720173 :CLSDYNAM:213481216: [ ora.asm]{0:0:4140} [clean]
2019-10-20 08:44:06.720275 : USRTHRD:213481216: {0:0:4140} Gimh::destructor gimh_dest_query_ctx rc=0
2019-10-20 08:44:06.720307 : USRTHRD:213481216: {0:0:4140} Gimh::destructor gimh_dest_inst_ctx rc=0
2019-10-20 08:44:06.720337 :CLSDYNAM:213481216: [ ora.asm]{0:0:4140} [clean] ConnectionPool::stopConnection
2019-10-20 08:44:06.720351 :CLSDYNAM:213481216: [ ora.asm]{0:0:4140} [clean] ConnectionPool::removeConnection connection count 0
2019-10-20 08:44:06.720357 :CLSDYNAM:213481216: [ ora.asm]{0:0:4140} [clean] ConnectionPool::removeConnection freed 0
2019-10-20 08:44:06.720363 :CLSDYNAM:213481216: [ ora.asm]{0:0:4140} [clean] ConnectionPool::stopConnection sid +ASM1 status 1
2019-10-20 08:44:06.720371 :CLSDYNAM:213481216: [ ora.asm]{0:0:4140} [clean] InstAgent::checkState 1 prev clsagfw_res_status 1 current clsagfw_res_status 2
2019-10-20 08:44:06.720417 :CLSDYNAM:213481216: [ ora.asm]{0:0:4140} [clean] InstAgent::stop shutdown mode: 4
2019-10-20 08:44:06.720440 :CLSDYNAM:213481216: [ ora.asm]{0:0:4140} [clean] ConnectionPool::getConnection 160 ERROR sid +ASM1 pool status 1
2019-10-20 08:44:06.720473 :CLSDYNAM:213481216: [ ora.asm]{0:0:4140} [clean] InstAgent::stop pool pConnxn 00000000
2019-10-20 08:44:06.720493 :CLSDYNAM:213481216: [ ora.asm]{0:0:4140} [clean] InstConnection:InstConnection: init:01a0abc0 oracleHome:/u01/app/12.1.0/grid oracleSid:+ASM1 instanceType:2 instanceVersion:12.1.0.1.0
2019-10-20 08:44:06.720623 :CLSDYNAM:213481216: [ ora.asm]{0:0:4140} [clean] clsnInstConnection::makeConnectStr UsrOraEnv m_oracleHome /u01/app/12.1.0/grid Crshome /u01/app/12.1.0/grid
2019-10-20 08:44:06.720640 :CLSDYNAM:213481216: [ ora.asm]{0:0:4140} [clean] makeConnectStr = (DESCRIPTION=(ADDRESS=(PROTOCOL=beq)(PROGRAM=/u01/app/12.1.0/grid/bin/oracle)(ARGV0=oracle+ASM1)(ENVS='ORACLE_HOME=/u01/app/12.1.0/grid,ORACLE_SID=+ASM1')(ARGS='(DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))')(CONNECT_DATA=(SID=+ASM1))))
2019-10-20 08:44:06.721081 :CLSDYNAM:213481216: [ ora.asm]{0:0:4140} [clean] InstAgent::stop non pool pConnxn 1 01a0abc0
2019-10-20 08:44:06.721110 :CLSDYNAM:213481216: [ ora.asm]{0:0:4140} [clean] InstConnection::connectInt: server not attached
2019-10-20 08:44:07.748552 :CLSDYNAM:213481216: [ ora.asm]{0:0:4140} [clean] InstConnection:connectInt connected
2019-10-20 08:44:07.749744 :CLSDYNAM:213481216: [ ora.asm]{0:0:4140} [clean] InstConnection::shutdown mode 4
2019-10-20 08:44:07.750553 :CLSDYNAM:213481216: [ ora.asm]{0:0:4140} [clean] ConnectionPool::removeConnection connection count 1
2019-10-20 08:44:07.750573 :CLSDYNAM:213481216: [ ora.asm]{0:0:4140} [clean] ConnectionPool::removeConnection sid +ASM1, InstConnection 01a0abc0
2019-10-20 08:44:07.750588 : USRTHRD:213481216: {0:0:4140} InstConnection::breakCall pConnxn:01a0abc0 DetachLock:00f31fe8 m_pSrvH:0190eff0, m_pSvcH:019cd6a8
2019-10-20 08:44:07.750610 : USRTHRD:213481216: {0:0:4140} InstConnection:~InstConnection: this 01a0abc0
2019-10-20 08:44:07.751230 :CLSDYNAM:213481216: [ ora.asm]{0:0:4140} [clean] ConnectionPool::removeConnection delete InstConnection 01a0abc0
2019-10-20 08:44:07.751261 :CLSDYNAM:213481216: [ ora.asm]{0:0:4140} [clean] ConnectionPool::removeConnection freed 1
2019-10-20 08:44:07.751282 :CLSDYNAM:213481216: [ ora.asm]{0:0:4140} [clean] ConnectionPool::stopConnection
2019-10-20 08:44:07.751309 :CLSDYNAM:213481216: [ ora.asm]{0:0:4140} [clean] ConnectionPool::removeConnection connection count 0
2019-10-20 08:44:07.751321 :CLSDYNAM:213481216: [ ora.asm]{0:0:4140} [clean] ConnectionPool::removeConnection freed 0
2019-10-20 08:44:07.751332 :CLSDYNAM:213481216: [ ora.asm]{0:0:4140} [clean] ConnectionPool::stopConnection sid +ASM1 status 1
2019-10-20 08:44:07.751350 :CLSDYNAM:213481216: [ ora.asm]{0:0:4140} [clean] InstAgent::stop db/asm
2019-10-20 08:44:07.751360 :CLSDYNAM:213481216: [ ora.asm]{0:0:4140} [clean] ConnectionPool::stopConnection
2019-10-20 08:44:07.751379 :CLSDYNAM:213481216: [ ora.asm]{0:0:4140} [clean] ConnectionPool::removeConnection connection count 0
2019-10-20 08:44:07.751390 :CLSDYNAM:213481216: [ ora.asm]{0:0:4140} [clean] ConnectionPool::removeConnection freed 0
2019-10-20 08:44:07.751400 :CLSDYNAM:213481216: [ ora.asm]{0:0:4140} [clean] ConnectionPool::stopConnection sid +ASM1 status 1
2019-10-20 08:44:07.751568 :CLSDYNAM:213481216: [ ora.asm]{0:0:4140} [clean] getResAttrib: attrib name USR_ORA_OPI value false len 5
2019-10-20 08:44:07.751607 :CLSDYNAM:213481216: [ ora.asm]{0:0:4140} [clean] InstAgent::stop Container->stop, isShutdown = 0
2019-10-20 08:44:07.751667 :CLSDYNAM:213481216: [ ora.asm]{0:0:4140} [clean] InstAgent::stop calling initCssData
2019-10-20 08:44:07.751698 :CLSDYNAM:213481216: [ ora.asm]{0:0:4140} [clean] CssData Constructor groupName:HB+ASM cssRegistrationType:0 memno:-1
2019-10-20 08:44:07.751715 : USRTHRD:213481216: {0:0:4140} CssData::initMutex &m_cssDataLock:01ad8978
2019-10-20 08:44:07.758994 :CLSDYNAM:213481216: [ ora.asm]{0:0:4140} [clean] CssGroup::regis registration of groupName:HB+ASM mbrid:-1 m_grpNum:-1 regType:0 succeeded
2019-10-20 08:44:07.759036 :CLSDYNAM:213481216: [ ora.asm]{0:0:4140} [clean] CssData::Constructor regis succeeded groupName:HB+ASM memno:-1
2019-10-20 08:44:07.759050 :CLSDYNAM:213481216: [ ora.asm]{0:0:4140} [clean] CssData::Constructor m_pCssGroup:019d2250 group HB+ASM
2019-10-20 08:44:07.759062 :CLSDYNAM:213481216: [ ora.asm]{0:0:4140} [clean] InstAgent::init m_groupName:HB+ASM m_pCssData 01ad68b0
2019-10-20 08:44:07.759076 : USRTHRD:213481216: {0:0:4140} CssData Destructor groupName:HB+ASM
2019-10-20 08:44:07.759090 : USRTHRD:213481216: {0:0:4140} CssGroup::dereg default regType:0
2019-10-20 08:44:07.765243 : USRTHRD:213481216: {0:0:4140} CssData::destroyMutex &m_cssDataLock:01ad8978
2019-10-20 08:44:07.765264 : USRTHRD:213481216: {0:0:4140} CssData::Destructor
2019-10-20 08:44:07.765287 :CLSDYNAM:213481216: [ ora.asm]{0:0:4140} [clean] InstAgent::stop prev clsagfw_res_status 2 current clsagfw_res_status 2
2019-10-20 08:44:07.765301 :CLSDYNAM:213481216: [ ora.asm]{0:0:4140} [clean] InstAgent::stop m_prevState:2 }
2019-10-20 08:44:07.765312 :CLSDYNAM:213481216: [ ora.asm]{0:0:4140} [clean] clean }
2019-10-20 08:44:07.765382 :CLSDYNAM:213481216: [ ora.asm]{0:0:4140} [clean] (:CLSN00106:) clsn_agent::clean }
2019-10-20 08:44:07.765403 : AGFW:213481216: {0:0:4140} Command: clean for resource: ora.asm 1 1 completed with status: SUCCESS
2019-10-20 08:44:07.766035 : AGFW:233195264: {0:0:4140} Agent sending reply for: RESOURCE_CLEAN[ora.asm 1 1] ID 4100:18937
2019-10-20 08:44:07.769116 :CLSDYNAM:217683712: [ ora.asm]{0:0:4140} [check] InstAgent::checkState gimh oracleSid:+ASM1
2019-10-20 08:44:07.769143 :CLSDYNAM:217683712: [ ora.asm]{0:0:4140} [check] Gimh::constructor ohome:/u01/app/12.1.0/grid sid:+ASM1
2019-10-20 08:44:07.769213 :CLSDYNAM:217683712: [ ora.asm]{0:0:4140} [check] ConnectionPool::resetConnection s_statusOfConnectionMap 00f3aed8
2019-10-20 08:44:07.769225 :CLSDYNAM:217683712: [ ora.asm]{0:0:4140} [check] ConnectionPool::resetConnection sid +ASM1 status 2
2019-10-20 08:44:07.769342 :CLSDYNAM:217683712: [ ora.asm]{0:0:4140} [check] Gimh::check OH /u01/app/12.1.0/grid SID +ASM1
2019-10-20 08:44:07.769422 :CLSDYNAM:217683712: [ ora.asm]{0:0:4140} [check] Gimh::check condition changes to (GIMH_NEXT_NUM) 0(Abnormal Termination),1(Instance Shutdown) exists
2019-10-20 08:44:07.769436 :CLSDYNAM:217683712: [ ora.asm]{0:0:4140} [check] (:CLSN00006:)AsmAgent::gimhChecks failed gimh state 0
2019-10-20 08:44:07.781140 :CLSDYNAM:217683712: [ ora.asm]{0:0:4140} [check] checkCrsStat 2 CLSCRS_STAT ret: 184
2019-10-20 08:44:07.781163 :CLSDYNAM:217683712: [ ora.asm]{0:0:4140} [check] clsnUtils::error Exception type=2 string=
2019-10-20 08:44:07.781177 :CLSDYNAM:217683712: [ ora.asm]{0:0:4140} [check] checkCrsStat 2 status:0 ret:184 usererror:184
2019-10-20 08:44:07.782906 :CLSDYNAM:217683712: [ ora.asm]{0:0:4140} [check] AsmAgent::checkCbk: Exception UserErrorException
2019-10-20 08:44:07.782929 :CLSDYNAM:217683712: [ ora.asm]{0:0:4140} [check]
2019-10-20 08:44:07.783943 : USRTHRD:217683712: {0:0:4140} Gimh::destructor gimh_dest_query_ctx rc=0
2019-10-20 08:44:07.784030 : USRTHRD:217683712: {0:0:4140} Gimh::destructor gimh_dest_inst_ctx rc=0
2019-10-20 08:44:07.784085 :CLSDYNAM:217683712: [ ora.asm]{0:0:4140} [check] ConnectionPool::stopConnection
2019-10-20 08:44:07.784112 :CLSDYNAM:217683712: [ ora.asm]{0:0:4140} [check] ConnectionPool::removeConnection connection count 0
2019-10-20 08:44:07.784126 :CLSDYNAM:217683712: [ ora.asm]{0:0:4140} [check] ConnectionPool::removeConnection freed 0
2019-10-20 08:44:07.784136 :CLSDYNAM:217683712: [ ora.asm]{0:0:4140} [check] ConnectionPool::stopConnection sid +ASM1 status 1
2019-10-20 08:44:07.784710 : AGFW:233195264: {0:0:4140} ora.asm 1 1 state changed from: CLEANING to: PLANNED_OFFLINE
2019-10-20 08:44:07.784818 : AGFW:233195264: {0:0:4140} Agent sending last reply for: RESOURCE_CLEAN[ora.asm 1 1] ID 4100:18937
2019-10-20 08:44:19.570364 :CLSDYNAM:230291200: [ora.gipcd]{0:21:2} [check] ClsdmClient::sendMessage clsdmc_respget return: status=0, ecode=0
2019-10-20 08:44:19.571806 :CLSDYNAM:235296512: [ora.gpnpd]{0:21:2} [check] ClsdmClient::sendMessage clsdmc_respget return: status=0, ecode=0
2019-10-20 08:44:19.572189 : CLSDMC:213481216: command 0 failed with status 1
2019-10-20 08:44:19.572219 :CLSDYNAM:213481216: [ora.evmd]{0:21:2} [check] ClsdmClient::sendMessage clsdmc_respget return: status=0, ecode=1
2019-10-20 08:44:19.572235 :CLSDYNAM:213481216: [ora.evmd]{0:21:2} [check] DaemonAgent::check returned 0
2019-10-20 08:44:19.572246 :CLSDYNAM:213481216: [ora.evmd]{0:21:2} [check] Deep check returned 1
2019-10-20 08:44:19.595032 : AGFW:250087008: Recvd request to shed the threads
2019-10-20 08:44:19.595053 :CLSFRAME:250087008: TM [MultiThread] is changing desired thread # to 5. Current # is 6
2019-10-20 08:44:19.595099 :CLSFRAME:217683712: {0:21:59} Worker thread is exiting in TM [MultiThread] to meet the desired count of 5. New count is 5
2019-10-20 08:44:19.597760 :CLSDYNAM:228189952: [ora.mdnsd]{0:21:2} [check] ClsdmClient::sendMessage clsdmc_respget return: status=0, ecode=0
2019-10-20 08:44:49.440826 : AGFW:233195264: {0:0:4016} Agent received the message: AGENT_HB[Engine] ID 12293:18952
2019-10-20 08:44:49.576007 :CLSDYNAM:230291200: [ora.gpnpd]{0:21:2} [check] ClsdmClient::sendMessage clsdmc_respget return: status=0, ecode=0
2019-10