Hi All,
I have a RAC database with 2 node instance. For last few days frequently the service ora.or10prd.or10prd.cs is going offline.
Below is the o/p of crs_stat -t.
$ crs_stat -t
Name Type Target State Host
------------------------------------------------------------
ora....SM1.asm application ONLINE ONLINE nmedbprdnd1
ora....D1.lsnr application ONLINE ONLINE nmedbprdnd1
ora....nd1.gsd application ONLINE ONLINE nmedbprdnd1
ora....nd1.ons application ONLINE ONLINE nmedbprdnd1
ora....nd1.vip application ONLINE ONLINE nmedbprdnd1
ora....SM2.asm application ONLINE ONLINE nmedbprdnd2
ora....D2.lsnr application ONLINE ONLINE nmedbprdnd2
ora....nd2.gsd application ONLINE ONLINE nmedbprdnd2
ora....nd2.ons application ONLINE ONLINE nmedbprdnd2
ora....nd2.vip application ONLINE ONLINE nmedbprdnd2
ora.or10prd.db application ONLINE ONLINE nmedbprdnd1
ora....0prd.cs application OFFLINE OFFLINE
ora....rd1.srv application ONLINE ONLINE nmedbprdnd1
ora....rd2.srv application ONLINE ONLINE nmedbprdnd2
ora....d1.inst application ONLINE ONLINE nmedbprdnd1
ora....d2.inst application ONLINE ONLINE nmedbprdnd2
It is starting without any issues when we tried to start the service manually. But automatically goes offline daily.
$ crs_start ora.or10prd.or10prd.cs
Cluster log from node2:
2015-02-26 06:00:29.888: [ OCRSRV][1273047392]th_select_handler: Failed to retrieve procctx from ht. constr = [-1741489328] retval lht [-27] Signal CV.
2015-02-26 06:21:27.996: [ OCRSRV][1273047392]th_select_handler: Failed to retrieve procctx from ht. constr = [-1741489328] retval lht [-27] Signal CV.
2015-02-26 06:21:28.039: [ OCRSRV][1273047392]th_select_handler: Failed to retrieve procctx from ht. constr = [-1741467008] retval lht [-27] Signal CV.
2015-02-26 06:46:42.137: [ OCRSRV][1273047392]th_select_handler: Failed to retrieve procctx from ht. constr = [-1741489328] retval lht [-27] Signal CV.
2015-02-26 06:58:27.082: [ CRSAPP][1556273504]0CheckResource error for ora.or10prd.or10prd.cs error code = 139
2015-02-26 06:58:27.084: [ CRSRES][1556273504]0In stateChanged, ora.or10prd.or10prd.cs target is ONLINE
2015-02-26 06:58:27.084: [ CRSRES][1556273504]0ora.or10prd.or10prd.cs on nmedbprdnd2 went OFFLINE unexpectedly
2015-02-26 06:58:27.084: [ CRSRES][1556273504]0StopResource: setting CLI values
2015-02-26 06:58:27.088: [ CRSRES][1556273504]0Attempting to stop `ora.or10prd.or10prd.cs` on member `nmedbprdnd2`
Cluster log from node1:
2015-02-26 08:39:04.927: [ CRSRES][1619212640]0Attempting to start `ora.or10prd.or10prd.cs` on member `nmedbprdnd2`
2015-02-26 08:39:04.978: [ CRSRES][1619212640]0Start of `ora.or10prd.or10prd.cs` on member `nmedbprdnd2` succeeded.
2015-02-26 08:39:04.984: [ COMMCRS][1472354656]clsc_receive: (0xc12a20) Lock release 1 failed, rc 2
2015-02-26 08:39:04.984: [ COMMCRS][1472354656]clsc_receive: (0xc12a20) error 2
2015-02-26 08:47:51.529: [ OCRSRV][1231087968]th_select_handler: Failed to retrieve procctx from ht. constr = [-1741634320] retval lht [-27] Signal CV.
2015-02-26 09:27:52.589: [ OCRSRV][1231087968]th_select_handler: Failed to retrieve procctx from ht. constr = [-1741634320] retval lht [-27] Signal CV.
2015-02-26 09:37:52.858: [ OCRSRV][1231087968]th_select_handler: Failed to retrieve procctx from ht. constr = [-1741634320] retval lht [-27] Signal CV.
Currently all services are online.
Database version 10.2.0.1 Enterprise edition
OS Version : Red Hat Enterprise Linux AS release 4
Thanks in advance.!