Hi All,
I have noticed in my 2 node Oracle RAC set on Rhel 7.2 over ASM . Out of 2 instances created in single cluster databse ,one instance are shutdown automatically .
[oracle@rhelrac2 logs]$ srvctl status database -d racdb1 -v
Instance racdb11 is running on node rhelrac2. Instance status: Open.
Instance racdb12 is running on node rhelrac1. Instance status: Open.
After some time the same command shows 1 instance down.
[oracle@rhelrac2 ~]$ srvctl status database -d racdb1 -v
Instance racdb11 is running on node rhelrac2. Instance status: Open.
Instance racdb12 is not running on node rhelrac1
[oracle@rhelrac1 oracle]$ srvctl config database -d racdb1
Database unique name: racdb1
Database name: racdb1
Oracle home: /u01/app/oracle/product/12.1.0/dbhome_1
Oracle user: oracle
Spfile: +DBDISK/RACDB1/PARAMETERFILE/spfile.276.929416381
Password file: +DBDISK/RACDB1/PASSWORD/pwdracdb1.256.929416039
Domain:
Start options: open
Stop options: immediate
Database role: PRIMARY
Management policy: AUTOMATIC
Server pools:
Disk Groups: DBDISK
Mount point paths:
Services:
Type: RAC
Start concurrency:
Stop concurrency:
OSDBA group: oinstall
OSOPER group: oper
Database instances: racdb11,racdb12
Configured nodes: rhelrac2,rhelrac1
Database is administrator managed
When i check alert log of the instance i see this errors reported.
Thu Jan 05 04:05:58 2017
Errors in file /u01/app/oracle/diag/rdbms/racdb1/racdb12/trace/racdb12_j000_6003.trc:
ORA-27157: OS post/wait facility removed
ORA-27300: OS system dependent operation:semop failed with status: 43
ORA-27301: OS failure message: Identifier removed
ORA-27302: failure occurred at: sskgpwwait1
Thu Jan 05 04:05:58 2017
Errors in file /u01/app/oracle/diag/rdbms/racdb1/racdb12/trace/racdb12_rmv0_5761.trc:
ORA-27157: OS post/wait facility removed
Thu Jan 05 04:05:58 2017
Errors in file /u01/app/oracle/diag/rdbms/racdb1/racdb12/trace/racdb12_gcr0_5759.trc:
ORA-27157: OS post/wait facility removed
ORA-27300: OS system dependent operation:semop failed with status: 43
ORA-27301: OS failure message: Identifier removed
ORA-27302: failure occurred at: sskgpwwait1
Thu Jan 05 04:05:58 2017
Errors in file /u01/app/oracle/diag/rdbms/racdb1/racdb12/trace/racdb12_qm02_5981.trc:
ORA-27157: OS post/wait facility removed
ORA-27300: OS system dependent operation:semop failed with status: 43
ORA-27301: OS failure message: Identifier removed
ORA-27302: failure occurred at: sskgpwwait1
Thu Jan 05 04:05:58 2017
Errors in file /u01/app/oracle/diag/rdbms/racdb1/racdb12/trace/racdb12_q004_5989.trc:
ORA-27157: OS post/wait facility removed
ORA-27300: OS system dependent operation:semop failed with status: 43
ORA-27301: OS failure message: Identifier removed
ORA-27302: failure occurred at: sskgpwwait1
Thu Jan 05 04:05:58 2017
Errors in file /u01/app/oracle/diag/rdbms/racdb1/racdb12/trace/racdb12_q005_6269.trc:
ORA-27157: OS post/wait facility removed
ORA-27300: OS system dependent operation:semop failed with status: 43
ORA-27301: OS failure message: Identifier removed
ORA-27302: failure occurred at: sskgpwwait1
Thu Jan 05 04:05:58 2017
System state dump requested by (instance=2, osid=5609 (PMON)), summary=[abnormal instance termination].
System State dumped to trace file /u01/app/oracle/diag/rdbms/racdb1/racdb12/trace/racdb12_diag_5668_20170105040558.trc
Thu Jan 05 04:05:58 2017
USER (ospid: 5609): terminating the instance due to error 27157
Thu Jan 05 04:06:01 2017
Dumping diagnostic data in directory=[cdmp_20170105040558], requested by (instance=2, osid=5609 (PMON)), summary=[abnormal instance termination].
Thu Jan 05 04:06:01 2017
License high water mark = 12
Thu Jan 05 04:06:01 2017
Instance terminated by USER, pid = 5609
Thu Jan 05 04:06:01 2017
[root@rhelrac2 ~]# crsctl stat res -t
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.ARCDISK.dg
ONLINE ONLINE rhelrac1 STABLE
ONLINE ONLINE rhelrac2 STABLE
ora.DATA.dg
ONLINE ONLINE rhelrac1 STABLE
ONLINE ONLINE rhelrac2 STABLE
ora.DBDISK.dg
ONLINE ONLINE rhelrac1 STABLE
ONLINE ONLINE rhelrac2 STABLE
ora.LISTENER.lsnr
ONLINE ONLINE rhelrac1 STABLE
ONLINE ONLINE rhelrac2 STABLE
ora.asm
ONLINE ONLINE rhelrac1 Started,STABLE
ONLINE ONLINE rhelrac2 Started,STABLE
ora.net1.network
ONLINE ONLINE rhelrac1 STABLE
ONLINE ONLINE rhelrac2 STABLE
ora.ons
ONLINE ONLINE rhelrac1 STABLE
ONLINE ONLINE rhelrac2 STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE rhelrac2 STABLE
ora.LISTENER_SCAN2.lsnr
1 ONLINE ONLINE rhelrac2 STABLE
ora.MGMTLSNR
1 ONLINE ONLINE rhelrac1 169.254.225.104,STAB
LE
ora.cvu
1 ONLINE ONLINE rhelrac1 STABLE
ora.mgmtdb
1 OFFLINE OFFLINE STABLE
ora.oc4j
1 ONLINE ONLINE rhelrac1 STABLE
ora.racdb1.db
1 ONLINE ONLINE rhelrac2 Open,STABLE
2 ONLINE ONLINE rhelrac1 Open,STABLE
ora.rhelrac1.vip
1 ONLINE ONLINE rhelrac1 STABLE
ora.rhelrac2.vip
1 ONLINE ONLINE rhelrac2 STABLE
ora.scan1.vip
1 ONLINE ONLINE rhelrac2 STABLE
ora.scan2.vip
1 ONLINE ONLINE rhelrac2 STABLE
--------------------------------------------------------------------------------
Please help in getting over this issue
Thanks,
Mallik S