Hi,
Oracle Version:11gR2
Operating System:Cent Os
Hi we have some problem in node 1 after sudden reboot of both the nodes when the servers are up the database in node 2 started automatically but in node 1 we started manually.
But inthe CRSCTL command it is showing that node 1 database is down as show below.
[root@rac1 bin]# ./crsctl stat res -t
--------------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.ASM_DATA.dg
ONLINE ONLINE rac1
ONLINE ONLINE rac2
ora.ASM_FRA.dg
ONLINE ONLINE rac1
ONLINE ONLINE rac2
ora.LISTENER.lsnr
ONLINE ONLINE rac1
ONLINE ONLINE rac2
ora.OCR_VOTE.dg
ONLINE ONLINE rac1
ONLINE ONLINE rac2
ora.asm
ONLINE ONLINE rac1 Started
ONLINE ONLINE rac2 Started
ora.eons
ONLINE ONLINE rac1
ONLINE ONLINE rac2
ora.gsd
OFFLINE OFFLINE rac1
OFFLINE OFFLINE rac2
ora.net1.network
ONLINE ONLINE rac1
ONLINE ONLINE rac2
ora.ons
ONLINE ONLINE rac1
ONLINE ONLINE rac2
ora.registry.acfs
ONLINE ONLINE rac1
ONLINE ONLINE rac2
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE rac2
ora.LISTENER_SCAN2.lsnr
1 ONLINE ONLINE rac1
ora.LISTENER_SCAN3.lsnr
1 ONLINE ONLINE rac1
ora.oc4j
1 OFFLINE OFFLINE
ora.qfundrac.db
1 OFFLINE OFFLINE
2 ONLINE ONLINE rac2 Open
ora.rac1.vip
1 ONLINE ONLINE rac1
ora.rac2.vip
1 ONLINE ONLINE rac2
ora.scan1.vip
1 ONLINE ONLINE rac2
ora.scan2.vip
1 ONLINE ONLINE rac1
ora.scan3.vip
1 ONLINE ONLINE rac1
but for the below command it is showing both the nodes are up
SQL> select inst_id,status,instance_role,active_state from gv$instance;
INST_ID STATUS INSTANCE_ROLE ACTIVE_ST
---------- ------------ ------------------ ---------
1 OPEN PRIMARY_INSTANCE NORMAL
2 OPEN PRIMARY_INSTANCE NORMAL
here is the output for cluvfy .
[grid@rac1 bin]$ ./cluvfy stage -post crsinst -n rac1,rac2 -verbose
Performing post-checks for cluster services setup
Checking node reachability...
Check: Node reachability from node "rac1"
Destination Node Reachable?
------------------------------------ ------------------------
rac2 yes
rac1 yes
Result: Node reachability check passed from node "rac1"
Checking user equivalence...
Check: User equivalence for user "grid"
Node Name Comment
------------------------------------ ------------------------
rac2 passed
rac1 passed
Result: User equivalence check passed for user "grid"
Checking time zone consistency...
Time zone consistency check passed.
Checking Cluster manager integrity...
Checking CSS daemon...
Node Name Status
------------------------------------ ------------------------
rac2 running
rac1 running
Oracle Cluster Synchronization Services appear to be online.
Cluster manager integrity check passed
UDev attributes check for OCR locations started...
Result: UDev attributes check passed for OCR locations
UDev attributes check for Voting Disk locations started...
Result: UDev attributes check passed for Voting Disk locations
Check default user file creation mask
Node Name Available Required Comment
------------ ------------------------ ------------------------ ----------
rac2 0022 0022 passed
rac1 0022 0022 passed
Result: Default user file creation mask check passed
Checking cluster integrity...
Node Name
------------------------------------
rac1
rac2
Cluster integrity check passed
Checking OCR integrity...
Checking the absence of a non-clustered configuration...
All nodes free of non-clustered, local-only configurations
ASM Running check passed. ASM is running on all cluster nodes
Checking OCR config file "/etc/oracle/ocr.loc"...
OCR config file "/etc/oracle/ocr.loc" check successful
Disk group for ocr location "+OCR_VOTE" available on all the nodes
Checking size of the OCR location "+OCR_VOTE" ...
Size check for OCR location "+OCR_VOTE" successful...
Size check for OCR location "+OCR_VOTE" successful...
WARNING:
This check does not verify the integrity of the OCR contents. Execute 'ocrcheck' as a privileged user to verify the contents of OCR.
OCR integrity check passed
Checking CRS integrity...
The Oracle clusterware is healthy on node "rac2"
The Oracle clusterware is healthy on node "rac1"
CRS integrity check passed
Checking node application existence...
Checking existence of VIP node application
Node Name Required Status Comment
------------ ------------------------ ------------------------ ----------
rac2 yes online passed
rac1 yes online passed
Result: Check passed.
Checking existence of ONS node application
Node Name Required Status Comment
------------ ------------------------ ------------------------ ----------
rac2 no online passed
rac1 no online passed
Result: Check passed.
Checking existence of GSD node application
Node Name Required Status Comment
------------ ------------------------ ------------------------ ----------
rac2 no does not exist ignored
rac1 no does not exist ignored
Result: Check ignored.
Checking existence of EONS node application
Node Name Required Status Comment
------------ ------------------------ ------------------------ ----------
rac2 no online passed
rac1 no online passed
Result: Check passed.
Checking existence of NETWORK node application
Node Name Required Status Comment
------------ ------------------------ ------------------------ ----------
rac2 no online passed
rac1 no online passed
Result: Check passed.
Checking Single Client Access Name (SCAN)...
SCAN VIP name Node Running? ListenerName Port Running?
---------------- ------------ ------------ ------------ ------------ ------------
qfund-rac.qfund.net rac2 true LISTENER 1521 true
Checking name resolution setup for "qfund-rac.qfund.net"...
SCAN Name IP Address Status Comment
------------ ------------------------ ------------------------ ----------
qfund-rac.qfund.net 192.168.8.118 passed
qfund-rac.qfund.net 192.168.8.119 passed
qfund-rac.qfund.net 192.168.8.117 passed
Verification of SCAN VIP and Listener setup passed
OCR detected on ASM. Running ACFS Integrity checks...
Starting check to see if ASM is running on all cluster nodes...
ASM Running check passed. ASM is running on all cluster nodes
Starting Disk Groups check to see if at least one Disk Group configured...
Disk Group Check passed. At least one Disk Group configured
Task ACFS Integrity check passed
Checking Oracle Cluster Voting Disk configuration...
Oracle Cluster Voting Disk configuration check passed
Checking to make sure user "grid" is not in "root" group
Node Name Status Comment
------------ ------------------------ ------------------------
rac2 does not exist passed
rac1 does not exist passed
Result: User "grid" is not part of "root" group. Check passed
Checking if Clusterware is installed on all nodes...
Check of Clusterware install passed
Checking if CTSS Resource is running on all nodes...
Check: CTSS Resource running on all nodes
Node Name Status
------------------------------------ ------------------------
rac2 passed
rac1 passed
Result: CTSS resource check passed
Querying CTSS for time offset on all nodes...
Result: Query of CTSS for time offset passed
Check CTSS state started...
Check: CTSS state
Node Name State
------------------------------------ ------------------------
rac2 Observer
rac1 Observer
CTSS is in Observer state. Switching over to clock synchronization checks using NTP
Starting Clock synchronization checks using Network Time Protocol(NTP)...
NTP Configuration file check started...
The NTP configuration file "/etc/ntp.conf" is available on all nodes
NTP Configuration file check passed
Checking daemon liveness...
Check: Liveness for "ntpd"
Node Name Running?
------------------------------------ ------------------------
rac2 yes
rac1 yes
Result: Liveness check passed for "ntpd"
Checking NTP daemon command line for slewing option "-x"
Check: NTP daemon command line
Node Name Slewing Option Set?
------------------------------------ ------------------------
rac2 yes
rac1 yes
Result:
NTP daemon slewing option check passed
Checking NTP daemon's boot time configuration, in file "/etc/sysconfig/ntpd", for slewing option "-x"
Check: NTP daemon's boot time configuration
Node Name Slewing Option Set?
------------------------------------ ------------------------
rac2 yes
rac1 yes
Result:
NTP daemon's boot time configuration check for slewing option passed
NTP common Time Server Check started...
NTP Time Server ".INIT." is common to all nodes on which the NTP daemon is running
NTP Time Server ".LOCL." is common to all nodes on which the NTP daemon is running
Check of common NTP Time Server passed
Clock time offset check from NTP Time Server started...
Checking on nodes "[rac2, rac1]"...
Check: Clock time offset from NTP Time Server
Time Server: .INIT.
Time Offset Limit: 1000.0 msecs
Node Name Time Offset Status
------------ ------------------------ ------------------------
rac2 0.0 passed
rac1 0.0 passed
Time Server ".INIT." has time offsets that are within permissible limits for nodes "[rac2, rac1]".
Time Server: .LOCL.
Time Offset Limit: 1000.0 msecs
Node Name Time Offset Status
------------ ------------------------ ------------------------
rac2 -29.328 passed
rac1 -84.385 passed
Time Server ".LOCL." has time offsets that are within permissible limits for nodes "[rac2, rac1]".
Clock time offset check passed
Result: Clock synchronization check using Network Time Protocol(NTP) passed
Oracle Cluster Time Synchronization Services check passed
Post-check for cluster services setup was successful.
[grid@rac1 bin]$
Please help me how to solve this problem.
Thanks & regards
Poorna Prasad.S