Hi,
We have rac server at site. yesterday morning suddenly application got hanged. we were not able to perform any transactions on tables.
When i checked the cluster services at node 2 , the ons service of node 1 was done. and we were unable to get the ./crs_stat -t output on node1. It jst got hanged.when we tried to stop the cluster services it was not getting stop. So we rebooted the machine.
After restarting the node1 and analyzing tables in the database. problem got resolved.Now we need to submit RCA this prob...
Alert log file of node1:-
Thu Sep 16 05:15:09 2010
WAITED TOO LONG FOR A ROW CACHE ENQUEUE LOCK! pid=55
System State dumped to trace file /home/database/admin/crbt/udump/crbt1_ora_1897
5.trc
Thu Sep 16 05:18:13 2010
WAITED TOO LONG FOR A ROW CACHE ENQUEUE LOCK! pid=51
System State dumped to trace file /home/database/admin/crbt/udump/crbt1_ora_2038
2.trc
Thu Sep 16 05:58:30 2010
Unable to restore resource manager plan to '':
ORA-02097: parameter cannot be modified because specified value is invalid
ORA-00439: feature not enabled: Database resource manager
Thu Sep 16 06:52:38 2010
GES: Potential blocker (pid=7918) on resource TM-0x2390-0x0;
enqueue info in file /home/database/admin/crbt/bdump/crbt1_lmd0_8151.trc and DI
AG trace file
Thu Sep 16 07:31:36 2010
GES: Potential blocker (pid=27831) on resource TM-0xcb1e-0x0;
More
GES: Potential blocker (pid=27831) on resource TM-0xcb1e-0x0;
enqueue info in file /home/database/admin/crbt/udump/crbt1_ora_4760.trc and DIA
G trace file
Thu Sep 16 07:43:20 2010
WAITED TOO LONG FOR A ROW CACHE ENQUEUE LOCK! pid=70
System State dumped to trace file /home/database/admin/crbt/udump/crbt1_ora_2839
2.trc
Thu Sep 16 07:46:03 2010
GES: Potential blocker (pid=27831) on resource TM-0xcb1e-0x0;
enqueue info in file /home/database/admin/crbt/bdump/crbt1_lmd0_8151.trc and DI
AG trace file
Thu Sep 16 08:16:18 2010
GES: Potential blocker (pid=27831) on resource TM-0xcb1e-0x0;
enqueue info in file /home/database/admin/crbt/udump/crbt1_ora_5132.trc and DIA
G trace file
Thu Sep 16 08:20:28 2010
GES: Potential blocker (pid=27831) on resource TM-0xcb1e-0x0;
enqueue info in file /home/database/admin/crbt/udump/crbt1_ora_11791.trc and DI
AG trace file
Thu Sep 16 08:23:48 2010
GES: Potential blocker (pid=27831) on resource TM-0xcb1e-0x0;
enqueue info in file /home/database/admin/crbt/udump/crbt1_ora_17000.trc and DI
AG trace file
Thu Sep 16 08:26:30 2010
GES: Potential blocker (pid=27831) on resource TM-0xcb1e-0x0;
enqueue info in file /home/database/admin/crbt/udump/crbt1_ora_21500.trc and DI
AG trace file
Thu Sep 16 08:28:40 2010
GES: Potential blocker (pid=27831) on resource TM-0xcb1e-0x0;
enqueue info in file /home/database/admin/crbt/bdump/crbt1_lmd0_8151.trc and DI
AG trace file
Thu Sep 16 08:30:53 2010
GES: Potential blocker (pid=27831) on resource TM-0xcb1e-0x0;
enqueue info in file /home/database/admin/crbt/bdump/crbt1_lmd0_8151.trc and DI
AG trace file
Thu Sep 16 09:34:25 2010
GES: Potential blocker (pid=8349) on resource TM-0xcb5b-0x0;
enqueue info in file /home/database/admin/crbt/bdump/crbt1_lmd0_8151.trc and DI
AG trace file
Thu Sep 16 09:46:53 2010
Errors in file /home/database/admin/crbt/bdump/crbt1_diag_27855.trc:
ORA-00600: internal error code, arguments: [kjzcreaprqhq1], [], [], [], [], [],
[], []
Thu Sep 16 09:47:50 2010
Thu Sep 16 09:47:50 2010
Restarting dead background process DIAG
DIAG started with pid=3, OS id=14337
Thu Sep 16 10:19:26 2010
GES: Potential blocker (pid=8349) on resource TM-0xcb5b-0x0;
enqueue info in file /home/database/admin/crbt/bdump/crbt1_lmd0_8151.trc and DI
AG trace file
Thu Sep 16 10:41:17 2010
GES: Potential blocker (pid=20583) on resource TX-0x70014-0x1663b4;
enqueue info in file /home/database/admin/crbt/udump/crbt1_ora_17000.trc and DI
AG trace file
Thu Sep 16 10:41:38 2010
GES: Potential blocker (pid=17000) on resource TX-0x18000b-0xb;
enqueue info in file /home/database/admin/crbt/udump/crbt1_ora_21500.trc and DI
AG trace file
Thu Sep 16 10:50:51 2010
GES: Potential blocker (pid=17000) on resource TX-0x18000b-0xb;
enqueue info in file /home/database/admin/crbt/bdump/crbt1_lmd0_8151.trc and DI
AG trace file
Thu Sep 16 11:10:43 2010
GES: Potential blocker (pid=11163) on resource LB-0xe3034538-0x7c7e2e10;
enqueue info in file /home/database/admin/crbt/bdump/crbt1_lmd0_8151.trc and DI
AG trace file
Thu Sep 16 13:40:39 2010
GES: Potential blocker (pid=11163) on resource LB-0xe3034538-0x7c7e2e10;
enqueue info in file /home/database/admin/crbt/udump/crbt1_ora_4121.trc and DIA
G trace file
Thu Sep 16 13:55:47 2010
GES: Potential blocker (pid=29139) on resource TM-0x2390-0x0;
enqueue info in file /home/database/admin/crbt/bdump/crbt1_lmd0_8151.trc and DI
AG trace file
Thu Sep 16 14:54:38 2010
Reconfiguration started (old inc 38, new inc 40)
List of nodes:
0
Global Resource Directory frozen
* dead instance detected - domain 0 invalid = TRUE
Communication channels reestablished
Master broadcasted resource hash value bitmaps
Non-local Process blocks cleaned out
Thu Sep 16 14:54:38 2010
LMS 0: 0 GCS shadows cancelled, 0 closed
Thu Sep 16 14:54:38 2010
LMS 1: 0 GCS shadows cancelled, 0 closed
Set master node info
Submitted all remote-enqueue requests
Dwn-cvts replayed, VALBLKs dubious
All grantable enqueues granted
Post SMON to start 1st pass IR
Thu Sep 16 14:54:39 2010
LMS 0: 59246 GCS shadows traversed, 0 replayed
Thu Sep 16 14:54:39 2010
LMS 1: 60169 GCS shadows traversed, 0 replayed
Thu Sep 16 14:54:39 2010
Submitted all GCS remote-cache requests
Post SMON to start 1st pass IR
Fix write in gcs resources
Thu Sep 16 14:54:39 2010
Instance recovery: looking for dead threads
Instance recovery: lock domain invalid but no dead threads
Reconfiguration complete
Thu Sep 16 15:15:32 2010
Starting background process EMN0
Thu Sep 16 15:17:30 2010
ERROR: Emon failed to start.
Shutting down instance: further logons disabled
Thu Sep 16 15:17:30 2010
Stopping background process QMNC
Thu Sep 16 15:17:30 2010
Stopping background process CJQ0
Thu Sep 16 15:17:30 2010
Errors in file /home/database/admin/crbt/bdump/crbt1_pmon_8143.trc:
ORA-00443: background process "EMN0" did not start
Thu Sep 16 15:17:30 2010
Errors in file /home/database/admin/crbt/bdump/crbt1_o001_29846.trc:
ORA-00000: normal, successful completion
Thu Sep 16 15:17:32 2010
Stopping background process MMNL
Thu Sep 16 15:19:34 2010
Stopping background process MMON
Thu Sep 16 15:19:35 2010
Errors in file /home/database/admin/crbt/udump/crbt1_ora_18628.trc:
ORA-00604: error occurred at recursive SQL level 1
ORA-12663: Services required by client not available on the server
ORA-36961: Oracle OLAP is not available.
ORA-06512: at "SYS.OLAPIHISTORYRETENTION", line 1
ORA-06512: at line 15
Thu Sep 16 15:19:35 2010
Shutting down instance (immediate)
License high water mark = 195
Thu Sep 16 15:19:35 2010
Stopping Job queue slave processes
Thu Sep 16 15:19:35 2010
Job queue slave processes stopped
Thu Sep 16 15:24:34 2010
Active call for process 11163 user 'oracle' program 'oracle@crbtnode1 (TNS V1-V3
)'
Active call for process 29566 user 'oracle' program 'oraclecrbt1@crbtnode1'
Active call for process 11661 user 'oracle' program 'oraclecrbt1@crbtnode1'
Active call for process 23079 user 'oracle' program 'oracle@crbtnode1 (TNS V1-V3
)'
Active call for process 8349 user 'oracle' program 'oraclecrbt1@crbtnode1'
SHUTDOWN: waiting for active calls to complete.
Thu Sep 16 15:25:42 2010
Trace dumping is performing id=[cdmp_20100916152542]
All dispatchers and shared servers shutdown
Thu Sep 16 15:25:48 2010
Errors in file /home/database/admin/crbt/bdump/crbt1_ckpt_8179.trc:
ORA-27091: unable to queue I/O
ORA-27072: File I/O error
Linux Error: 5: Input/output error
Additional information: 4
Additional information: 1720416
Additional information: -1
Thu Sep 16 15:25:49 2010
Errors in file /home/database/admin/crbt/bdump/crbt1_asmb_8276.trc:
ORA-15064: communication failure with ASM instance
ORA-03113: end-of-file on communication channel
Thu Sep 16 15:25:49 2010
ASMB: terminating instance due to error 15064
Thu Sep 16 15:25:49 2010
System state dump is made for local instance
System State dumped to trace file /home/database/admin/crbt/bdump/crbt1_diag_143
37.trc
Thu Sep 16 15:25:50 2010
Trace dumping is performing id=[cdmp_20100916152549]
Thu Sep 16 15:31:26 2010
Starting ORACLE instance (normal)
LICENSE_MAX_SESSION = 0
Feel free to ask for any inputs, help would be appreciated.