Oracle 10.2.0.3 on AIX 5L; single instance
The database was frozen; it was not possible to shutdown the DB anymore. Error messages are listed below.
This error now occured the second time within a month. What might be the cause? What should be check if the error show up the next time?
Any ideas appreciated.
Errors from alert_log:
Mon Jul 07 13:49:00 2008
GES: Potential blocker (pid=1101878) on resource CF-00000000-00000000;
enqueue info in file /u01/oracle/stdr/DGHFLR/bdump/DGHFLR_lgwr_798989.trc
Mon Jul 07 13:50:00 2008
Killing enqueue blocker (pid=1101878) on resource CF-00000000-00000000
by killing session 3294.119
Mon Jul 07 13:57:31 2008
GES: Potential blocker (pid=1101878) on resource CF-00000000-00000000;
enqueue info in file /u01/oracle/stdr/DGHFLR/bdump/DGHFLR_lgwr_798989.trc
Mon Jul 07 13:58:31 2008
Killing enqueue blocker (pid=1101878) on resource CF-00000000-00000000
by terminating the process
Mon Jul 07 14:06:01 2008
GES: Potential blocker (pid=1101878) on resource CF-00000000-00000000;
enqueue info in file /u01/oracle/stdr/DGHFLR/bdump/DGHFLR_lgwr_798989.trc
Mon Jul 07 14:07:02 2008
Killing enqueue blocker (pid=1101878) on resource CF-00000000-00000000
by terminating the process
Mon Jul 07 14:07:32 2008
Errors in file /u01/oracle/stdr/DGHFLR/bdump/DGHFLR_lgwr_798989.trc:
ORA-00600: Interner Fehlercode, Argumente: [2103], [0], [0], [1], [900], [], [], []
Mon Jul 07 14:34:27 2008
WAITED TOO LONG FOR A ROW CACHE ENQUEUE LOCK! pid=42
System State dumped to trace file /u01/oracle/stdr/DGHFLR/udump/DGHFLR_ora_1507514.trc
Mon Jul 07 14:34:37 2008
WAITED TOO LONG FOR A ROW CACHE ENQUEUE LOCK! pid=173
Mon Jul 07 14:35:02 2008
WAITED TOO LONG FOR A ROW CACHE ENQUEUE LOCK! pid=75
Mon Jul 07 14:35:02 2008
WAITED TOO LONG FOR A ROW CACHE ENQUEUE LOCK! pid=164
Mon Jul 07 14:35:08 2008
WAITED TOO LONG FOR A ROW CACHE ENQUEUE LOCK! pid=30
Mon Jul 07 14:35:15 2008
WAITED TOO LONG FOR A ROW CACHE ENQUEUE LOCK! pid=36
Mon Jul 07 14:36:43 2008
WAITED TOO LONG FOR A ROW CACHE ENQUEUE LOCK! pid=13
Mon Jul 07 14:48:32 2008
WAITED TOO LONG FOR A ROW CACHE ENQUEUE LOCK! pid=43
System State dumped to trace file /u01/oracle/stdr/DGHFLR/udump/DGHFLR_ora_678788.trc
Mon Jul 07 14:48:42 2008
WAITED TOO LONG FOR A ROW CACHE ENQUEUE LOCK! pid=56
Mon Jul 07 14:48:49 2008
WAITED TOO LONG FOR A ROW CACHE ENQUEUE LOCK! pid=61
Mon Jul 07 14:49:51 2008
WAITED TOO LONG FOR A ROW CACHE ENQUEUE LOCK! pid=195
Mon Jul 07 14:50:59 2008
WAITED TOO LONG FOR A ROW CACHE ENQUEUE LOCK! pid=40
Mon Jul 07 14:52:48 2008
WAITED TOO LONG FOR A ROW CACHE ENQUEUE LOCK! pid=198
Mon Jul 07 14:55:05 2008
WAITED TOO LONG FOR A ROW CACHE ENQUEUE LOCK! pid=191
and excerpt from /u01/oracle/stdr/DGHFLR/bdump/DGHFLR_lgwr_798989.trc
*** 2008-07-10 13:49:00.238
Unable to get enqueue on resource CF-00000000-00000000 (ges mode req=4 held=6)
Possible local blocker ospid=1101878 sid=3294 sser=119 time_held=19238 secs (ges mode req=6 held=3)
Killing blocker (pid=1101878) on resource CF-00000000-00000000
DUMP LOCAL BLOCKER: initiate state dump for KILL BLOCKER
possible owner[122.1101878]
Dumping process info of pid[122.1101878] requested by pid[6.405614]
Dumping process 122.1101878 info:
*** 2008-07-10 13:49:00.312
Dumping diagnostic information for ospid 1101878:
OS pid = 1101878
loadavg : 1.98 2.06 2.12
swap info: free_mem = 1781.95M rsv = 16.00M
alloc = 27.69M avail = 4096.00M swap_free = 4068.31M
F S UID PID PPID C PRI NI ADDR SZ WCHAN STIME TTY TIME CMD
240001 A oracle 1101878 1 0 60 20 411e5510 91912 f1000100343e2e70 08:13:22 - 0:00 oracleDGHFLR (LOCAL=NO)
open: Permission denied
skgpgpstack: read() for cmd /bin/sh -c '/usr/bin/procstack 1101878 2>&1' timed out after 60 seconds
*** 2008-07-10 13:50:00.605
Additional remark:
"ls -lrt" was run in the directory where the datafiles, redofiles and one of the controlfiles is stored: the command hung and did not come back. The directoriy is mounted by nfs. Client nfs messages were running.