Hi all,
DB: 10.2.0.3 EE
OS: Solaris 10
Given below is the summary of the problem I am facing, followed by the details.
Summary:
I am trying to do a 'backup as copy' operation in RMAN, but it is held up by the db writer process. The wait event for both rman and db writer process is 'rdbms ipc reply'. The db writer process is in turn held up by the 'RBAL' process. The RBAL process seems to be waiting on the same db writer process with the wait event 'enq: FP - global fob contention'. Any ideas what might be causing this and what I can do to resolve this?
Details:
I am running the following operation in RMAN:
RMAN> backup as copy database format '+DEV1';
However the process is stuck on the wait event 'rdbms ipc reply'.
SID OSUSER PROGRAM LAST_CALL_ET EVENT
----- -------------------- ------------------------------ ------------ ------------------------------
664 xxxxxx rman@xxxxxx (TNS V1-V3) 4509 rdbms ipc reply
Information from v$session_wait is:
SEQ# SID EVENT P1 P2 STATE WAIT_TIME SECONDS_IN_WAIT
---------- ----- ------------------------------ ---------- ---------- ------------------- ---------- ---------------
2848 664 rdbms ipc reply 2 24 WAITING 0 3
After a bit of googling I found out that the P1 parameter indicates the background process causing the wait, so I checked which process it is:
11:51:50 xxxx>SELECT NAME FROM V$BGPROCESS WHERE PADDR =
11:54:40 2 (SELECT ADDR FROM V$PROCESS WHERE PID = 2);
NAME
------------------------------
DBW0
So it is the so it is the db writer process 'DBW0', that is held up. So I checked the wait event for the DBW0 process (I found out the sid - it was 167)
SEQ# SID EVENT P1 P2 STATE WAIT_TIME SECONDS_IN_WAIT
---------- ----- ------------------------------ ---------- ---------- ------------------- ---------- ---------------
30315 167 rdbms ipc reply 55 2147423061 WAITING 0 0
So the 'DBW0' process is also waiting on 'rdbms ipc reply'. So I find out the process with pid = 55.
11:59:48 xxxx>SELECT NAME FROM V$BGPROCESS WHERE PADDR =
12:04:39 2 (SELECT ADDR FROM V$PROCESS WHERE PID = 55);
NAME
------------------------------
RBAL
It turns out to be RBAL. The sid for RBAL process is 321. So I find out what it is waiting on:
12:04:41 xxxx>select event, blocking_session blocker from v$session where sid =321;
EVENT BLOCKER
------------------------------ ----------
enq: FP - global fob 167
contention
12:06:14 xxxx>@sesswait
Enter value for sid: 321
SEQ# SID EVENT P1 P2 STATE WAIT_TIME SECONDS_IN_WAIT
---------- ----- ------------------------------ ---------- ---------- ------------------- ---------- ---------------
12 321 enq: FP - global fob 1179648006 0 WAITING 0 61470
contention
The blocker session for RBAL is 167, which is 'DBW0'. I couldn't find any information on this wait event 'enq: FP - global fob contention'. I am not able to progress any further on this, so any help will be much appreciated.
Edited by: joshic on 30-Jun-2010 04:25