Hi Experts,
I am on Oracle 11.2.0.2 on Linux. In my database, there is a deadlock reported and I am not able to find out reason for it or even which two different sessions participated in it. Following is the info from the deadlock trace file. I will be thankful for any pointers on the cause of the issue.
*** 2012-10-25 15:38:06.189
ENQUEUE DUMP REQUEST: from 2 spnum 12 on [0xa001f][0x33e15],[TX][ext 0x2,0x0] for reason 3 mtype 0
DUMP LOCAL BLOCKER/HOLDER: block level 3 res [0xa001f][0x33e15],[TX][ext 0x2,0x0]
----------resource 0xdd61cea0----------------------
resname : [0xa001f][0x33e15],[TX][ext 0x2,0x0]
hash mask : x3
Local inst : 1
dir_inst : 1
master_inst : 1
hv idx : 8
hv last r.inc : 8
current inc : 8
hv status : 0
hv master : 1
open options : dd
grant_bits : KJUSERNL KJUSEREX
grant mode : KJUSERNL KJUSERCR KJUSERCW KJUSERPR KJUSERPW KJUSEREX
count : 1 0 0 0 0 1
val_state : KJUSERVS_NOVALUE
valblk : 0x00000000000000000000000000000000 .
access_inst : 1
vbreq_state : 0
state : x0
resp : 0xdd61cea0
On Scan_q? : N
Total accesses: 6860
Imm. accesses: 6057
Granted_locks : 1
Cvting_locks : 1
value_block: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
GRANTED_Q :
lp 0xcb4fabd0 gl KJUSEREX rp 0xdd61cea0 [0xa001f][0x33e15],[TX][ext 0x2,0x0]
master 1 gl owner 0xd4c45958 possible pid 582 xid 77000-0001-00000C6F bast 0 rseq 767 mseq 0 history 0xd497d495
open opt KJUSERDEADLOCK
CONVERT_Q:
lp 0xcc5dae48 gl KJUSERNL rl KJUSERPR rp 0xdd61cea0 [0xa001f][0x33e15],[TX][ext 0x2,0x0]
master 1 owner 2 bast 1 rseq 264 mseq 0x1 history 0x77d497ad
convert opt KJUSERGETVALUE
----------enqueue 0xcb4fabd0------------------------
lock version : 535
Owner inst : 1
grant_level : KJUSEREX
req_level : KJUSEREX
bast_level : KJUSERNL
notify_func : (nil)
resp : 0xdd61cea0
procp : 0xdcf1f730
pid : 18995
proc version : 0
oprocp : (nil)
opid : 18995
group lock owner : 0xd4c45958
possible pid : 582
xid : 77000-0001-00000C6F
dd_time : 0.0 secs
dd_count : 0
timeout : 0.0 secs
On_timer_q? : N
On_dd_q? : N
lock_state : GRANTED
ast_flag : 0x0
Open Options : KJUSERDEADLOCK
Convert options : KJUSERNOQUEUE KJUSERNODEADLOCKWAIT
History : 0xd497d495
Msg_Seq : 0x0
res_seq : 767
valblk : 0x00000000000000000000000000000000 .
user session for deadlock lock 0xcb4fabd0 <----this seems to be the same session as below and I dont see info of any other session
sid: 349 ser: 12203 audsid: 102101678 user: 72/PORTALWFN45_ACS
flags: (0x41) USR/- flags_idl: (0x1) BSY/-/-/-/-/-
flags2: (0x40009) -/-/INC
pid: 119 O/S info: user: grid, term: UNKNOWN, ospid: 582
image: oracle@cdlt3f002db01
client details:
O/S info: user: wasadmin, term: unknown, ospid: 1234
machine: cdldvjassvap048.nj.adp.com program: JDBC Thin Client
application name: JDBC Thin Client, hash value=2546894660
current SQL:
INSERT INTO ACCESS_CLNT_GRP_XRF (CLNT_OID, GRP_ID, INCLUDED, ADDED_MANUALLY, CREATED_DATE, CREATED_BY) SELECT DISTINCT X.CLNT_OID, X.GRP_OID, :"SYS_B_0" AS INCLUDED, :"SYS_B_1" AS ADDED_MANUALLY, SYSDATE AS CREATED_DATE, :1 AS CREATED_BY FROM ACCESS_CLNT_GRP_RULE_XRF X WHERE X.GRP_ID = :2 AND NOT EXISTS (SELECT :"SYS_B_2" FROM ACCESS_CLNT_GRP_XRF UG WHERE UG.CLNT_OID= X.CLNT_OID AND UG.GRP_ID= X.GRP_ID)
DUMP LOCAL BLOCKER: initiate state dump for DEADLOCK
possible owner[119.582] on resource TX-000A001F-00033E15
*** 2012-10-25 15:38:06.190
Submitting asynchronized dump request [28]. summary=[ges process stack dump (kjdglblkrdm1)].
Global blockers dump start:---------------------------------
DUMP LOCAL BLOCKER/HOLDER: block level 3 res [0x170011][0x10ab5],[TX][ext 0x4,0x0]
----------resource 0xda168d40----------------------
resname : [0x170011][0x10ab5],[TX][ext 0x4,0x0]
hash mask : x3
Local inst : 1
dir_inst : 2
master_inst : 2
hv idx : 26
hv last r.inc : 8
current inc : 8
hv status : 0
hv master : 1
open options : dd
Held mode : KJUSERNL
Cvt mode : KJUSERPR
Next Cvt mode : KJUSERNL
msg_seq : 0x1
res_seq : 290
grant_bits : KJUSERNL
grant mode : KJUSERNL KJUSERCR KJUSERCW KJUSERPR KJUSERPW KJUSEREX
count : 1 0 0 0 0 0
val_state : KJUSERVS_NOVALUE
valblk : 0x70529d50ff7f00000000000000000000 pRP
access_inst : 2
vbreq_state : 0
state : x8
resp : 0xda168d40
On Scan_q? : N
Total accesses: 8496
Imm. accesses: 7633
Granted_locks : 0
Cvting_locks : 1
value_block: 70 52 9d 50 ff 7f 00 00 00 00 00 00 00 00 00 00
GRANTED_Q :
CONVERT_Q:
lp 0xdb922560 gl KJUSERNL rl KJUSERPR rp 0xda168d40 [0x170011][0x10ab5],[TX][ext 0x4,0x0]
master 2 gl owner 0xdf73f828 possible pid 582 xid 77000-0001-00000C6F bast 0 rseq 290 mseq 0 history 0x495149da
convert opt KJUSERGETVALUE
DUMP LOCAL BLOCKER/HOLDER: block level 3 res [0xa001f][0x33e15],[TX][ext 0x2,0x0]
----------resource 0xdd61cea0----------------------
resname : [0xa001f][0x33e15],[TX][ext 0x2,0x0]
hash mask : x3
Local inst : 1
dir_inst : 1
master_inst : 1
hv idx : 8
hv last r.inc : 8
current inc : 8
hv status : 0
hv master : 1
open options : dd
grant_bits : KJUSERNL KJUSEREX
grant mode : KJUSERNL KJUSERCR KJUSERCW KJUSERPR KJUSERPW KJUSEREX
count : 1 0 0 0 0 1
val_state : KJUSERVS_NOVALUE
valblk : 0x70529d50ff7f00000000000000000000 pRP
access_inst : 1
vbreq_state : 0
state : x0
resp : 0xdd61cea0
On Scan_q? : N
Total accesses: 6862
Imm. accesses: 6058
Granted_locks : 1
Cvting_locks : 1
value_block: 70 52 9d 50 ff 7f 00 00 00 00 00 00 00 00 00 00
GRANTED_Q :
lp 0xcb4fabd0 gl KJUSEREX rp 0xdd61cea0 [0xa001f][0x33e15],[TX][ext 0x2,0x0]
master 1 gl owner 0xd4c45958 possible pid 582 xid 77000-0001-00000C6F bast 0 rseq 767 mseq 0 history 0xd497d495
open opt KJUSERDEADLOCK
CONVERT_Q:
lp 0xcc5dae48 gl KJUSERNL rl KJUSERPR rp 0xdd61cea0 [0xa001f][0x33e15],[TX][ext 0x2,0x0]
master 1 owner 2 bast 1 rseq 264 mseq 0x1 history 0x77d497ad
convert opt KJUSERGETVALUE
----------enqueue 0xcb4fabd0------------------------
lock version : 535
Owner inst : 1
grant_level : KJUSEREX
req_level : KJUSEREX
bast_level : KJUSERNL
notify_func : (nil)
resp : 0xdd61cea0
procp : 0xdcf1f730
pid : 18995
proc version : 0
oprocp : (nil)
opid : 18995
group lock owner : 0xd4c45958
possible pid : 582
xid : 77000-0001-00000C6F
dd_time : 0.0 secs
dd_count : 0
timeout : 0.0 secs
On_timer_q? : N
On_dd_q? : N
lock_state : GRANTED
ast_flag : 0x0
Open Options : KJUSERDEADLOCK
Convert options : KJUSERNOQUEUE KJUSERNODEADLOCKWAIT
History : 0xd497d495
Msg_Seq : 0x0
res_seq : 767
valblk : 0x00000000000000003100050200000000 .1
user session for deadlock lock 0xcb4fabd0 <----this seems to be the same session as above and I dont see info of any other session
sid: 349 ser: 12203 audsid: 102101678 user: 72/PORTALWFN45_ACS
sid: 349 ser: 12203 audsid: 102101678 user: 72/PORTALWFN45_ACS
flags: (0x41) USR/- flags_idl: (0x1) BSY/-/-/-/-/-
flags2: (0x40009) -/-/INC
pid: 119 O/S info: user: grid, term: UNKNOWN, ospid: 582
image: oracle@cdlt3f002db01
client details:
O/S info: user: wasadmin, term: unknown, ospid: 1234
machine: cdldvjassvap048.nj.adp.com program: JDBC Thin Client
application name: JDBC Thin Client, hash value=2546894660
current SQL:
INSERT INTO ACCESS_CLNT_GRP_XRF (CLNT_OID, GRP_ID, INCLUDED, ADDED_MANUALLY, CREATED_DATE, CREATED_BY) SELECT DISTINCT X.CLNT_OID, X.GRP_OID, :"SYS_B_0" AS INCLUDED, :"SYS_B_1" AS ADDED_MANUALLY, SYSDATE AS CREATED_DATE, :1 AS CREATED_BY FROM ACCESS_CLNT_GRP_RULE_XRF X WHERE X.GRP_ID = :2 AND NOT EXISTS (SELECT :"SYS_B_2" FROM ACCESS_CLNT_GRP_XRF UG WHERE UG.CLNT_OID= X.CLNT_OID AND UG.GRP_ID= X.GRP_ID)
DUMP LOCAL BLOCKER: initiate state dump for DEADLOCK
possible owner[119.582] on resource TX-000A001F-00033E15
*** 2012-10-25 15:38:06.191
Submitting asynchronized dump request [28]. summary=[ges process stack dump (kjdglblkrdm1)].
Global blockers dump end:-----------------------------------
Global Wait-For-Graph(WFG) at ddTS[0.8a] :
BLOCKED 0xdb922560 3 wq 2 cvtops x1 TX 0x170011.0x10ab5(ext 0x4,0x0)[77000-0001-00000C6F] inst 1
BLOCKER 0xdb339ad0 3 wq 1 cvtops x28 TX 0x170011.0x10ab5(ext 0x4,0x0)[71000-0002-00002D5B] inst 2
BLOCKED 0xdb7f3648 3 wq 2 cvtops x1 TX 0xa001f.0x33e15(ext 0x2,0x0)[71000-0002-00002D5B] inst 2
BLOCKER 0xcb4fabd0 3 wq 1 cvtops x28 TX 0xa001f.0x33e15(ext 0x2,0x0)[77000-0001-00000C6F] inst 1
*** 2012-10-25 15:38:07.113
* Cancel deadlock victim lockp 0xdb922560
2012-10-25 17:26:16.039638 : Setting 3-way CR grants to 0 global-lru off? 0
*** 2012-10-25 17:27:49.961
2012-10-25 17:27:49.961892 : Setting 3-way CR grants to 1 global-lru off? 0
2012-10-25 17:52:42.505182 : Setting 3-way CR grants to 1 global-lru off? 0
* received DRM start msg from 2 (cnt 1, last 1, rmno 50)
*** 2012-10-25 19:16:31.443
Rcvd DRM(50) AFFINITY Transfer pkey 88431.0 from 1 to 2 oscan 1.1
ftd (30) received from node 2 (8 0.31/0.0)
all ftds received
ftd (33) received from node 2 (8 0.32/0.0)
all ftds received
* kjxftdn: break from kjxftdn, post lmon later
ftd (35) received from node 2 (8 0.36/0.0)
all ftds received
ftd (37) received from node 2 (8 0.38/0.0)
all ftds received
ftd (30) received from node 2 (8 0.31/0.0)
all ftds received
ftd (33) received from node 2 (8 0.33/0.0)
all ftds received
* kjxftdn: break from kjxftdn, post lmon later
ftd (35) received from node 2 (8 0.36/0.0)
all ftds received
ftd (37) received from node 2 (8 0.38/0.0)
all ftds received
ftd (30) received from node 2 (8 0.31/0.0)
all ftds received
ftd (33) received from node 2 (8 0.33/0.0)
all ftds received
* kjxftdn: break from kjxftdn, post lmon later
ftd (35) received from node 2 (8 0.36/0.0)
all ftds received
ftd (37) received from node 2 (8 0.38/0.0)
all ftds received
ftd (30) received from node 2 (8 0.31/0.0)
all ftds received
ftd (33) received from node 2 (8 0.33/0.0)
all ftds received
* kjxftdn: break from kjxftdn, post lmon later
ftd (35) received from node 2 (8 0.36/0.0)
all ftds received
ftd (37) received from node 2 (8 0.37/0.0)
all ftds received
* kjxftdn: break from kjxftdn, post lmon later
ftd (30) received from node 2 (8 0.31/0.0)
all ftds received
ftd (33) received from node 2 (8 0.32/0.0)
all ftds received
* kjxftdn: break from kjxftdn, post lmon later
ftd (35) received from node 2 (8 0.36/0.0)
all ftds received
ftd (37) received from node 2 (8 0.38/0.0)
all ftds received
ftd (30) received from node 2 (8 0.31/0.0)
all ftds received
ftd (33) received from node 2 (8 0.33/0.0)
all ftds received
* kjxftdn: break from kjxftdn, post lmon later
ftd (35) received from node 2 (8 0.36/0.0)
all ftds received
ftd (37) received from node 2 (8 0.38/0.0)
all ftds received
ftd (30) received from node 2 (8 0.31/0.0)
all ftds received
ftd (33) received from node 2 (8 0.33/0.0)
all ftds received
* kjxftdn: break from kjxftdn, post lmon later
ftd (35) received from node 2 (8 0.36/0.0)
all ftds received
ftd (37) received from node 2 (8 0.38/0.0)
all ftds received
ftd (30) received from node 2 (8 0.31/0.0)
all ftds received
ftd (33) received from node 2 (8 0.33/0.0)
all ftds received
* kjxftdn: break from kjxftdn, post lmon later
ftd (35) received from node 2 (8 0.36/0.0)
all ftds received
ftd (37) received from node 2 (8 0.38/0.0)
all ftds received
2012-10-25 19:16:31.687987 :
End DRM(50) for pkey transfer request(s) from 2
stale cvak(res) fr 2:0xdfff5898([0x1978f378][0x8fba1c6e],[LB][ext 0x0,0x0])[h=KJUSERNL,n=KJUSEREX,b=KJUSERNL,ls=KJUSERSTAT_NOVALUE]mseq:0x10003(msg),0x0(resp)rseq:0x416(msg),0x417(resp)
*** 2012-10-25 22:06:03.671
----------resource 0xdfff5898----------------------
resname : [0x1978f378][0x8fba1c6e],[LB][ext 0x0,0x0]
hash mask : x3
Local inst : 1
dir_inst : 2
master_inst : 2
hv idx : 50
hv last r.inc : 8
current inc : 8
hv status : 0
hv master : 1
open options : dd
Held mode : KJUSERNL
Cvt mode : KJUSERPR
Next Cvt mode : KJUSERNL
msg_seq : (nil)
res_seq : 1047
grant_bits : KJUSERNL
grant mode : KJUSERNL KJUSERCR KJUSERCW KJUSERPR KJUSERPW KJUSEREX
count : 1 0 0 0 0 0
val_state : KJUSERVS_NOVALUE
valblk : 0x94160ec3eccc040094160ec3eccc0400 .
access_inst : 2
vbreq_state : 0
state : x4
resp : 0xdfff5898
On Scan_q? : N
Total accesses: 13084
Imm. accesses: 11996
Granted_locks : 0
Cvting_locks : 1
value_block: 94 16 0e c3 ec cc 04 00 94 16 0e c3 ec cc 04 00
GRANTED_Q :
CONVERT_Q:
lp 0xdb6588d0 gl KJUSERNL rl KJUSERPR rp 0xdfff5898 [0x1978f378][0x8fba1c6e],[LB][ext 0x0,0x0]
master 2 gl owner 0xdc5d6240 possible pid 6224 xid 4B000-0001-000012F8 bast 0 rseq 1047 mseq 0 history 0x495149da
convert opt
MSG [16:KJX_CONV_AST_IND] [0x1978f378][0x8fba1c6e],[LB][ext 0x0,0x0] RMno=65706 inc=8 len=112 sender=(2,2) seq=0
fg=p stat=KJUSERSTAT_NOVALUE spnum=12 flg=x0
flow ctrl: ver=8 flag=161 len=112 tkts=0 seq=0 wrp=0
sndr=2 dest=1 rcvr=0
rseq x416 mseq x10003 copts x1 hl 5 cl 0 nl 0
valblk 0x01000000815f93600000000000000000 ._`
stale cvak(res) fr 2:0xdfff5898([0x1978f378][0x8fba1c6e],[LB][ext 0x0,0x0])[h=KJUSERNL,n=KJUSEREX,b=KJUSERPR,ls=KJUSERSTAT_NOVALUE]mseq:0x10004(msg),0x0(resp)rseq:0x416(msg),0x417(resp)
----------resource 0xdfff5898----------------------
resname : [0x1978f378][0x8fba1c6e],[LB][ext 0x0,0x0]
hash mask : x3
Local inst : 1
dir_inst : 2
master_inst : 2
hv idx : 50
hv last r.inc : 8
current inc : 8
hv status : 0
hv master : 1
open options : dd
Held mode : KJUSERNL
Cvt mode : KJUSERPR
Next Cvt mode : KJUSERNL
msg_seq : (nil)
res_seq : 1047
grant_bits : KJUSERNL
grant mode : KJUSERNL KJUSERCR KJUSERCW KJUSERPR KJUSERPW KJUSEREX
count : 1 0 0 0 0 0
val_state : KJUSERVS_NOVALUE
valblk : 0x24000000000000000008000000000000 $
access_inst : 2
vbreq_state : 0
state : x4
resp : 0xdfff5898
On Scan_q? : N
Total accesses: 13085
Imm. accesses: 11997
Granted_locks : 0
Cvting_locks : 1
value_block: 24 00 00 00 00 00 00 00 00 08 00 00 00 00 00 00
GRANTED_Q :
CONVERT_Q:
lp 0xdb6588d0 gl KJUSERNL rl KJUSERPR rp 0xdfff5898 [0x1978f378][0x8fba1c6e],[LB][ext 0x0,0x0]
master 2 gl owner 0xdc5d6240 possible pid 6224 xid 4B000-0001-000012F8 bast 0 rseq 1047 mseq 0 history 0x495149da
convert opt
MSG [16:KJX_CONV_AST_IND] [0x1978f378][0x8fba1c6e],[LB][ext 0x0,0x0] RMno=3 inc=8 len=112 sender=(2,2) seq=0
fg=p stat=KJUSERSTAT_NOVALUE spnum=12 flg=xc
flow ctrl: ver=-106 flag=161 len=112 tkts=0 seq=0 wrp=0
sndr=2 dest=1 rcvr=0
rseq x416 mseq x10004 copts x0 hl 5 cl 0 nl 3
valblk 0x0000000000a907006800000000000000 .h
kjddt2vb: valblk [0.8c] > local ts [0.8b]
*** 2012-10-25 22:16:54.817
Begin DRM(51) (swin 0) - AFFINITY transfer pkey 88431.0 to 1 oscan 0.1
kjiobjscn 1
ftd (30) received from node 2 (8 0.0/0.0)
all ftds received
* kjxftdn: break from kjxftdn, post lmon later
ftd (33) received from node 2 (8 0.32/0.0)
all ftds received
* kjxftdn: break from kjxftdn, post lmon later
ftd (37) received from node 2 (8 0.36/0.0)
all ftds received
ftd (35) received from node 2 (8 0.36/0.0)
all ftds received
* kjxftdn: break from kjxftdn, post lmon later
ftd (30) received from node 2 (8 0.30/0.0)
all ftds received
* kjxftdn: break from kjxftdn, post lmon later
ftd (33) received from node 2 (8 0.34/0.0)
all ftds received
ftd (37) received from node 2 (8 0.36/0.0)
all ftds received
ftd (35) received from node 2 (8 0.36/0.0)
all ftds received
* kjxftdn: break from kjxftdn, post lmon later
ftd (30) received from node 2 (8 0.30/0.0)
all ftds received
* kjxftdn: break from kjxftdn, post lmon later
ftd (33) received from node 2 (8 0.34/0.0)
all ftds received
ftd (35) received from node 2 (8 0.36/0.0)
all ftds received
ftd (37) received from node 2 (8 0.37/0.0)
all ftds received
* kjxftdn: break from kjxftdn, post lmon later
ftd (30) received from node 2 (8 0.31/0.0)
all ftds received
ftd (33) received from node 2 (8 0.34/0.0)
all ftds received
ftd (35) received from node 2 (8 0.36/0.0)
all ftds received
ftd (37) received from node 2 (8 0.37/0.0)
all ftds received
* kjxftdn: break from kjxftdn, post lmon later
ftd (30) received from node 2 (8 0.31/0.0)
all ftds received
ftd (33) received from node 2 (8 0.34/0.0)
all ftds received
ftd (37) received from node 2 (8 0.36/0.0)
all ftds received
ftd (35) received from node 2 (8 0.36/0.0)
all ftds received
ftd (30) received from node 2 (8 0.31/0.0)
all ftds received
ftd (33) received from node 2 (8 0.34/0.0)
all ftds received
ftd (37) received from node 2 (8 0.36/0.0)
all ftds received
ftd (35) received from node 2 (8 0.36/0.0)
all ftds received
ftd (30) received from node 2 (8 0.30/0.0)
all ftds received
* kjxftdn: break from kjxftdn, post lmon later
ftd (33) received from node 2 (8 0.34/0.0)
all ftds received
ftd (37) received from node 2 (8 0.36/0.0)
all ftds received
ftd (35) received from node 2 (8 0.36/0.0)
all ftds received
ftd (30) received from node 2 (8 0.30/0.0)
all ftds received
* kjxftdn: break from kjxftdn, post lmon later
ftd (33) received from node 2 (8 0.34/0.0)
all ftds received
ftd (37) received from node 2 (8 0.36/0.0)
all ftds received
ftd (35) received from node 2 (8 0.36/0.0)
all ftds received
* kjxftdn: break from kjxftdn, post lmon later
2012-10-25 22:16:55.112752 :
* End DRM for pkey remastering request(s) (locally requested)
*** 2012-10-25 22:22:35.175
2012-10-25 22:22:35.175781 : Setting 3-way CR grants to 0 global-lru off? 0
*** 2012-10-25 22:24:09.000
2012-10-25 22:24:09.000900 : Setting 3-way CR grants to 1 global-lru off? 0
2012-10-25 22:29:20.348861 : Setting 3-way CR grants to 1 global-lru off? 0
2012-10-25 23:15:02.016255 : Setting 3-way CR grants to 0 global-lru off? 0
*** 2012-10-25 23:16:05.212
2012-10-25 23:16:05.212156 : Setting 3-way CR grants to 1 global-lru off? 0
*** 2012-10-25 23:16:36.041
2012-10-25 23:16:36.041650 : Setting 3-way CR grants to 0 global-lru off? 0
*** 2012-10-25 23:18:10.146
2012-10-25 23:18:10.146473 : Setting 3-way CR grants to 1 global-lru off? 0
*** 2012-10-25 23:23:24.514
2012-10-25 23:23:24.514488 : Setting 3-way CR grants to 0 global-lru off? 0
*** 2012-10-25 23:24:26.699
2012-10-25 23:24:26.699355 : Setting 3-way CR grants to 1 global-lru off? 0
2012-10-25 23:32:11.212627 : Setting 3-way CR grants to 0 global-lru off? 0
*** 2012-10-25 23:33:13.028
2012-10-25 23:33:13.028646 : Setting 3-way CR grants to 1 global-lru off? 0
*** 2012-10-25 23:36:22.121
2012-10-25 23:36:22.121783 : Setting 3-way CR grants to 0 global-lru off? 0
*** 2012-10-25 23:36:58.785
* received DRM start msg from 2 (cnt 1, last 1, rmno 52)
Rcvd DRM(52) AFFINITY Transfer pkey 88431.0 from 1 to 2 oscan 1.1
*** 2012-10-25 23:36:58.875
ftd (30) received from node 2 (8 0.31/0.0)
all ftds received
ftd (33) received from node 2 (8 0.33/0.0)
all ftds received
* kjxftdn: break from kjxftdn, post lmon later
ftd (35) received from node 2 (8 0.36/0.0)
all ftds received
ftd (37) received from node 2 (8 0.37/0.0)
all ftds received
* kjxftdn: break from kjxftdn, post lmon later
ftd (30) received from node 2 (8 0.31/0.0)
all ftds received
ftd (33) received from node 2 (8 0.33/0.0)
all ftds received
* kjxftdn: break from kjxftdn, post lmon later
ftd (35) received from node 2 (8 0.36/0.0)
all ftds received
ftd (37) received from node 2 (8 0.38/0.0)
all ftds received
ftd (30) received from node 2 (8 0.31/0.0)
all ftds received
ftd (33) received from node 2 (8 0.32/0.0)
all ftds received
* kjxftdn: break from kjxftdn, post lmon later
ftd (35) received from node 2 (8 0.36/0.0)
all ftds received
ftd (37) received from node 2 (8 0.38/0.0)
all ftds received
ftd (30) received from node 2 (8 0.31/0.0)
all ftds received
ftd (33) received from node 2 (8 0.33/0.0)
all ftds received
* kjxftdn: break from kjxftdn, post lmon later
ftd (35) received from node 2 (8 0.36/0.0)
all ftds received
ftd (37) received from node 2 (8 0.38/0.0)
all ftds received
ftd (30) received from node 2 (8 0.31/0.0)
all ftds received
ftd (33) received from node 2 (8 0.33/0.0)
all ftds received
* kjxftdn: break from kjxftdn, post lmon later
ftd (35) received from node 2 (8 0.36/0.0)
all ftds received
ftd (37) received from node 2 (8 0.38/0.0)
all ftds received
ftd (30) received from node 2 (8 0.31/0.0)
all ftds received
ftd (33) received from node 2 (8 0.33/0.0)
all ftds received
* kjxftdn: break from kjxftdn, post lmon later
ftd (35) received from node 2 (8 0.36/0.0)
all ftds received
ftd (37) received from node 2 (8 0.38/0.0)
all ftds received
ftd (30) received from node 2 (8 0.31/0.0)
all ftds received
ftd (33) received from node 2 (8 0.32/0.0)
all ftds received
* kjxftdn: break from kjxftdn, post lmon later
ftd (35) received from node 2 (8 0.36/0.0)
all ftds received
ftd (37) received from node 2 (8 0.38/0.0)
all ftds received
ftd (30) received from node 2 (8 0.31/0.0)
all ftds received
ftd (33) received from node 2 (8 0.32/0.0)
all ftds received
* kjxftdn: break from kjxftdn, post lmon later
ftd (35) received from node 2 (8 0.36/0.0)
all ftds received
ftd (37) received from node 2 (8 0.38/0.0)
all ftds received
2012-10-25 23:36:59.124760 :
End DRM(52) for pkey transfer request(s) from 2
*** 2012-10-25 23:38:26.130
2012-10-25 23:38:26.130755 : Setting 3-way CR grants to 1 global-lru off? 0
2012-10-25 23:58:38.071978 : Setting 3-way CR grants to 0 global-lru off? 0
*** 2012-10-26 00:00:44.096
2012-10-26 00:00:44.096005 : Setting 3-way CR grants to 1 global-lru off? 0
*** 2012-10-26 00:04:22.583
2012-10-26 00:04:22.583348 : Setting 3-way CR grants to 0 global-lru off? 0
*** 2012-10-26 00:08:31.315
2012-10-26 00:08:31.315412 : Setting 3-way CR grants to 1 global-lru off? 0
2012-10-26 00:20:57.846700 : Setting 3-way CR grants to 0 global-lru off? 0
*** 2012-10-26 00:25:05.085
2012-10-26 00:25:05.085042 : Setting 3-way CR grants to 1 global-lru off? 0
2012-10-26 00:44:17.764966 : Setting 3-way CR grants to 0 global-lru off? 0
*** 2012-10-26 00:48:25.602
2012-10-26 00:48:25.602491 : Setting 3-way CR grants to 1 global-lru off? 0
2012-10-26 01:14:50.222098 : Setting 3-way CR grants to 0 global-lru off? 0
*** 2012-10-26 01:19:00.590
2012-10-26 01:19:00.590826 : Setting 3-way CR grants to 1 global-lru off? 0
2012-10-26 02:44:09.453387 : Setting 3-way CR grants to 0 global-lru off? 0
*** 2012-10-26 02:49:19.105
2012-10-26 02:49:19.105284 : Setting 3-way CR grants to 1 global-lru off? 0
2012-10-26 02:59:41.444548 : Setting 3-way CR grants to 0 global-lru off? 0
*** 2012-10-26 03:02:47.808
2012-10-26 03:02:47.808071 : Setting 3-way CR grants to 1 global-lru off? 0
*** 2012-10-26 03:05:22.310
2012-10-26 03:05:22.310631 : Setting 3-way CR grants to 0 global-lru off? 0
*** 2012-10-26 03:07:16.408
* received DRM start msg from 2 (cnt 2, last 1, rmno 53)
Rcvd DRM(53) AFFINITY Transfer pkey 74335.0 from 1 to 2 oscan 1.1
Rcvd DRM(53) AFFINITY Transfer pkey 88398.0 from 1 to 2 oscan 1.1
ftd (30) received from node 2 (8 0.0/0.0)
all ftds received
* kjxftdn: break from kjxftdn, post lmon later
ftd (33) received from node 2 (8 0.32/0.0)
all ftds received
* kjxftdn: break from kjxftdn, post lmon later
ftd (35) received from node 2 (8 0.36/0.0)
all ftds received
ftd (37) received from node 2 (8 0.38/0.0)
all ftds received
ftd (30) received from node 2 (8 0.31/0.0)
all ftds received
ftd (33) received from node 2 (8 0.33/0.0)
all ftds received
* kjxftdn: break from kjxftdn, post lmon later
ftd (35) received from node 2 (8 0.36/0.0)
all ftds received
ftd (37) received from node 2 (8 0.38/0.0)
all ftds received
ftd (30) received from node 2 (8 0.31/0.0)
all ftds received
ftd (33) received from node 2 (8 0.33/0.0)
all ftds received
* kjxftdn: break from kjxftdn, post lmon later
ftd (35) received from node 2 (8 0.36/0.0)
all ftds received
ftd (37) received from node 2 (8 0.38/0.0)
all ftds received
ftd (30) received from node 2 (8 0.31/0.0)
all ftds received
ftd (33) received from node 2 (8 0.33/0.0)
all ftds received
* kjxftdn: break from kjxftdn, post lmon later
ftd (35) received from node 2 (8 0.36/0.0)
all ftds received
ftd (37) received from node 2 (8 0.38/0.0)
all ftds received
ftd (30) received from node 2 (8 0.31/0.0)
all ftds received
ftd (33) received from node 2 (8 0.33/0.0)
all ftds received
* kjxftdn: break from kjxftdn, post lmon later
ftd (35) received from node 2 (8 0.36/0.0)
all ftds received
ftd (37) received from node 2 (8 0.38/0.0)
all ftds received
ftd (30) received from node 2 (8 0.31/0.0)
all ftds received
ftd (33) received from node 2 (8 0.33/0.0)
all ftds received
* kjxftdn: break from kjxftdn, post lmon later
ftd (35) received from node 2 (8 0.36/0.0)
all ftds received
ftd (37) received from node 2 (8 0.38/0.0)
all ftds received
ftd (30) received from node 2 (8 0.31/0.0)
all ftds received
ftd (33) received from node 2 (8 0.33/0.0)
all ftds received
* kjxftdn: break from kjxftdn, post lmon later
ftd (35) received from node 2 (8 0.36/0.0)
all ftds received
ftd (37) received from node 2 (8 0.38/0.0)
all ftds received
ftd (30) received from node 2 (8 0.31/0.0)
all ftds received
ftd (33) received from node 2 (8 0.33/0.0)
all ftds received
* kjxftdn: break from kjxftdn, post lmon later
ftd (35) received from node 2 (8 0.36/0.0)
all ftds received
ftd (37) received from node 2 (8 0.38/0.0)
all ftds received
2012-10-26 03:07:16.718164 :
End DRM(53) for pkey transfer request(s) from 2
*** 2012-10-26 03:11:35.100
2012-10-26 03:11:35.100544 : Setting 3-way CR grants to 1 global-lru off? 0
2012-10-26 03:44:44.666224 : Setting 3-way CR grants to 0 global-lru off? 0
2012-10-26 04:00:49.272473 : Setting 3-way CR grants to 1 global-lru off? 0
*** 2012-10-26 04:08:35.037
2012-10-26 04:08:35.037716 : Setting 3-way CR grants to 0 global-lru off? 0
*** 2012-10-26 04:11:41.226
2012-10-26 04:11:41.226172 : Setting 3-way CR grants to 1 global-lru off? 0