oracle error ORA-03113: end-of-file on communication channel
780076Jun 23 2010 — edited Jun 23 2010Dear Support
I am getting "ORA-03113: end-of-file on communication channel" when I am trying to access my application.
The current setup is following
- two node Sun Cluster with node name as sep1n1 and sep2n2
- running Oracle RAC (oracle database 10g) with instance name as ADVFRW1 and ADVFRW2
Please see a part of oracle trace file which was generated at the time of problem.
Can you please help me to know what can be the reason of this oracle error/crash?
Thanks
Regards
VS
/export/home/oracle/admin/ADVFRW/udump/advfrw2_ora_29477.trc
Oracle Database 10g Enterprise Edition Release 10.1.0.4.0 - 64bit Production
With the Partitioning, Real Application Clusters and Data Mining options
ORACLE_HOME = /export/home/oracle/products/10.1.0
System name: SunOS
Node name: sep2n2
Release: 5.9
Version: Generic_122300-32
Machine: sun4u
Instance name: ADVFRW2
Redo thread mounted by this instance: 2
Oracle process number: 165
Unix process pid: 29477, image: oracleADVFRW2@sep2n2
*** 2010-06-23 01:04:05.270
*** SERVICE NAME:(SYS$USERS) 2010-06-23 01:04:05.269
*** SESSION ID:(1061.7934) 2010-06-23 01:04:05.269
Exception signal: 11 (SIGSEGV), code: 1 (Address not mapped to object), addr: 0x34, PC: [0x101132b5c, kkdllpoEsc()+1084]
*** 2010-06-23 01:04:05.277
ksedmp: internal or fatal error
ORA-07445: exception encountered: core dump [kkdllpoEsc()+1084] [SIGSEGV] [Address not mapped to object] [0x000000034] [] []
----- Call Stack Trace -----
calling call entry argument values in hex
location type point (? means dubious value)
-------------------- -------- -------------------- ----------------------------
ksedmp()+1008 CALL ksedst() 000000042 ? 104756808 ?
104756818 ? 000000000 ?
104DECE18 ? 000000008 ?
ssexhd()+992 CALL ksedmp() 000000002 ? 000104C00 ?
104C44000 ? 000104C44 ?
000104C00 ? 000000001 ?
sigacthandler()+44 PTR_CALL 0000000000000000 00000000B ? 104DF3EF0 ?
000104C00 ? 000104C00 ?
000000011 ? 104C47000 ?
kkdllpoEsc()+1084 PTR_CALL 0000000000000000 00000000B ? 104DF3EF0 ?
104DF3C10 ? 000000000 ?
00000000C ? 000000000 ?
kkdllpo()+76 CALL kkdllpoEsc() 0000003FC ?
FFFFFFFF7FFF9720 ?
FFFFFFFF7FFF9718 ?
FFFFFFFF7FFF9728 ?
FFFFFFFF7FFF9730 ?
000000002 ?
koklgext()+68 CALL kkdllpo() 000002D7A ? 000000002 ?
FFFFFFFF7FFF9728 ?
000000002 ? 000000002 ?
000000002 ?
koklcprv()+332 CALL koklgext() 000002D7A ?
FFFFFFFF7FFF98E0 ?
FFFFFFFF7FFF98F7 ?
259F31A88 ? 000000002 ?
000000101 ?
koklglfn()+304 CALL koklcprv() FFFFFFFF7FFFBFD8 ?
000000000 ? 00000000A ?
00000000A ?
FFFFFFFF7FFF9BB0 ?
000003150 ?
kokltrim()+436 CALL koklglfn() FFFFFFFF7FFFBFD8 ?
000000000 ?
FFFFFFFF7FFF9BA0 ?
000000000 ?
FFFFFFFF7FFFAEC4 ?
000104C3E ?
kpolob()+1816 CALL kokltrim() 104C3F260 ?
FFFFFFFF7FFFB950 ?
000000003 ? 104177CB0 ?
104177000 ? 000104177 ?
opiodr()+3516 PTR_CALL 0000000000000000 000000000 ?
FFFFFFFF7FFFBFD8 ?
000000020 ? 104C47E90 ?
104C4B500 ?
FFFFFFFF7FFFD780 ?
ttcpip()+7232 PTR_CALL 0000000000000000 104C41000 ?
FFFFFFFF79C3AC90 ?
104C41810 ? 000104C41 ?
000104C00 ? 000000001 ?
opitsk()+4380 CALL ttcpip() 104C47E90 ? 102D0F5A0 ?
FFFFFFFF7FFFD780 ?
000000001 ?
FFFFFFFF7FFFD8D8 ?
FFFFFFFF7FFFCA74 ?
opiino()+2488 CALL opitsk() 000000012 ? 000000001 ?
000000000 ? 104C47E98 ?
104C47E90 ? 000000000 ?
opiodr()+3516 PTR_CALL 0000000000000000 000104C41 ? 104C417F8 ?
104C41000 ? 104C417F8 ?
0000000F5 ? 104C46CC8 ?
opidrv()+1892 CALL opiodr() 000104C47 ? 104C47000 ?
104C47000 ? 000104C00 ?
000104C00 ? 000104C47 ?
sou2o()+156 CALL opidrv() 000104C00 ? 000000000 ?
FFFFFFFF7FFFE348 ?
104C41000 ? 104177CB0 ?
000000000 ?
main()+208 CALL sou2o() FFFFFFFF7FFFE5B8 ?
00000003C ? 000000004 ?
FFFFFFFF7FFFE598 ?
1046AC000 ? 0001046AC ?
_start()+380 CALL main() 000000002 ?
FFFFFFFF7FFFE698 ?
FFFFFFFF7FFFEA56 ?
000000001 ? 000000000 ?
100000000 ?
--------------------- Binary Stack Dump ---------------------
-------------------
===================================================
Files currently opened by this process:
===================================================
PROCESS STATE
-------------
Process global information:
process: 4026a00bae8, call: 4026b91f1b8, xact: 4026b82c578, curses: 4026a341358, usrses: 4026a341358
----------------------------------------
SO: 4026a00bae8, type: 2, owner: 0, flag: INIT/-/-/0x00
(process) Oracle pid=165, calls cur/top: 4026b91f1b8/4026b91f1b8, flag: (0) -
int error: 0, call error: 0, sess error: 0, txn error 0
(post info) last post received: 0 0 154
last post received-location: kqlmbpil: last steps
last process to post me: 402e90030c8 1 6
last post sent: 0 0 26
last post sent-location: ksasnd
last process posted by me: 4026a001528 1 6
(latch info) wait_event=0 bits=0
Process Group: DEFAULT, pseudo proc: 3eccf5ec0
O/S info: user: rtp99, term: UNKNOWN, ospid: 29477
OSD pid info: Unix process pid: 29477, image: oracleADVFRW2@sep2n2
----------------------------------------
SO: 4026b64b380, type: 7, owner: 4026a00bae8, flag: INIT/-/-/0x00
(FOB) flags=2 fib ptr=ea6e1500 incno=0 pending i/o cnt=0
fname=/export/home/oracle/dev/ADVFRW/ADVFRW.tbs_appl_idx
----------------------------------------
SO: 4026b64b1e8, type: 7, owner: 4026a00bae8, flag: INIT/-/-/0x00
(FOB) flags=2 fib ptr=ea6dded0 incno=0 pending i/o cnt=0
fname=/export/home/oracle/dev/ADVFRW/ADVFRW.system
----------------------------------------
SO: 4026b64b128, type: 7, owner: 4026a00bae8, flag: INIT/-/-/0x00
(FOB) flags=2 fib ptr=ea6e0688 incno=0 pending i/o cnt=0
fname=/export/home/oracle/dev/ADVFRW/ADVFRW.tbs_rollback2
----------------------------------------
SO: 4026b64afa8, type: 7, owner: 4026a00bae8, flag: INIT/-/-/0x00
(FOB) flags=2 fib ptr=ea6e1168 incno=0 pending i/o cnt=0
fname=/export/home/oracle/dev/ADVFRW/ADVFRW.tbs_appl_tab
----------------------------------------
SO: 4026a341358, type: 4, owner: 4026a00bae8, flag: INIT/-/-/0x00
(session) trans: 4026b82c578, creator: 4026a00bae8, flag: (100041) USR/- BSY/-/-/-/-/-
DID: 0002-00A5-0000007F, short-term DID: 0002-00A5-00000080
txn branch: 0
oct: 0, prv: 0, sql: 0, psql: 3c16fdfc8, user: 24/ADV
O/S info: user: rtp99, term: , ospid: 22593, machine: sep2n2
program: SmcfProcGroup_513@sep2n2 (TNS V1-V3)
application name: SmcfProcGroup_513@sep2n2 (TNS V1-V3), hash value=0
last wait for 'library cache pin' blocking sess=0x0 seq=8728 wait_time=1048
handle address=3c6aaaf30, pin address=3cbd18a98, 100*mode+namespace=c9
Dumping Session Wait History
for 'library cache pin' count=1 wait_time=1048
handle address=3c6aaaf30, pin address=3cbd18a98, 100*mode+namespace=c9
for 'library cache lock' count=1 wait_time=238870
handle address=3c6aaaf30, lock address=3c6108d70, 100*mode+namespace=12d
for 'SQL*Net message from client' count=1 wait_time=200
driver id=54435000, #bytes=1, =0
for 'SQL*Net message to client' count=1 wait_time=1
driver id=54435000, #bytes=1, =0
for 'SQL*Net message from client' count=1 wait_time=159
driver id=54435000, #bytes=1, =0
for 'SQL*Net message to client' count=1 wait_time=1
driver id=54435000, #bytes=1, =0
for 'SQL*Net message from client' count=1 wait_time=3603
driver id=54435000, #bytes=1, =0
for 'SQL*Net message to client' count=1 wait_time=1
driver id=54435000, #bytes=1, =0
for 'SQL*Net message from client' count=1 wait_time=124
driver id=54435000, #bytes=1, =0
for 'SQL*Net message to client' count=1 wait_time=1
driver id=54435000, #bytes=1, =0
temporary object counter: 0
----------------------------------------
SO: 4026b82c578, type: 41, owner: 4026a341358, flag: INIT/-/-/0x00
(trans) bsn = 99159, flg = 0x1e03, flg2 = 0xc0000, prx = 0x0, ros = 2147483647, spn = 99173
efd = 4
parent xid: 0x0000.000.00000000
env: (scn: 0x0001.4d47b041 xid: 0x004b.013.00018207 uba: 0x03006847.147b.0f statement num=0 parent xid: xid: 0x0000.000.00000000 scn: 0x0001.4d47b04e 0sch: scn: 0x0000.00000000)
cev: (spc = 4420 arsp = 3cfec39e0 ubk tsn: 12 rdba: 0x03006847 useg tsn: 12 rdba: 0x030009c1
hwm uba: 0x03006847.147b.0f col uba: 0x00000000.0000.00
num bl: 1 bk list: 0x3d3108758)
Trans IMU st: 0 Pool index 65535, Redo pool 0x4026b82cc40, Undo pool 0x4026b82cd28
Redo pool range [0x104e0c3e0 0x104e0c3e0 0x104e0ebe0] Undo pool range [0x104e09be0 0x104e09be0 0x104e0c3e0]
cr opc: 0x0 spc: 4420 uba: 0x03006847.147b.0f
(enqueue) TX-004B0013-00018207 DID: 0002-00A5-0000007F
lv: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 flag: 0x47
res: 3c3b2f948, mode: X, prv: 3c3b2f958, own: 4026a341358, sess: 4026a341358, proc: 4026a00bae8
slk: 0
xga: 0x0, heap: UGA
----------------------------------------
SO: 3d3108758, type: 40, owner: 4026b82c578, flag: -/-/-/0x00
(List of Blocks) next index = 1
index savepoint itli buffer hint rdba
--------------------------------------------------
0 99160 1 0x40187f1d060 0x3c0b8e2
----------------------------------------
SO: 4026b6e6230, type: 37, owner: 4026b82c578, flag: INIT/-/-/0x00
DML LOCK: tab=11679 flg=11 chi=1
his[0]: mod=2 spn=99159
his[1]: mod=3 spn=99167
(enqueue) TM-00002D9F-00000000 DID: 0002-00A5-0000007F
lv: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 flag: 0x7
res: 3c61313f0, mode: SX, prv: 3c6131400, own: 4026a341358, sess: 4026a341358, proc: 4026a00bae8
slk: 4026a7eb588
----------------------------------------
SO: 4026b6e6160, type: 37, owner: 4026b82c578, flag: INIT/-/-/0x00
DML LOCK: tab=11642 flg=11 chi=0
his[0]: mod=3 spn=99159
(enqueue) TM-00002D7A-00000000 DID: 0002-00A5-0000007F
lv: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 flag: 0x7
res: 3efdd3ed8, mode: SX, prv: 402ea6e6198, own: 4026a341358, sess: 4026a341358, proc: 4026a00bae8
slk: 4026a7eb440
----------------------------------------
SO: 3c74d4e30, type: 54, owner: 4026a341358, flag: INIT/-/-/0x00
LIBRARY OBJECT LOCK: lock=3c74d4e30 handle=3e6f98560 mode=N
call pin=0 session pin=0 hpc=0000 hlc=0000
htl=3c74d4ea0[3cbcf2a88,3cbcf2a88] htb=3cbcf2a88
user=4026a341358 session=4026a341358 count=0 flags=LRU/[4000] savepoint=35232
LIBRARY OBJECT HANDLE: handle=3e6f98560
name=SYS.TABPART$
hash=83541a3f9de24d0f5699f1f32ca1a4ad timestamp=10-22-2008 10:11:35
namespace=TABL flags=KGHP/TIM/SML/[02000000]
kkkk-dddd-llll=0000-0749-0749 lock=N pin=0 latch#=3 hpc=0024 hlc=0024
lwt=3e6f98608[3e6f98608,3e6f98608] ltm=3e6f98618[3e6f98618,3e6f98618]
pwt=3e6f985d0[3e6f985d0,3e6f985d0] ptm=3e6f985e0[3e6f985e0,3e6f985e0]
ref=3e6f98638[3e6f98638, 3e6f98638] lnd=3e6f98650[3e6f6a9e8,3e6ef5ef0]
LOCK INSTANCE LOCK: id=LB83541a3f9de24d0f
PIN INSTANCE LOCK: id=NB83541a3f9de24d0f mode=S release=F flags=[00]
INVALIDATION INSTANCE LOCK: id=IV000000e1160b0c24 mode=S
LIBRARY OBJECT: object=3aebb4978
type=TABL flags=EXS/LOC[0005] pflags= [00] status=VALD load=0
DATA BLOCKS:
data# heap pointer status pins change
----- -------- -------- ------ ---- ------
0 3aebb4c98 3aebb4a78 I/-/A 0 NONE
3 3e6faadb8 3aebc7d28 I/-/A 0 NONE
6 3aefece98 0 -/P/- 0 NONE
8 3e6faab28 3aed7f200 I/-/A 0 NONE
9 3e6faabd8 3ae3dcd90 I/-/A 0 NONE
10 3aefecf48 3aebc7580 I/-/A 0 NONE
----------------------------------------
SO: 3cbd17fe8, type: 55, owner: 4026a341358, flag: INIT/-/-/0x00
LIBRARY OBJECT PIN: pin=3cbd17fe8 handle=3bfe2aa38 mode=S lock=3c8abd710
user=4026a341358 session=4026a341358 count=1 mask=0041 savepoint=35232 flags=[00]
----------------------------------------
SO: 3c8abd710, type: 54, owner: 4026a341358, flag: INIT/-/-/0x00
LIBRARY OBJECT LOCK: lock=3c8abd710 handle=3bfe2aa38 mode=N
call pin=0 session pin=3cbd17fe8 hpc=0000 hlc=0000
htl=3c8abd780[3b45b58a8,3cbcf2908] htb=3cbcf2908
user=4026a341358 session=4026a341358 count=1 flags=PNS/[0800] savepoint=35232
LIBRARY OBJECT HANDLE: handle=3bfe2aa38
namespace=CRSR flags=RON/KGHP/PN0/[10010000]
kkkk-dddd-llll=0000-0041-0041 lock=N pin=S latch#=34 hpc=0002 hlc=0002
lwt=3bfe2aae0[3bfe2aae0,3bfe2aae0] ltm=3bfe2aaf0[3bfe2aaf0,3bfe2aaf0]
pwt=3bfe2aaa8[3bfe2aaa8,3bfe2aaa8] ptm=3bfe2aab8[3bfe2aab8,3bfe2aab8]
ref=3bfe2ab10[3b1d29738, 3b1d29738] lnd=3bfe2ab28[3bfe2ab28,3bfe2ab28]
LIBRARY OBJECT: object=3bfe64f98
type=CRSR flags=EXS[0001] pflags= [00] status=VALD load=0
DEPENDENCIES: count=1 size=16
ACCESSES: count=1 size=16
DATA BLOCKS:
data# heap pointer status pins change
----- -------- -------- ------ ---- ------
0 3b8b185f0 3b82879e8 I/P/A 0 NONE
6 3b8b98488 3b82865a8 I/P/A 1 NONE
----------------------------------------
SO: 3b45b5838, type: 54, owner: 4026a341358, flag: INIT/-/-/0x00
LIBRARY OBJECT LOCK: lock=3b45b5838 handle=3c6eff690 mode=N
call pin=3cbd22d28 session pin=0 hpc=0000 hlc=0000
htl=3b45b58a8[3cbcf2908,3c8abd780] htb=3cbcf2908
user=4026a341358 session=4026a341358 count=1 flags=[0000] savepoint=35231
LIBRARY OBJECT HANDLE: handle=3c6eff690
name=table_4_d_e1_0_0_0
hash=9bd183a3a089dfb7845d29d953482695 timestamp=06-18-2010 02:00:12
namespace=CRSR flags=RON/KGHP/TIM/PN0/SML/[12010000]
kkkk-dddd-llll=0000-0001-0001 lock=N pin=0 latch#=34 hpc=0034 hlc=0034
lwt=3c6eff738[3c6eff738,3c6eff738] ltm=3c6eff748[3c6eff748,3c6eff748]
pwt=3c6eff700[3c6eff700,3c6eff700] ptm=3c6eff710[3c6eff710,3c6eff710]
ref=3c6eff768[3c6eff768, 3c6eff768] lnd=3c6eff780[3c6eff780,3c6eff780]
LIBRARY OBJECT: object=3c6eff358
type=CRSR flags=EXS[0001] pflags= [00] status=VALD load=0
CHILDREN: size=128