Oracle Database Discussions

Hi
We have an oracle 11.1.0.7.0 database running on windows 2003 R2. All of a sudden as of today the server crashed.
Trace logs are as below.

Can anyone provide any help on how to diagnose the issue.

Thanks

*** 2011-04-26 00:30:15.291
Process diagnostic dump for W000, OS id=6212
-------------------------------------------------------------------------------
Memory (Avail/Total): Ph:1919M/3839M, Ph+PgF:2554M/4446M, VA:1260M/2047M
CPU Load: 61%
Stack:
------------------- Call Stack Trace ---------------------
calling location entry point arg #1 arg #2 arg #3 arg #4
------------------------------------------- ------------------------------------------- -------- -------- -------- --------
7C82847C 00000000 00000000 00000000 00000000 00000000
77E61C88 77E61C96 00001aa8 00001388 00000000 0f45e55c
_skgpwwait()+128 00000000 00001aa8 00001388 004c4b40 0bab7ca0
ksliwat()+2180 skgpwwait()+0 0f45e5c8 0babb460 2f371f7c 004c4b40
kslwaitctx()+146 ksliwat()+0 2ea568b0 2ea568b0 000001f4 00000000
kslwait()+158 kslwaitctx()+0 0f45e7bc 000001f4 0f45e7bc 06097a59
ksvrdp()+26715 kslwait()+0 000001f4 0000003e 00000000 0f45fd90
opirip()+813 ksvrdp()+0 00000003 0dc6e918 0dc85e78 0d6ff2dc
opidrv()+1357 opirip()+0 00000032 00000004 0f45fea8 00000000
sou2o()+45 opidrv()+0 00000032 00000004 0f45fea8 00000000
opimaireal()+263 _sou2o()+0 0f45fe9c 00000032 00000004 0f45fea8
opimai()+92 opimai_real()+0 00000003 0f45fed4 00000000 00000000
BackgroundThreadStart@4()+529 opimai()+0 00000003 0f45ff20 89a96710 0d6f0030
77E6482C 00000000 0b1a4c18 00000000 00000000 0b1a4c18
00000000 00000000 00000000 00000000 00000000 00000000
---------------- End of Call Stack Trace -----------------
Call stack acquisition performance stats:
setup time (lock acquis., memory alloc.): 0 ms
frame get time (time the target proc was suspended): 0 ms
symbol translation time: 0 ms
total time: 0 ms
-------------------------------------------------------------------------------
Process diagnostic dump actual duration=0.109000 sec
(max dump time=30.000000 sec)

*** 2011-04-26 00:30:15.400
Waited for process W000 to initialize for 107276 seconds

*** 2011-04-26 00:50:21.400
Process diagnostic dump for W000, OS id=11304
-------------------------------------------------------------------------------
Memory (Avail/Total): Ph:1923M/3839M, Ph+PgF:2563M/4446M, VA:1268M/2047M
CPU Load: 43%
Stack:
------------------- Call Stack Trace ---------------------
calling location entry point arg #1 arg #2 arg #3 arg #4
------------------------------------------- ------------------------------------------- -------- -------- -------- --------
7C82847C 00000000 00000000 00000000 00000000 00000000
77E61C88 77E61C96 000019b0 00001388 00000000 0d85e55c
_skgpwwait()+128 00000000 000019b0 00001388 004c4b40 0bab7ca0
ksliwat()+2180 skgpwwait()+0 0d85e5c8 0babb460 2f371494 004c4b40
kslwaitctx()+146 ksliwat()+0 2ea26d20 2ea26d20 000001f4 00000000
kslwait()+158 kslwaitctx()+0 0d85e7bc 000001f4 0d85e7bc 06097a59
ksvrdp()+26715 kslwait()+0 000001f4 0000003e 00000000 0d85fd90
opirip()+813 ksvrdp()+0 00000003 0c8ee914 0c935e6c 0c79f2d8
opidrv()+1357 opirip()+0 00000032 00000004 0d85fea8 00000000
sou2o()+45 opidrv()+0 00000032 00000004 0d85fea8 00000000
opimaireal()+263 _sou2o()+0 0d85fe9c 00000032 00000004 0d85fea8
opimai()+92 opimai_real()+0 00000003 0d85fed4 00000000 00000000
BackgroundThreadStart@4()+529 opimai()+0 00000003 0d85ff20 89b057d0 0c790030
77E6482C 00000000 0b18e3c8 00000000 00000000 0b18e3c8
00000000 00000000 00000000 00000000 00000000 00000000
---------------- End of Call Stack Trace -----------------
Call stack acquisition performance stats:
setup time (lock acquis., memory alloc.): 0 ms
frame get time (time the target proc was suspended): 0 ms
symbol translation time: 0 ms
total time: 0 ms
-------------------------------------------------------------------------------
Process diagnostic dump actual duration=0.109000 sec
(max dump time=30.000000 sec)

*** 2011-04-26 00:50:21.509
Waited for process W000 to initialize for 110881 seconds

*** 2011-04-26 01:50:27.697
Process diagnostic dump for W000, OS id=10832
-------------------------------------------------------------------------------
Memory (Avail/Total): Ph:1926M/3839M, Ph+PgF:2566M/4446M, VA:1266M/2047M
CPU Load: 11%
Stack:
------------------- Call Stack Trace ---------------------
calling location entry point arg #1 arg #2 arg #3 arg #4
------------------------------------------- ------------------------------------------- -------- -------- -------- --------
7C82847C 00000000 00000000 00000000 00000000 00000000
77E61C88 77E61C96 000018fc 00001388 00000000 0dd0e55c
_skgpwwait()+128 00000000 000018fc 00001388 004c4b40 0bab7ca0
ksliwat()+2180 skgpwwait()+0 0dd0e5c8 0babb460 2f36f3dc 004c4b40
kslwaitctx()+146 ksliwat()+0 2ea26d20 2ea26d20 000001f4 00000000
kslwait()+158 kslwaitctx()+0 0dd0e7bc 000001f4 0dd0e7bc 06097a59
ksvrdp()+26715 kslwait()+0 000001f4 0000003e 00000000 0dd0fd90
opirip()+813 ksvrdp()+0 00000003 0c54e918 0c565e78 09a2f2dc
opidrv()+1357 opirip()+0 00000032 00000004 0dd0fea8 00000000
sou2o()+45 opidrv()+0 00000032 00000004 0dd0fea8 00000000
opimaireal()+263 _sou2o()+0 0dd0fe9c 00000032 00000004 0dd0fea8
opimai()+92 opimai_real()+0 00000003 0dd0fed4 00000000 00000000
BackgroundThreadStart@4()+529 opimai()+0 00000003 0dd0ff20 89c36268 09a20030
77E6482C 00000000 0b187fb8 00000000 00000000 0b187fb8
00000000 00000000 00000000 00000000 00000000 00000000
---------------- End of Call Stack Trace -----------------
Call stack acquisition performance stats:
setup time (lock acquis., memory alloc.): 0 ms
frame get time (time the target proc was suspended): 0 ms
symbol translation time: 0 ms
total time: 0 ms
-------------------------------------------------------------------------------
Process diagnostic dump actual duration=0.109000 sec
(max dump time=30.000000 sec)

*** 2011-04-26 01:50:27.806
Waited for process W000 to initialize for 112681 seconds

*** 2011-04-26 02:20:27.884
Process diagnostic dump for W000, OS id=9720
-------------------------------------------------------------------------------
Memory (Avail/Total): Ph:1919M/3839M, Ph+PgF:2557M/4446M, VA:1266M/2047M
CPU Load: 43%
Stack:
------------------- Call Stack Trace ---------------------
calling location entry point arg #1 arg #2 arg #3 arg #4
------------------------------------------- ------------------------------------------- -------- -------- -------- --------
7C82847C 00000000 00000000 00000000 00000000 00000000
77E61C88 77E61C96 00001a00 00001388 00000000 0dd0e55c
_skgpwwait()+128 00000000 00001a00 00001388 004c4b40 0bab7ca0
ksliwat()+2180 skgpwwait()+0 0dd0e5c8 0babb460 2f36f3dc 004c4b40
kslwaitctx()+146 ksliwat()+0 2ea26d20 2ea26d20 000001f4 00000000
kslwait()+158 kslwaitctx()+0 0dd0e7bc 000001f4 0dd0e7bc 06097a59
ksvrdp()+26715 kslwait()+0 000001f4 0000003e 00000000 0dd0fd90
opirip()+813 ksvrdp()+0 00000003 0c52e918 0c545e78 09a2f2dc
opidrv()+1357 opirip()+0 00000032 00000004 0dd0fea8 00000000
sou2o()+45 opidrv()+0 00000032 00000004 0dd0fea8 00000000
opimaireal()+263 _sou2o()+0 0dd0fe9c 00000032 00000004 0dd0fea8
opimai()+92 opimai_real()+0 00000003 0dd0fed4 00000000 00000000
BackgroundThreadStart@4()+529 opimai()+0 00000003 0dd0ff20 80828c73 09a20030
77E6482C 00000000 0b187f70 00000000 00000000 0b187f70
00000000 00000000 00000000 00000000 00000000 00000000
---------------- End of Call Stack Trace -----------------
Call stack acquisition performance stats:
setup time (lock acquis., memory alloc.): 0 ms
frame get time (time the target proc was suspended): 0 ms
symbol translation time: 0 ms
total time: 0 ms
-------------------------------------------------------------------------------
Process diagnostic dump actual duration=0.110000 sec
(max dump time=30.000000 sec)

*** 2011-04-26 02:20:27.994
Waited for process W000 to initialize for 115689 seconds

*** 2011-04-26 03:10:36.994
Process diagnostic dump for W000, OS id=10068
-------------------------------------------------------------------------------

*** 2011-04-26 03:10:37.103
Memory (Avail/Total): Ph:1933M/3839M, Ph+PgF:2575M/4446M, VA:1267M/2047M
CPU Load: 32%
Stack:
------------------- Call Stack Trace ---------------------
calling location entry point arg #1 arg #2 arg #3 arg #4
------------------------------------------- ------------------------------------------- -------- -------- -------- --------
7C82847C 00000000 00000000 00000000 00000000 00000000
77E61C88 77E61C96 00001a00 00001388 00000000 0dd0e55c
_skgpwwait()+128 00000000 00001a00 00001388 004c4b40 0d1dceb8
ksliwat()+2180 skgpwwait()+0 0dd0e5c8 0d1e0678 2f36f3dc 004c4b40
kslwaitctx()+146 ksliwat()+0 2ea26d20 2ea26d20 000001f4 00000000
kslwait()+158 kslwaitctx()+0 0dd0e7bc 000001f4 0dd0e7bc 06097a59
ksvrdp()+26715 kslwait()+0 000001f4 0000003e 00000000 0dd0fd90
opirip()+813 ksvrdp()+0 00000003 0c52e918 0c545e78 09a2f2dc
opidrv()+1357 opirip()+0 00000032 00000004 0dd0fea8 00000000
sou2o()+45 opidrv()+0 00000032 00000004 0dd0fea8 00000000
opimaireal()+263 _sou2o()+0 0dd0fe9c 00000032 00000004 0dd0fea8
opimai()+92 opimai_real()+0 00000003 0dd0fed4 00000000 00000000
BackgroundThreadStart@4()+529 opimai()+0 00000003 0dd0ff20 8a7c5db0 09a20030
77E6482C 00000000 0b1a9bc0 00000000 00000000 0b1a9bc0
00000000 00000000 00000000 00000000 00000000 00000000
---------------- End of Call Stack Trace -----------------
Call stack acquisition performance stats:
setup time (lock acquis., memory alloc.): 0 ms
frame get time (time the target proc was suspended): 0 ms
symbol translation time: 0 ms
total time: 0 ms
-------------------------------------------------------------------------------
Process diagnostic dump actual duration=0.109000 sec
(max dump time=30.000000 sec)

*** 2011-04-26 03:10:37.103
error 2103 detected in background process
OPIRIP: Uncaught error 447. Error stack:

*** 2011-04-26 03:35:37.103
ORA-00447: fatal error in background process
ORA-02103: PCC: inconsistent cursor cache (out-of-range cuc ref)

Edited by: user8952784 on Apr 25, 2011 12:53 PM