kkjcre1p: unable to spawn jobq slave process
Dear Experts :
our production Oracle DB(10.0.2.4) was hanging this morning.But the wiered thing is
Then I check alter.log. I found
" kkjcre1p: unable to spawn jobq slave process " message shows up
and serval
"ORA-3136" come after this error reported.
as below
---------------------------------------------------------------
alert log :
Tue Jul 13 09:15:17 2010
kkjcre1p: unable to spawn jobq slave process
Tue Jul 13 09:15:17 2010
Errors in file /oradata/db10g/acepro10/admin/bdump/acepro10_cjq0_1044634.trc:
Tue Jul 13 09:19:25 2010
WARNING: inbound connection timed out (ORA-3136)
Tue Jul 13 09:19:25 2010
WARNING: inbound connection timed out (ORA-3136)
Tue Jul 13 09:19:25 2010
WARNING: inbound connection timed out (ORA-3136)
............................
........................
----------------------------------------------------------------
to check for the content of tracefile ,
as below
-----------------------------------------------------------------
/oradata/db10g/acepro10/admin/bdump/acepro10_cjq0_1044634.trc
Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options
ORACLE_HOME = /oradata/db10g/oracle10g
System name: AIX
Node name: twtplbdb001
Release: 2
Version: 5
Machine:
Instance name: acepro10
Redo thread mounted by this instance: 1
Oracle process number: 10
Unix process pid: 1044634, image: oracle@twtplbdb001 (CJQ0)
*** 2010-07-13 03:19:10.254
*** 2010-07-13 09:13:57.906
Waited for process J002 to initialize for 60 seconds
*** 2010-07-13 09:13:57.934
Dumping diagnostic information for J002:
OS pid = 958564
loadavg : 0.11 0.09 0.17
swap info: free_mem = 10.86M rsv = 48.00M
alloc = 3329.84M avail = 12288.00M swap_free = 8958.16M
F S UID PID PPID C PRI NI ADDR SZ WCHAN STIME TTY TIME CMD
240001 A oracle3 958564 1 0 60 20 2ae0a400 92392 09:12:57 - 0:00 ora_j002_acepro10
958564: ora_j002_acepro10
0x00000001000fc7e0 sskgpwwait(??, ??, ??, ??, ??) + ??
0x00000001000f9dfc skgpwwait(??, ??, ??, ??, ??) + 0xbc
0x000000010011db2c kslges(??, ??, ??, ??, ??) + 0x54c
0x000000010012195c kslgetl(??, ??, ??, ??) + 0x33c
0x00000001049f0978 ksfglt(??, ??, ??, ??, ??) + 0x198
0x00000001002fe6ec kgllkalt(??, ??) + 0x8c
0x00000001002f4efc kglssi(??, ??, ??, ??) + 0x5c
0x000000010013fb40 ksucri(??, ??) + 0x560
0x0000000100140420 ksucre(??) + 0xc0
0x0000000103081ab0 kkjrdp() + 0x350
0x000000010430c190 opirip(??, ??, ??) + 0x4f0
0x0000000102d9a598 opidrv(??, ??, ??) + 0x458
0x000000010370b0b0 sou2o(??, ??, ??, ??) + 0x90
0x0000000100000870 opimai_real(??, ??) + 0x150
0x00000001000006d8 main(??, ??) + 0x98
0x0000000100000360 __start() + 0x90
.........................
......................
increased by seconds..
.....................
" Waited for process J002 to initialize for 70 seconds "
"Waited for process J002 to initialize for 80 seconds"
"Waited for process J002 to initialize for 90 seconds"
.................
........up to 120 seconds.
----------------------------------------------------------------
I've google for answer about the reason of cause. some said i may be process overloaded problem..
i check for process parameter current value is 1000, and avg process loading is up to less then 100.
does anyone knows what happen with that ?