Hello everybody.
I've been using Oracle RDBMS 11.2.0.4 and Oracle Linux 6.7
$ uname -r
3.8.13-118.el6uek.x86_64
I've been having the problem on a production server the third time during a couple of months .
The problem appeared on Mon Nov 30 11:05:35 (A.M.) 2015 and this is reflected in the alert log in the way described below
Mon Nov 30 11:05:35 2015
Process W003 died, see its trace file
Mon Nov 30 11:05:48 2015
Process J000 died, see its trace file
kkjcre1p: unable to spawn jobq slave process
Errors in file /u01/app/oracle/diag/rdbms/prim_pay/payment/trace/payment_cjq0_16651.trc:
Process J000 died, see its trace file
kkjcre1p: unable to spawn jobq slave process
Errors in file /u01/app/oracle/diag/rdbms/prim_pay/payment/trace/payment_cjq0_16651.trc:
Mon Nov 30 11:05:55 2015
Process W006 died, see its trace file
Process J000 died, see its trace file
I've been getting the errors similar to the above since now. Our company apps connected to the instace still running but the instance is unavailable.
The problem is partly similar to the problem described in the Doc ID 1578491.1 but not the same!
So here are some outputs from the different commands
----
$ sqlplus / as sysdba
SQL*Plus: Release 11.2.0.4.0 Production on Mon Nov 30 13:05:00 2015
Copyright (c) 1982, 2013, Oracle. All rights reserved.
Connected to an idle instance.
SQL>
---------------------------------------------------------------------------------------------
$ sysresv
IPC Resources for ORACLE_SID "payment" :
Shared Memory
ID KEY
No shared memory segments used
Semaphores:
ID KEY
589836 0x8a63c9e0
622605 0x8a63c9e1
655374 0x8a63c9e2
688143 0x8a63c9e3
720912 0x8a63c9e4
753681 0x8a63c9e5
786450 0x8a63c9e6
819219 0x8a63c9e7
851988 0x8a63c9e8
Oracle Instance not alive for sid "payment"
----------------------
sar -A -f /var/log/sa/sa30 -s 10:30:01 -e 11:40:01| less
10:30:01 AM proc/s cswch/s
10:40:01 AM 0.87 9417.37
10:50:01 AM 0.82 9251.64
11:00:01 AM 1.12 8989.49
11:10:01 AM 1.29 9258.67
11:20:01 AM 1.75 8734.31
11:30:01 AM 1.88 8697.01
11:40:01 AM 1.78 8728.29
Average: 1.36 9010.86
10:30:01 AM pgpgin/s pgpgout/s fault/s majflt/s pgfree/s pgscank/s pgscand/s pgsteal/s %vmeff
10:40:01 AM 3643.60 4199.19 15078.22 0.00 12260.08 0.00 0.00 0.00 0.00
10:50:01 AM 3989.61 4451.42 14439.07 0.00 11528.82 0.00 0.00 0.00 0.00
11:00:01 AM 1920.99 2437.58 18695.42 0.00 11005.53 0.00 0.00 0.00 0.00
11:10:01 AM 2045.93 2586.25 19002.76 0.00 11536.91 0.00 0.00 0.00 0.00
11:20:01 AM 2278.71 2807.81 16465.08 0.00 11495.41 0.00 0.00 0.00 0.00
11:30:01 AM 4638.37 5131.80 17111.84 0.00 12677.36 0.00 0.00 0.00 0.00
11:40:01 AM 4173.46 4756.59 15830.98 0.00 11915.80 0.00 0.00 0.00 0.00
Average: 3241.74 3767.45 16660.60 0.00 11774.37 0.00 0.00 0.00 0.00
10:30:01 AM tps rtps wtps bread/s bwrtn/s
10:40:01 AM 628.65 154.29 474.37 7287.20 8398.39
10:50:01 AM 609.71 158.86 450.85 7979.22 8902.83
11:00:01 AM 563.51 133.68 429.83 3841.98 4875.17
11:10:01 AM 596.27 150.61 445.65 4091.86 5172.51
11:20:01 AM 537.90 131.84 406.06 4557.43 5615.62
11:30:01 AM 558.74 138.73 420.01 9276.73 10263.60
11:40:01 AM 546.84 133.98 412.86 8346.92 9513.19
Average: 577.36 143.14 434.23 6483.47 7534.89
10:30:01 AM frmpg/s bufpg/s campg/s
10:40:01 AM 3.52 0.01 0.06
10:50:01 AM -6.85 0.02 0.06
11:00:01 AM -414.73 0.02 0.30
11:10:01 AM -210.63 0.07 0.56
11:20:01 AM -71.48 0.07 0.72
11:30:01 AM -44.02 0.07 0.79
11:40:01 AM -46.93 0.07 0.75
Average: -113.02 0.05 0.46
10:30:01 AM kbmemfree kbmemused %memused kbbuffers kbcached kbcommit %commit
10:40:01 AM 9565856 23085564 70.70 343656 16210648 19229228 38.90
10:50:01 AM 9549592 23101828 70.75 343712 16210780 19229332 38.90
11:00:01 AM 8564332 24087088 73.77 343756 16211496 20078688 40.62
11:10:01 AM 8064188 24587232 75.30 343916 16212832 20134836 40.74
11:20:01 AM 7894396 24757024 75.82 344092 16214532 20139904 40.75
11:30:01 AM 7789740 24861680 76.14 344252 16216420 20123968 40.71
11:40:01 AM 7678212 24973208 76.48 344428 16218196 20145292 40.76
Average: 8443759 24207661 74.14 343973 16213558 19868750 40.20
10:30:01 AM dentunusd file-nr inode-nr pty-nr
10:40:01 AM 54978 119616 62281 0
10:50:01 AM 54972 118976 62285 0
11:00:01 AM 55266 154176 63074 0
11:10:01 AM 55440 155200 63505 0
11:20:01 AM 55754 154240 63838 0
11:30:01 AM 56100 153664 64212 2
11:40:01 AM 56904 153920 65017 2
Average: 55631 144256 63459 1
10:30:01 AM runq-sz plist-sz ldavg-1 ldavg-5 ldavg-15
10:40:01 AM 2 748 1.03 0.85 0.89
10:50:01 AM 2 744 0.77 0.88 0.90
11:00:01 AM 0 822 0.83 1.03 0.96
11:10:01 AM 0 827 0.46 0.67 0.83
11:20:01 AM 4 829 0.99 0.97 0.90
11:30:01 AM 3 831 1.17 1.08 0.97
11:40:01 AM 3 839 0.91 0.99 0.99
Average: 2 806 0.88 0.92 0.92
10:30:01 AM DEV tps rd_sec/s wr_sec/s avgrq-sz avgqu-sz await svctm %util
10:40:01 AM dev8-16 183.50 4.91 1640.59 8.97 0.01 0.06 0.05 0.99
10:40:01 AM dev8-0 445.15 7282.32 6757.80 31.54 0.22 0.50 0.30 13.19
10:50:01 AM dev8-16 183.33 9.58 4668.85 25.52 0.39 2.10 0.08 1.53
10:50:01 AM dev8-0 426.37 7969.61 4233.98 28.62 0.19 0.44 0.27 11.35
11:00:01 AM dev8-16 171.57 4.62 1456.50 8.52 0.01 0.05 0.05 0.85
11:00:01 AM dev8-0 391.94 3837.36 3418.67 18.51 0.13 0.33 0.24 9.40
11:10:01 AM dev8-16 178.32 4.70 1677.36 9.43 0.01 0.05 0.05 0.87
11:10:01 AM dev8-0 417.95 4087.16 3495.15 18.14 0.15 0.35 0.25 10.35
11:20:01 AM dev8-16 156.94 8.89 1426.07 9.14 0.01 0.06 0.05 0.85
11:20:01 AM dev8-0 380.96 4548.54 4189.55 22.94 0.13 0.34 0.24 8.99
11:30:01 AM dev8-16 154.75 3.88 1340.22 8.69 0.01 0.05 0.05 0.80
11:30:01 AM dev8-0 403.99 9272.86 8923.38 45.04 0.21 0.52 0.29 11.66
11:40:01 AM dev8-16 164.74 4.81 4538.23 27.58 0.39 2.35 0.09 1.47
11:40:01 AM dev8-0 382.10 8342.11 4974.96 34.85 0.18 0.47 0.24 9.31
Average: dev8-16 170.45 5.91 2392.50 14.07 0.12 0.69 0.06 1.05
Average: dev8-0 406.92 6477.56 5142.39 28.56 0.17 0.42 0.26 10.61
10:30:01 AM IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s
10:40:01 AM eth0 96.04 297.25 6.20 377.23 0.00 0.00 0.00
10:40:01 AM eth1 0.00 0.00 0.00 0.00 0.00 0.00 0.00
10:40:01 AM eth2 2265.27 2138.20 1141.38 1629.97 0.00 0.00 1.10
10:40:01 AM eth3 0.00 0.00 0.00 0.00 0.00 0.00 0.00
10:40:01 AM lo 0.74 0.74 0.09 0.09 0.00 0.00 0.00
10:50:01 AM eth0 91.50 284.33 5.91 361.18 0.00 0.00 0.00
10:50:01 AM eth1 0.00 0.00 0.00 0.00 0.00 0.00 0.00
10:50:01 AM eth2 2246.57 2136.91 1119.90 1651.80 0.00 0.00 1.09
10:50:01 AM eth3 0.00 0.00 0.00 0.00 0.00 0.00 0.00
10:50:01 AM lo 0.67 0.67 0.06 0.06 0.00 0.00 0.00
11:00:01 AM eth0 89.75 274.82 5.79 347.54 0.00 0.00 0.00
11:00:01 AM eth1 0.00 0.00 0.00 0.00 0.00 0.00 0.00
11:00:01 AM eth2 2092.42 1972.65 1045.88 1490.20 0.00 0.00 1.09
11:00:01 AM eth3 0.00 0.00 0.00 0.00 0.00 0.00 0.00
11:00:01 AM lo 0.82 0.82 0.13 0.13 0.00 0.00 0.00
11:10:01 AM eth0 92.86 291.71 5.99 371.51 0.00 0.00 0.00
11:10:01 AM eth1 0.00 0.00 0.00 0.00 0.00 0.00 0.00
11:10:01 AM eth2 2239.63 2131.28 1117.57 1655.16 0.00 0.00 1.09
11:10:01 AM eth3 0.00 0.00 0.00 0.00 0.00 0.00 0.00
11:10:01 AM lo 0.66 0.66 0.05 0.05 0.00 0.00 0.00
11:20:01 AM eth0 82.45 252.70 5.32 320.60 0.00 0.00 0.00
11:20:01 AM eth1 0.00 0.00 0.00 0.00 0.00 0.00 0.00
11:20:01 AM eth2 2015.92 1910.65 1005.81 1482.86 0.00 0.00 1.09
11:20:01 AM eth3 0.00 0.00 0.00 0.00 0.00 0.00 0.00
11:20:01 AM lo 0.65 0.65 0.05 0.05 0.00 0.00 0.00
11:30:01 AM eth0 81.54 250.58 5.26 318.07 0.00 0.00 0.00
11:30:01 AM eth1 0.00 0.00 0.00 0.00 0.00 0.00 0.00
11:30:01 AM eth2 1944.96 1843.36 964.56 1421.18 0.00 0.00 1.10
11:30:01 AM eth3 0.00 0.00 0.00 0.00 0.00 0.00 0.00
11:30:01 AM lo 0.65 0.65 0.05 0.05 0.00 0.00 0.00
11:40:01 AM eth0 82.27 253.94 5.31 322.39 0.00 0.00 0.00
11:40:01 AM eth1 0.00 0.00 0.00 0.00 0.00 0.00 0.00
11:40:01 AM eth2 1976.77 1858.41 998.89 1423.29 0.00 0.00 1.09
11:40:01 AM eth3 0.00 0.00 0.00 0.00 0.00 0.00 0.00
11:40:01 AM lo 0.64 0.64 0.05 0.05 0.00 0.00 0.00
Average: eth0 88.05 272.18 5.69 345.49 0.00 0.00 0.00
Average: eth1 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Average: eth2 2111.60 1998.73 1056.26 1536.31 0.00 0.00 1.09
Average: eth3 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Average: lo 0.69 0.69 0.07 0.07 0.00 0.00 0.00
10:30:01 AM totsck tcpsck udpsck rawsck ip-frag tcp-tw
10:40:01 AM 1938 18 9 0 0 40
10:50:01 AM 1954 18 9 0 0 38
11:00:01 AM 2287 18 9 0 0 48
11:10:01 AM 2544 18 9 0 0 40
11:20:01 AM 2570 18 9 0 0 40
11:30:01 AM 2609 20 9 0 0 41
11:40:01 AM 2616 20 9 0 0 40
Average: 2360 19 9 0 0 41
----------
In addition want to highlight that i took some steps as described in Doc ID 1578491.1 I removed prelink package completely both from the server and from the crontab but the problem still persists.
Could someone please shed some light on the problem?