Skip to Main Content

Oracle Database Discussions

Announcement

For appeals, questions and feedback about Oracle Forums, please email oracle-forums-moderators_us@oracle.com. Technical questions should be asked in the appropriate category. Thank you!

Connected to an idle instance, while database instance is running

DBA092012Nov 30 2015 — edited Dec 2 2015

Hello everybody.

I've been using Oracle RDBMS 11.2.0.4 and Oracle Linux 6.7

$ uname -r

3.8.13-118.el6uek.x86_64

I've been having the problem on a production server the third time during a couple of months .

The problem appeared on Mon Nov 30 11:05:35 (A.M.) 2015 and this is reflected in the alert log in the way described below

Mon Nov 30 11:05:35 2015

Process W003 died, see its trace file

Mon Nov 30 11:05:48 2015

Process J000 died, see its trace file

kkjcre1p: unable to spawn jobq slave process

Errors in file /u01/app/oracle/diag/rdbms/prim_pay/payment/trace/payment_cjq0_16651.trc:

Process J000 died, see its trace file

kkjcre1p: unable to spawn jobq slave process

Errors in file /u01/app/oracle/diag/rdbms/prim_pay/payment/trace/payment_cjq0_16651.trc:

Mon Nov 30 11:05:55 2015

Process W006 died, see its trace file

Process J000 died, see its trace file

I've been getting the errors similar to the above since now. Our company apps connected to the instace still running but the instance is unavailable.

The problem is partly similar to the problem described in the Doc ID 1578491.1 but not the same!


So here are some outputs from the different commands

----

$ sqlplus / as sysdba

SQL*Plus: Release 11.2.0.4.0 Production on Mon Nov 30 13:05:00 2015

Copyright (c) 1982, 2013, Oracle.  All rights reserved.

Connected to an idle instance.

SQL>

---------------------------------------------------------------------------------------------

$ sysresv

IPC Resources for ORACLE_SID "payment" :

Shared Memory

ID              KEY

No shared memory segments used

Semaphores:

ID              KEY

589836          0x8a63c9e0

622605          0x8a63c9e1

655374          0x8a63c9e2

688143          0x8a63c9e3

720912          0x8a63c9e4

753681          0x8a63c9e5

786450          0x8a63c9e6

819219          0x8a63c9e7

851988          0x8a63c9e8

Oracle Instance not alive for sid "payment"

----------------------

sar -A -f /var/log/sa/sa30 -s 10:30:01 -e 11:40:01| less

10:30:01 AM    proc/s   cswch/s

10:40:01 AM      0.87   9417.37

10:50:01 AM      0.82   9251.64

11:00:01 AM      1.12   8989.49

11:10:01 AM      1.29   9258.67

11:20:01 AM      1.75   8734.31

11:30:01 AM      1.88   8697.01

11:40:01 AM      1.78   8728.29

Average:         1.36   9010.86

10:30:01 AM  pgpgin/s pgpgout/s   fault/s  majflt/s  pgfree/s pgscank/s pgscand/s pgsteal/s    %vmeff

10:40:01 AM   3643.60   4199.19  15078.22      0.00  12260.08      0.00      0.00      0.00      0.00

10:50:01 AM   3989.61   4451.42  14439.07      0.00  11528.82      0.00      0.00      0.00      0.00

11:00:01 AM   1920.99   2437.58  18695.42      0.00  11005.53      0.00      0.00      0.00      0.00

11:10:01 AM   2045.93   2586.25  19002.76      0.00  11536.91      0.00      0.00      0.00      0.00

11:20:01 AM   2278.71   2807.81  16465.08      0.00  11495.41      0.00      0.00      0.00      0.00

11:30:01 AM   4638.37   5131.80  17111.84      0.00  12677.36      0.00      0.00      0.00      0.00

11:40:01 AM   4173.46   4756.59  15830.98      0.00  11915.80      0.00      0.00      0.00      0.00

Average:      3241.74   3767.45  16660.60      0.00  11774.37      0.00      0.00      0.00      0.00

10:30:01 AM       tps      rtps      wtps   bread/s   bwrtn/s

10:40:01 AM    628.65    154.29    474.37   7287.20   8398.39

10:50:01 AM    609.71    158.86    450.85   7979.22   8902.83

11:00:01 AM    563.51    133.68    429.83   3841.98   4875.17

11:10:01 AM    596.27    150.61    445.65   4091.86   5172.51

11:20:01 AM    537.90    131.84    406.06   4557.43   5615.62

11:30:01 AM    558.74    138.73    420.01   9276.73  10263.60

11:40:01 AM    546.84    133.98    412.86   8346.92   9513.19

Average:       577.36    143.14    434.23   6483.47   7534.89

10:30:01 AM   frmpg/s   bufpg/s   campg/s

10:40:01 AM      3.52      0.01      0.06

10:50:01 AM     -6.85      0.02      0.06

11:00:01 AM   -414.73      0.02      0.30

11:10:01 AM   -210.63      0.07      0.56

11:20:01 AM    -71.48      0.07      0.72

11:30:01 AM    -44.02      0.07      0.79

11:40:01 AM    -46.93      0.07      0.75

Average:      -113.02      0.05      0.46

10:30:01 AM kbmemfree kbmemused  %memused kbbuffers  kbcached  kbcommit   %commit

10:40:01 AM   9565856  23085564     70.70    343656  16210648  19229228     38.90

10:50:01 AM   9549592  23101828     70.75    343712  16210780  19229332     38.90

11:00:01 AM   8564332  24087088     73.77    343756  16211496  20078688     40.62

11:10:01 AM   8064188  24587232     75.30    343916  16212832  20134836     40.74

11:20:01 AM   7894396  24757024     75.82    344092  16214532  20139904     40.75

11:30:01 AM   7789740  24861680     76.14    344252  16216420  20123968     40.71

11:40:01 AM   7678212  24973208     76.48    344428  16218196  20145292     40.76

Average:      8443759  24207661     74.14    343973  16213558  19868750     40.20

10:30:01 AM dentunusd   file-nr  inode-nr    pty-nr

10:40:01 AM     54978    119616     62281         0

10:50:01 AM     54972    118976     62285         0

11:00:01 AM     55266    154176     63074         0

11:10:01 AM     55440    155200     63505         0

11:20:01 AM     55754    154240     63838         0

11:30:01 AM     56100    153664     64212         2

11:40:01 AM     56904    153920     65017         2

Average:        55631    144256     63459         1

10:30:01 AM   runq-sz  plist-sz   ldavg-1   ldavg-5  ldavg-15

10:40:01 AM         2       748      1.03      0.85      0.89

10:50:01 AM         2       744      0.77      0.88      0.90

11:00:01 AM         0       822      0.83      1.03      0.96

11:10:01 AM         0       827      0.46      0.67      0.83

11:20:01 AM         4       829      0.99      0.97      0.90

11:30:01 AM         3       831      1.17      1.08      0.97

11:40:01 AM         3       839      0.91      0.99      0.99

Average:            2       806      0.88      0.92      0.92

10:30:01 AM       DEV       tps  rd_sec/s  wr_sec/s  avgrq-sz  avgqu-sz     await     svctm     %util

10:40:01 AM   dev8-16    183.50      4.91   1640.59      8.97      0.01      0.06      0.05      0.99

10:40:01 AM    dev8-0    445.15   7282.32   6757.80     31.54      0.22      0.50      0.30     13.19

10:50:01 AM   dev8-16    183.33      9.58   4668.85     25.52      0.39      2.10      0.08      1.53

10:50:01 AM    dev8-0    426.37   7969.61   4233.98     28.62      0.19      0.44      0.27     11.35

11:00:01 AM   dev8-16    171.57      4.62   1456.50      8.52      0.01      0.05      0.05      0.85

11:00:01 AM    dev8-0    391.94   3837.36   3418.67     18.51      0.13      0.33      0.24      9.40

11:10:01 AM   dev8-16    178.32      4.70   1677.36      9.43      0.01      0.05      0.05      0.87

11:10:01 AM    dev8-0    417.95   4087.16   3495.15     18.14      0.15      0.35      0.25     10.35

11:20:01 AM   dev8-16    156.94      8.89   1426.07      9.14      0.01      0.06      0.05      0.85

11:20:01 AM    dev8-0    380.96   4548.54   4189.55     22.94      0.13      0.34      0.24      8.99

11:30:01 AM   dev8-16    154.75      3.88   1340.22      8.69      0.01      0.05      0.05      0.80

11:30:01 AM    dev8-0    403.99   9272.86   8923.38     45.04      0.21      0.52      0.29     11.66

11:40:01 AM   dev8-16    164.74      4.81   4538.23     27.58      0.39      2.35      0.09      1.47

11:40:01 AM    dev8-0    382.10   8342.11   4974.96     34.85      0.18      0.47      0.24      9.31

Average:      dev8-16    170.45      5.91   2392.50     14.07      0.12      0.69      0.06      1.05

Average:       dev8-0    406.92   6477.56   5142.39     28.56      0.17      0.42      0.26     10.61

10:30:01 AM     IFACE   rxpck/s   txpck/s    rxkB/s    txkB/s   rxcmp/s   txcmp/s  rxmcst/s

10:40:01 AM      eth0     96.04    297.25      6.20    377.23      0.00      0.00      0.00

10:40:01 AM      eth1      0.00      0.00      0.00      0.00      0.00      0.00      0.00

10:40:01 AM      eth2   2265.27   2138.20   1141.38   1629.97      0.00      0.00      1.10

10:40:01 AM      eth3      0.00      0.00      0.00      0.00      0.00      0.00      0.00

10:40:01 AM        lo      0.74      0.74      0.09      0.09      0.00      0.00      0.00

10:50:01 AM      eth0     91.50    284.33      5.91    361.18      0.00      0.00      0.00

10:50:01 AM      eth1      0.00      0.00      0.00      0.00      0.00      0.00      0.00

10:50:01 AM      eth2   2246.57   2136.91   1119.90   1651.80      0.00      0.00      1.09

10:50:01 AM      eth3      0.00      0.00      0.00      0.00      0.00      0.00      0.00

10:50:01 AM        lo      0.67      0.67      0.06      0.06      0.00      0.00      0.00

11:00:01 AM      eth0     89.75    274.82      5.79    347.54      0.00      0.00      0.00

11:00:01 AM      eth1      0.00      0.00      0.00      0.00      0.00      0.00      0.00

11:00:01 AM      eth2   2092.42   1972.65   1045.88   1490.20      0.00      0.00      1.09

11:00:01 AM      eth3      0.00      0.00      0.00      0.00      0.00      0.00      0.00

11:00:01 AM        lo      0.82      0.82      0.13      0.13      0.00      0.00      0.00

11:10:01 AM      eth0     92.86    291.71      5.99    371.51      0.00      0.00      0.00

11:10:01 AM      eth1      0.00      0.00      0.00      0.00      0.00      0.00      0.00

11:10:01 AM      eth2   2239.63   2131.28   1117.57   1655.16      0.00      0.00      1.09

11:10:01 AM      eth3      0.00      0.00      0.00      0.00      0.00      0.00      0.00

11:10:01 AM        lo      0.66      0.66      0.05      0.05      0.00      0.00      0.00

11:20:01 AM      eth0     82.45    252.70      5.32    320.60      0.00      0.00      0.00

11:20:01 AM      eth1      0.00      0.00      0.00      0.00      0.00      0.00      0.00

11:20:01 AM      eth2   2015.92   1910.65   1005.81   1482.86      0.00      0.00      1.09

11:20:01 AM      eth3      0.00      0.00      0.00      0.00      0.00      0.00      0.00

11:20:01 AM        lo      0.65      0.65      0.05      0.05      0.00      0.00      0.00

11:30:01 AM      eth0     81.54    250.58      5.26    318.07      0.00      0.00      0.00

11:30:01 AM      eth1      0.00      0.00      0.00      0.00      0.00      0.00      0.00

11:30:01 AM      eth2   1944.96   1843.36    964.56   1421.18      0.00      0.00      1.10

11:30:01 AM      eth3      0.00      0.00      0.00      0.00      0.00      0.00      0.00

11:30:01 AM        lo      0.65      0.65      0.05      0.05      0.00      0.00      0.00

11:40:01 AM      eth0     82.27    253.94      5.31    322.39      0.00      0.00      0.00

11:40:01 AM      eth1      0.00      0.00      0.00      0.00      0.00      0.00      0.00

11:40:01 AM      eth2   1976.77   1858.41    998.89   1423.29      0.00      0.00      1.09

11:40:01 AM      eth3      0.00      0.00      0.00      0.00      0.00      0.00      0.00

11:40:01 AM        lo      0.64      0.64      0.05      0.05      0.00      0.00      0.00

Average:         eth0     88.05    272.18      5.69    345.49      0.00      0.00      0.00

Average:         eth1      0.00      0.00      0.00      0.00      0.00      0.00      0.00

Average:         eth2   2111.60   1998.73   1056.26   1536.31      0.00      0.00      1.09

Average:         eth3      0.00      0.00      0.00      0.00      0.00      0.00      0.00

Average:           lo      0.69      0.69      0.07      0.07      0.00      0.00      0.00

10:30:01 AM    totsck    tcpsck    udpsck    rawsck   ip-frag    tcp-tw

10:40:01 AM      1938        18         9         0         0        40

10:50:01 AM      1954        18         9         0         0        38

11:00:01 AM      2287        18         9         0         0        48

11:10:01 AM      2544        18         9         0         0        40

11:20:01 AM      2570        18         9         0         0        40

11:30:01 AM      2609        20         9         0         0        41

11:40:01 AM      2616        20         9         0         0        40

Average:         2360        19         9         0         0        41

----------

In addition want to highlight that i took some steps as described in Doc ID 1578491.1 I removed prelink package completely both from the server and from the crontab but the problem still persists.


Could someone please shed some light on the problem?

Comments
Locked Post
New comments cannot be posted to this locked post.
Post Details
Locked on Dec 30 2015
Added on Nov 30 2015
48 comments
10,437 views