Hi, everyone!
I had an instance failure last night and one more this morning. It was saying that TNS connection closed. It havn't happened for 2 years at least.
I had to restart database. I didn't change any db parameters recently.
The log alert is saying
Process J006 died, see its trace file
Sat Dec 28 03:53:00 2013
kkjcre1p: unable to spawn jobq slave process
Sat Dec 28 03:53:00 2013
Errors in file /opt/u01/app/oracle/admin/orcl/bdump/orcl_cjq0_6034.trc:
Sat Dec 28 03:54:01 2013
Process J006 died, see its trace file
Sat Dec 28 03:54:01 2013
kkjcre1p: unable to spawn jobq slave process
Sat Dec 28 03:54:01 2013
Errors in file /opt/u01/app/oracle/admin/orcl/bdump/orcl_cjq0_6034.trc:
Sat Dec 28 03:54:08 2013
Shutting down instance (abort)
License high water mark = 981
Instance terminated by USER, pid = 22021
Sat Dec 28 03:54:19 2013
Starting ORACLE instance (normal)
LICENSE_MAX_SESSION = 0
LICENSE_SESSIONS_WARNING = 0
Picked latch-free SCN scheme 3
Autotune of undo retention is turned on.
IMODE=BR
ILAT =121
LICENSE_MAX_USERS = 0
SYS auditing is disabled
ksdpec: called for event 13740 prior to event group initialization
Starting up ORACLE RDBMS Version: 10.2.0.4.0.
System parameters with non-default values:
processes = 1000
timed_statistics = TRUE
sga_max_size = 20971520000
__shared_pool_size = 6358564864
__large_pool_size = 16777216
__java_pool_size = 67108864
__streams_pool_size = 16777216
sga_target = 20971520000
control_files = /opt/u04/oradata/orcl/control01.ctl, /opt/u05/oradata/orcl/control02.ctl, /opt/u06/oradata/orcl/control03.ctl
db_block_size = 8192
__db_cache_size = 14495514624
compatible = 10.2.0.0.0
log_archive_dest = /opt/u01/app/oracle/admin/orcl/arch
db_file_multiblock_read_count= 32
fast_start_mttr_target = 300
undo_management = AUTO
undo_tablespace = UNDOTBS1
undo_retention = 10800
remote_login_passwordfile= EXCLUSIVE
db_domain = Pavlodar
instance_name = orcl
utl_file_dir = /opt/u01/app/oracle/admin/orcl/utl_file
job_queue_processes = 10
parallel_max_servers = 0
background_dump_dest = /opt/u01/app/oracle/admin/orcl/bdump
user_dump_dest = /opt/u01/app/oracle/admin/orcl/udump
core_dump_dest = /opt/u01/app/oracle/admin/orcl/cdump
db_name = orcl
open_cursors = 300
star_transformation_enabled= FALSE
query_rewrite_enabled = FALSE
pga_aggregate_target = 7340032000
I looked into file orcl_cjq0_6034.trc, it is saying
Process J006 is dead (pid=21807, state=3):
*** 2013-12-28 03:46:54.515
Process J006 is dead (pid=21809, state=3):
*** 2013-12-28 03:47:55.540
Process J006 is dead (pid=21879, state=3):
*** 2013-12-28 03:48:56.566
Process J006 is dead (pid=21914, state=3):
*** 2013-12-28 03:49:57.590
Process J006 is dead (pid=21940, state=3):
*** 2013-12-28 03:50:58.619
Process J006 is dead (pid=22027, state=3):
*** 2013-12-28 03:51:59.714
Process J006 is dead (pid=22061, state=3):
*** 2013-12-28 03:53:00.740
Process J006 is dead (pid=22133, state=3):
*** 2013-12-28 03:54:01.765
Process J006 is dead (pid=22155, state=3):
The last information was not really informative for me. Can you, guys, tell me what happens here?
I am thinking about "License high water mark = 981", can it be the reason of instance failure, since it too close my processes and sessions parameters?
Its 48GB memory server with 20 + 7 GB dedicated to Oracle DB.
Any help would be appreciated, thanks!