Hi,
We have three databases on one HP-UX 11.31 server, the DB version is 10.2.0.4, we started facing slowness in all the databases, when I tried to connect to the server it was very slow, even the sqlplus was taking very long time, when checked the alert logs of all the databases, found the below error in each alert log file;
ksvcreate: Process(m000) creation failed
Sun Apr 10 12:56:56 2011
ksvcreate: Process(m000) creation failed
Sun Apr 10 12:59:00 2011
The server resource utilization was fine with almost 15%CPU utilization, 55% Memory utilization.
I aborted the databases then tried to start them, the startup failed with the below error for all databases;
ORA-00445: background process "PMON" did not start after 120 seconds
The following is the alert log content for one database;
ksvcreate: Process(m000) creation failed
Sun Apr 10 13:09:17 2011
ksvcreate: Process(m000) creation failed
Sun Apr 10 13:24:29 2011
Starting ORACLE instance (normal)
LICENSE_MAX_SESSION = 0
LICENSE_SESSIONS_WARNING = 0
WARNING: Could not create SMR
Picked latch-free SCN scheme 3
Using LOG_ARCHIVE_DEST_10 parameter default value as USE_DB_RECOVERY_FILE_DEST
Autotune of undo retention is turned on.
IMODE=BR
ILAT =242
LICENSE_MAX_USERS = 0
SYS auditing is disabled
Sun Apr 10 13:27:10 2011
ksdpec: called for event 13740 prior to event group initialization
Starting up ORACLE RDBMS Version: 10.2.0.4.0.
System parameters with non-default values:
processes = 2000
sga_max_size = 3221225472
__shared_pool_size = 1828716544
__large_pool_size = 16777216
__java_pool_size = 16777216
__streams_pool_size = 16777216
nls_date_format = dd-mm-yyyy
nls_calendar = Arabic Hijrah
filesystemio_options = NONE
disk_asynch_io = FALSE
sga_target = 3221225472
control_files = /oradb/oradata/centl/control01.ctl, /oradb/oradat
a/cent/control02.ctl, /oradb/oradata/cent/control03.ctl
db_block_size = 8192
__db_cache_size = 1325400064
db_writer_processes = 4
compatible = 10.2.0.3.0
db_file_multiblock_read_count= 16
db_recovery_file_dest = /arch
db_recovery_file_dest_size= 32212254720
undo_management = AUTO
undo_tablespace = UNDOTBS1
remote_login_passwordfile= EXCLUSIVE
db_domain =
dispatchers = (PROTOCOL=TCP) (SERVICE=centlXDB)
smtp_out_server = 10.241.16.118:25
job_queue_processes = 10
background_dump_dest = /u01/app/oracle/admin/cent/bdump
user_dump_dest = /u01/app/oracle/admin/centl/udump
max_dump_file_size = 1024
core_dump_dest = /u01/app/oracle/admin/cent/cdump
audit_file_dest = /u01/app/oracle/admin/cent/adump
db_name = cent
open_cursors = 2000
pga_aggregate_target = 1699741696
Sun Apr 10 13:31:50 2011
USER: terminating instance due to error 445
Instance terminated by USER, pid = 2089
Content for the Trace file for one of the databases;
ORACLE_HOME = /u01/app/oracle/db
System name: HP-UX
Node name: db01
Release: B.11.31
Version: U
Machine: ia64
Instance name: cent
Redo thread mounted by this instance: 0 <none>
Oracle process number: 0
Unix process pid: 4667, image: oracle@db01
*** 2011-04-10 15:44:19.475
Dynamic strand is set to TRUE
Running with 2 shared and 242 private strand(s). Zero-copy redo is FALSE
*** 2011-04-10 15:50:40.280
Waited for process PMON to initialize for 60 seconds
*** 2011-04-10 15:50:40.280
Dumping diagnostic information for PMON:
OS pid = 4701
loadavg : 1.36 1.35 1.29
Swapinfo :
Avail = 15520.08Mb Used = 2490.52Mb
Swap free = 13029.57Mb Kernel rsvd = 1018.08Mb
Free Mem = 10161.05Mb
F S UID PID PPID C PRI NI ADDR SZ WCHAN S
TIME TTY TIME COMD
1401 S oracle 4701 1 0 154 20 e000000413447080 53772 e00000017cefa140 15
:49:39 ? 0:00 ora_pmon_cent
/bin/sh[2]: /opt/langtools/bin/gdb: not found.
*** 2011-04-10 15:50:40.372
*** 2011-04-10 15:50:50.470
Waited for process PMON to initialize for 111 seconds
*** 2011-04-10 15:51:31.180
Dumping diagnostic information for PMON:
OS pid = 4701
loadavg : 1.36 1.35 1.30
Swapinfo :
Avail = 15520.08Mb Used = 2490.52Mb
Swap free = 13029.57Mb Kernel rsvd = 1018.08Mb
Free Mem = 10160.98Mb
F S UID PID PPID C PRI NI ADDR SZ WCHAN S
TIME TTY TIME COMD
1401 S oracle 4701 1 0 154 20 e000000413447080 53772 e00000017cefa140 15
:49:39 ? 0:00 ora_pmon_cent
/bin/sh[2]: /opt/langtools/bin/gdb: not found.
*** 2011-04-10 15:51:31.250
Killing process (ospid 4701): (reason=x4 error=0)
... and the process is still alive after kill!
*** 2011-04-10 15:51:40.190
KSTDUMP: In-memory trace dump
TIME(usecs):SEQ# ORAPID SID EVENT OP DATA
========================================================================
kground PMON
2C463A42:000000F8 1999 0 10254 42 KSBS1P: timed out while waiting
2C463A48:000000F9 1999 0 10254 51 KSBS1P: out of loop: process did not sta
rt
KSTDUMP: End of in-memory trace dump
*** 2011-04-10 15:51:40.190
All the databases has similar trace files, I rebooted the server and still none of the database is starting.
The Server resources utilization seems to be fine;
oracle$ iostat 3 3
device bps sps msps
disk3 93 6.9 1.0
disk5 17 2.4 1.0
disk17 0 0.1 1.0
disk3 21 4.3 1.0
disk5 20 4.0 1.0
disk17 0 0.3 1.0
disk3 0 0.0 1.0
disk5 0 0.0 1.0
disk17 0 0.0 1.0
oracle$ swapinfo -tan
Kb Kb Kb PCT START/ Kb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 24576000 0 24576000 0% 0 - 1 /dev/vg00/lvol2
reserve - 918204 -918204
memory 15892564 2550292 13342272 16%
total 40468564 3468496 37000068 9% - 0 -
System: hrdbdr01 Sun Apr 10 16:25:28 2011
Load averages: 1.61, 1.60, 1.55
237 processes: 182 sleeping, 55 running
Cpu states:
CPU LOAD USER NICE SYS IDLE BLOCK SWAIT INTR SSYS
0 1.61 27.7% 0.0% 71.3% 1.0% 0.0% 0.0% 0.0% 0.0%
1 1.59 32.0% 0.0% 67.0% 1.0% 0.0% 0.0% 0.0% 0.0%
2 1.61 13.0% 14.0% 71.0% 2.0% 0.0% 0.0% 0.0% 0.0%
3 1.60 10.0% 20.0% 69.0% 1.0% 0.0% 0.0% 0.0% 0.0%
--- ---- ----- ----- ----- ----- ----- ----- ----- -----
avg 1.61 20.8% 8.9% 69.3% 1.0% 0.0% 0.0% 0.0% 0.0%
System Page Size: 4Kbytes
Memory: 974032K (512772K) real, 1317016K (704304K) virtual, 12491712K free Page
# 1/24
CPU TTY PID USERNAME PRI NI SIZE RES STATE TIME %WCPU %CPU COMMAND
3 ? 4249 root 255 39 20140K 1612K run 42:41 49.96 49.88 stm
2 ? 3360 root 255 39 20140K 1612K run 55:11 49.65 49.56 stm
0 ? 3040 root 240 20 20140K 1612K run 72:27 49.08 48.99 stm
1 ? 2419 root 241 20 20140K 1616K run 77:20 48.81 48.73 stm
1 ? 4850 root 168 20 20140K 1612K sleep 4:15 23.99 23.95 stm
2 ? 4583 root 168 20 20140K 1612K sleep 12:21 22.93 22.89 stm