Hi ,
My database in oracle 11G and is running on SPARC-64 bit.
server vendors did some activity on the server , due to which today when i came 3 of my DB were down.
I checked the alert log and the trace file and that was what i found :
Errors in file /apps/oracle/admin/XXXX/diag/rdbms/XXXX/XXXX/trace/XXXX_pmon_20404.trc:
ORA-00490: PSP process terminated with error
PMON (ospid: 20404): terminating the instance due to error 490
Instance terminated by PMON, pid = 20404
Mon Apr 11 00:31:59 2011
and trace file :
Trace file /apps/oracle/admin/XXXX/diag/rdbms/XXXX/XXXX/trace/XXXX_pmon_20404.trc
Oracle Database 11g Enterprise Edition Release 11.1.0.7.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options
ORACLE_HOME = /apps/oracle/product/11.1.0.7.2010Q3
System name: SunOS
Node name: usa0300ux830
Release: 5.9
Version: Generic_122300-54
Machine: sun4u
Instance name: CPRIV
Redo thread mounted by this instance: 1
Oracle process number: 2
Unix process pid: 20404, image: oracle@usa0300ux830 (PMON)
*** 2011-04-10 11:35:29.674
*** SESSION ID:(38.1) 2011-04-10 11:35:29.844
*** CLIENT ID:() 2011-04-10 11:35:29.844
*** SERVICE NAME:(SYS$BACKGROUND) 2011-04-10 11:35:29.844
*** MODULE NAME:() 2011-04-10 11:35:29.844
*** ACTION NAME:() 2011-04-10 11:35:29.844
rdel: Failed to kill slaves
rdel: Failed to kill slaves
rdel: Failed to kill slaves
rdel: Failed to kill slaves
rdel: Failed to kill slaves
rdel: Failed to kill slaves
rdel: Failed to kill slaves
rdel: Failed to kill slaves
*** 2011-04-10 12:06:21.536
rdel: Failed to kill slaves
rdel: Failed to kill slaves
rdel: Failed to kill slaves
rdel: Failed to kill slaves
rdel: Failed to kill slaves
rdel: Failed to kill slaves
rdel: Failed to kill slaves
*** 2011-04-10 12:08:34.840
Waited for process SMCO to initialize for 60 seconds
*** 2011-04-10 12:08:34.875
Process diagnostic dump for SMCO, OS id=9316
-------------------------------------------------------------------------------
*** 2011-04-10 12:09:15.892
loadavg : 18.41 18.10 17.77
swap info: free_mem = 1030.56M rsv = 107162.29M
alloc = 99854.79M avail = 48202.88 swap_free = 55510.38M
skgpgcmdout: read() for cmd /bin/ps -elf | /bin/egrep 'PID | 9316' | /bin/grep -v grep timed out after 15.000 seconds
Skipping stack dump because max dump time exceeded.
-------------------------------------------------------------------------------
*** 2011-04-10 12:09:15.998
Process diagnostic dump actual duration=41.045000 sec
(max dump time=30.000000 sec)
*** 2011-04-10 12:09:15.998
*** 2011-04-10 12:09:25.412
Background process PSP0 found dead
Oracle pid = 14
OS pid (from detached process) = 20415
OS pid (from process state) = 20415
dtp = 38000f890, proc = 463004058
Dump of memory from 0x000000038000F890 to 0x000000038000F8E8
*** 2011-04-10 12:09:25.430
38000F890 00000009 00000000 00000004 63004058 [............c.@X]
38000F8A0 00000000 00000000 00000000 50535030 [............PSP0]
38000F8B0 00020000 00000000 00004FBF 00000000 [..........O.....]
38000F8C0 00000000 4DA06AE3 00000001 00000000 [....M.j.........]
38000F8D0 00000000 00000000 00000001 003199F9 [.............1..]
38000F8E0 00040081 00000000 [........]
Dump of memory from 0x0000000463004058 to 0x0000000463005050
463004050 01030000 00000000 [........]
463004060 02030000 00000000 00000000 00000000 [................]
463004070 00000004 63004058 00000004 63004078 [....c.@X....c.@x]
463004080 00000004 63004078 00000000 00000000 [....c.@x........]
463004090 00000000 00000000 00000000 00000000 [................]
I checked on metalink also but did not find much info.
What i understand from this is that space was not allocated due to which pmon crashed.
Can someone throw more light on this ?
Currently , i gave the startup command and its been a long time nothing is happening. should i abort the startup ?
regards
Kk