Reg LGWR got killed by OS utility
Hi,
Last saturday i faced a strange issue in my database.
My environment:
Oracle EBS.
Two node installation.
Forms and Web in One server and
Concurrent and Database in Another.
We have sheduled backup on every saturday.
Duing last saturday,after backup was completed,We started the instance it was normal,
We also checked the working on instance and concurrent managers by running reports.
it was around 9 pm.
Then sysadmin guy has started restoration from san storage using tivoli client to one mount point in
same database server.
It was around 10.24 pm LGWR process was killed automatically by OOM(out of memory killer) in Linux.
I have copied the alert log entries during that time.
_____________________________________________________________________________________________
DETAILS FROM bdump from Oracle Database
Sat Jun 5 20:58:20 2010
Starting ORACLE instance (normal)
PMON started with pid=2, OS id=13375
PSP0 started with pid=3, OS id=13377
MMAN started with pid=4, OS id=13379
DBW0 started with pid=5, OS id=13381
DBW1 started with pid=6, OS id=13383
LGWR started with pid=7, OS id=13385
CKPT started with pid=8, OS id=13387
SMON started with pid=9, OS id=13389
RECO started with pid=10, OS id=13391
CJQ0 started with pid=11, OS id=13393
MMON started with pid=12, OS id=13395
MMNL started with pid=13, OS id=13397
Sat Jun 5 22:24:42 2010
Errors in file /oracle/PROD/app/proddb/10.2.0/admin/PROD_dbserv1/bdump/prod_pmon_13375.trc:
ORA-00470: LGWR process terminated with error
Sat Jun 5 22:24:45 2010
Errors in file /oracle/PROD/app/proddb/10.2.0/admin/PROD_dbserv1/bdump/prod_ckpt_13387.trc:
ORA-00470: LGWR process terminated with error
Sat Jun 5 22:24:48 2010
PMON: terminating instance due to error 470
Sat Jun 5 22:24:48 2010
Errors in file /oracle/PROD/app/proddb/10.2.0/admin/PROD_dbserv1/bdump/prod_dbw0_13381.trc:
ORA-00470: LGWR process terminated with error
Sat Jun 5 22:24:48 2010
Errors in file /oracle/PROD/app/proddb/10.2.0/admin/PROD_dbserv1/bdump/prod_psp0_13377.trc:
ORA-00470: LGWR process terminated with error
Sat Jun 5 22:24:48 2010
Errors in file /oracle/PROD/app/proddb/10.2.0/admin/PROD_dbserv1/bdump/prod_mman_13379.trc:
ORA-00470: LGWR process terminated with error
Instance terminated by PMON, pid = 13375
________________________________________________________________________
Below are trace file contents.
prod_pmon_13375.trc
*** 2010-06-05 22:24:42.417
*** SERVICE NAME:(SYS$BACKGROUND) 2010-06-05 22:24:38.983
*** SESSION ID:(1000.1) 2010-06-05 22:24:38.983
Background process LGWR found dead
Oracle pid = 7
OS pid (from detached process) = 13385
OS pid (from process state) = 13385
dtp = 0x2000bbf0, proc = 0x52a15e84
error 470 detected in background process
ORA-00470: LGWR process terminated with error
==========
prod_ckpt_13387.trc
*** 2010-06-05 21:34:57.543
*** SERVICE NAME:(SYS$BACKGROUND) 2010-06-05 21:34:56.241
*** SESSION ID:(994.1) 2010-06-05 21:34:56.241
*** 2010-06-05 21:34:56.241 62692 kcrr.c
ARCH: Detected ARCH process failure
*** 2010-06-05 21:40:32.152
*** 2010-06-05 21:40:32.152 62692 kcrr.c
ARCH: Detected ARCH process failure
*** 2010-06-05 21:45:04.723
*** 2010-06-05 21:45:04.723 62692 kcrr.c
ARCH: Detected ARCH process failure
*** 2010-06-05 22:24:45.661
error 470 detected in background process
ORA-00470: LGWR process terminated with error
==================
prod_dbw0_13381.trc
*** 2010-06-05 22:24:48.507
*** SERVICE NAME:(SYS$BACKGROUND) 2010-06-05 22:24:45.800
*** SESSION ID:(997.1) 2010-06-05 22:24:45.800
error 470 detected in background process
ORA-00470: LGWR process terminated with error
===================
prod_psp0_13377.trc
*** 2010-06-05 22:24:48.509
** SERVICE NAME:(SYS$BACKGROUND) 2010-06-05 22:24:46.405
*** SESSION ID:(999.1) 2010-06-05 22:24:46.405
error 470 detected in background process
ORA-00470: LGWR process terminated with error
===================
prod_mman_13379.trc
*** 2010-06-05 22:24:48.510
*** SERVICE NAME:(SYS$BACKGROUND) 2010-06-05 22:24:47.705
*** SESSION ID:(998.1) 2010-06-05 22:24:47.705
error 470 detected in background process
ORA-00470: LGWR process terminated with error
_____________________________________________________________________
DETAILS FROM /var/log/messages from OS
Jun 5 22:39:24 dbserv2 kernel: Out of Memory: Killed process 27262 (oracle).
Jun 5 22:35:27 dbserv2 kernel: Out of Memory: Killed process 21588 (oracle).
Jun 5 22:30:17 dbserv2 kernel: Out of Memory: Killed process 20745 (oracle).
Jun 5 22:29:59 dbserv2 kernel: Out of Memory: Killed process 20101 (oracle).
Jun 5 22:29:41 dbserv2 kernel: Out of Memory: Killed process 19896 (oracle).
Jun 5 22:29:25 dbserv2 kernel: Out of Memory: Killed process 16243 (oracle).
Jun 5 22:25:23 dbserv2 kernel: Out of Memory: Killed process 13670 (oracle).
Jun 5 22:24:47 dbserv2 kernel: Out of Memory: Killed process 13035 (oracle).
Jun 5 22:24:39 dbserv2 kernel: Out of Memory: Killed process 13385 (oracle). LGWR started with pid=7, OS id=13385
Jun 5 22:24:30 dbserv2 kernel: Out of Memory: Killed process 2436 (oracle).
Jun 5 22:24:30 dbserv2 kernel: Out of Memory: Killed process 2192 (oracle).
Jun 5 22:24:30 dbserv2 kernel: Out of Memory: Killed process 12585 (oracle).
Jun 5 22:23:58 dbserv2 kernel: Out of Memory: Killed process 6985 (oracle).
Jun 5 21:30:47 dbserv2 kernel: Out of Memory: Killed process 20644 (oracle).
If any of you have faced the same issue kindly help me.
Regards
Krishna