Oracle Database Discussions

We are having a problem with out of memory errors showing up in our alert log. I have found messages about getting it during installs, shutting down the DB but nothing about it showing up in the alert log during run time. It created big problems last weekend because it caused the archive logs to have problems. Here is what I am seeing in the alert log. I could really use some help on this. I was thinking it was an application, but after what I have found on the web, I am not sure it is now.

Errors in file /db/oraprod01/app/oracle/admin/pcen/bdump/pcen_j003_7035.trc:
ORA-12012: error on auto execute of job 162389
ORA-04030: out of process memory when trying to allocate ORA-04030: out of process memory when trying to allocate 8224 bytes (sort subheap,sort key)
ORA-06512: at "LPM.LPM_DATAMAINT", line 3961
ORA-06512: at line 4
bytes (,)
Sun Sep 21 15:09:30 2008
Errors in file /db/oraprod01/app/oracle/admin/pcen/bdump/pcen_j003_7035.trc:
ORA-04030: out of process memory when trying to allocate ORA-04030: out of process memory when trying to allocate 520 bytes (Typecheck,frodef : qcpitnm)
ORA-04030: out of process memory when trying to allocate ORA-04030: out of process memory when trying to allocate 8224 bytes (sort subheap,sort key)
ORA-06512: at "LPM.LPM_DATAMAINT", line 3961
ORA-06512: at line 4
bytes (,)
bytes (,)
ORA-04030: out of process memory when trying to allocate ORA-04030: out of process memory when trying to allocate 8224 bytes (sort subheap,sort key)
ORA-06512: at "LPM.LPM_DATAMAINT", line 3961
ORA-06512: at line 4
bytes (,)
Sun Sep 21 15:09:57 2008
Errors in file /db/oraprod01/app/oracle/admin/pcen/bdump/pcen_j002_10439.trc:
ORA-27102: out of memory
HPUX-ia64 Error: 12: Not enough space
Additional information: 108
Additional information: 655360
Additional information: 2506208
Sun Sep 21 15:10:22 2008
Errors in file /db/oraprod01/app/oracle/admin/pcen/bdump/pcen_j002_10439.trc:
ORA-04030: out of process memory when trying to allocate 753136 bytes (pga heap,kco buffer)
ORA-27102: out of memory
HPUX-ia64 Error: 12: Not enough space
Additional information: 108
Additional information: 655360
Additional information: 2506208
Sun Sep 21 15:10:22 2008
Errors in file /db/oraprod01/app/oracle/admin/pcen/bdump/pcen_j002_10439.trc:
ORA-00081: address range [0x60000000000A8210, 0x60000000000A8214) is not readable
ORA-27102: out of memory
HPUX-ia64 Error: 12: Not enough space
Additional information: 108
Additional information: 655360
Additional information: 2506208
Sun Sep 21 15:10:30 2008
Process startup failed, error stack:
Sun Sep 21 15:10:30 2008
Errors in file /db/oraprod01/app/oracle/admin/pcen/bdump/pcen_psp0_21007.trc:
ORA-27300: OS system dependent operation:fork failed with status: 11
ORA-27301: OS failure message: Resource temporarily unavailable
ORA-27302: failure occurred at: skgpspawn5
Sun Sep 21 15:10:30 2008
Errors in file /db/oraprod01/app/oracle/admin/pcen/bdump/pcen_j003_7035.trc:
ORA-12012: error on auto execute of job 162389
ORA-04030: out of process memory when trying to allocate ORA-04030: out of process memory when trying to allocate 8224 bytes (sort subheap,sort key)
ORA-06512: at "LPM.LPM_DATAMAINT", line 3961
ORA-06512: at line 4
bytes (,)
Sun Sep 21 15:10:30 2008
Errors in file /db/oraprod01/app/oracle/admin/pcen/bdump/pcen_j003_7035.trc:
ORA-04030: out of process memory when trying to allocate ORA-04030: out of process memory when trying to allocate 520 bytes (Typecheck,frodef : qcpitnm)
ORA-04030: out of process memory when trying to allocate ORA-04030: out of process memory when trying to allocate 8224 bytes (sort subheap,sort key)
ORA-06512: at "LPM.LPM_DATAMAINT", line 3961
ORA-06512: at line 4
bytes (,)
bytes (,)
ORA-04030: out of process memory when trying to allocate ORA-04030: out of process memory when trying to allocate 8224 bytes (sort subheap,sort key)
ORA-06512: at "LPM.LPM_DATAMAINT", line 3961
ORA-06512: at line 4
bytes (,)
Sun Sep 21 15:10:30 2008
Process J002 died, see its trace file
Sun Sep 21 15:10:30 2008
kkjcre1p: unable to spawn jobq slave process
Sun Sep 21 15:10:30 2008
Errors in file /db/oraprod01/app/oracle/admin/pcen/bdump/pcen_cjq0_21084.trc:

Sun Sep 21 15:11:05 2008
Errors in file /db/oraprod01/app/oracle/admin/pcen/udump/pcen_ora_4575.trc:
ORA-27102: out of memory
HPUX-ia64 Error: 12: Not enough space
Additional information: 108
Additional information: 655360
Additional information: -444973056
Sun Sep 21 15:11:08 2008
Errors in file /db/oraprod01/app/oracle/admin/pcen/udump/pcen_ora_4575.trc:
ORA-07445: exception encountered: core dump [kghalf()+624] [SIGSEGV] [Address not mapped to object] [0xFFFFFFFFFFFFFFF0] [] []
ORA-27102: out of memory
HPUX-ia64 Error: 12: Not enough space
Additional information: 108
Additional information: 655360
Additional information: -444973056
Sun Sep 21 15:11:17 2008
Errors in file /db/oraprod01/app/oracle/admin/pcen/udump/pcen_ora_7438.trc:
ORA-27102: out of memory
HPUX-ia64 Error: 12: Not enough space
Additional information: 108
Additional information: 262144
Additional information: -444973056
Sun Sep 21 15:11:26 2008
Process startup failed, error stack:
Sun Sep 21 15:11:26 2008
Errors in file /db/oraprod01/app/oracle/admin/pcen/bdump/pcen_psp0_21007.trc:
ORA-27300: OS system dependent operation:fork failed with status: 12
ORA-27301: OS failure message: Not enough space
ORA-27302: failure occurred at: skgpspawn3
Sun Sep 21 15:11:27 2008
Process J002 died, see its trace file
Sun Sep 21 15:11:27 2008
kkjcre1p: unable to spawn jobq slave process
Sun Sep 21 15:11:27 2008
Errors in file /db/oraprod01/app/oracle/admin/pcen/bdump/pcen_cjq0_21084.trc:

Sun Sep 21 15:11:27 2008
Process startup failed, error stack:
Sun Sep 21 15:11:27 2008
Errors in file /db/oraprod01/app/oracle/admin/pcen/bdump/pcen_psp0_21007.trc:
ORA-27300: OS system dependent operation:fork failed with status: 12
ORA-27301: OS failure message: Not enough space
ORA-27302: failure occurred at: skgpspawn3
Sun Sep 21 15:11:28 2008
Process J002 died, see its trace file
Sun Sep 21 15:11:28 2008
kkjcre1p: unable to spawn jobq slave process
Sun Sep 21 15:11:28 2008
Errors in file /db/oraprod01/app/oracle/admin/pcen/bdump/pcen_cjq0_21084.trc:

Sun Sep 21 15:11:32 2008
Process startup failed, error stack:
Sun Sep 21 15:11:32 2008
Errors in file /db/oraprod01/app/oracle/admin/pcen/bdump/pcen_psp0_21007.trc:
ORA-27300: OS system dependent operation:fork failed with status: 12
ORA-27301: OS failure message: Not enough space
ORA-27302: failure occurred at: skgpspawn3
Sun Sep 21 15:11:32 2008
Process J002 died, see its trace file
Sun Sep 21 15:11:32 2008
kkjcre1p: unable to spawn jobq slave process
Sun Sep 21 15:11:32 2008
Errors in file /db/oraprod01/app/oracle/admin/pcen/bdump/pcen_cjq0_21084.trc:

Sun Sep 21 15:11:32 2008
Process startup failed, error stack:
Sun Sep 21 15:11:32 2008
Errors in file /db/oraprod01/app/oracle/admin/pcen/bdump/pcen_psp0_21007.trc:
ORA-27300: OS system dependent operation:fork failed with status: 12
ORA-27301: OS failure message: Not enough space
ORA-27302: failure occurred at: skgpspawn3
Sun Sep 21 15:11:32 2008
Errors in file /db/oraprod01/app/oracle/admin/pcen/bdump/pcen_j003_7035.trc:
ORA-12012: error on auto execute of job 162389
ORA-04030: out of process memory when trying to allocate ORA-04030: out of process memory when trying to allocate 123416 bytes (QERGH hash-agg,kllcqas:kllsltba)
ORA-06512: at "LPM.LPM_DATAMAINT", line 3961
ORA-06512: at line 4
bytes (,)
Sun Sep 21 15:11:32 2008
Errors in file /db/oraprod01/app/oracle/admin/pcen/bdump/pcen_j003_7035.trc:
ORA-04030: out of process memory when trying to allocate ORA-04030: out of process memory when trying to allocate 520 bytes (Typecheck,frodef : qcpitnm)
ORA-04030: out of process memory when trying to allocate ORA-04030: out of process memory when trying to allocate 123416 bytes (QERGH hash-agg,kllcqas:kllsltba)
ORA-06512: at "LPM.LPM_DATAMAINT", line 3961
ORA-06512: at line 4
bytes (,)
bytes (,)
ORA-04030: out of process memory when trying to allocate ORA-04030: out of process memory when trying to allocate 123416 bytes (QERGH hash-agg,kllcqas:kllsltba)
ORA-06512: at "LPM.LPM_DATAMAINT", line 3961
ORA-06512: at line 4
bytes (,)
Sun Sep 21 15:11:32 2008
Process J002 died, see its trace file
Sun Sep 21 15:11:32 2008
kkjcre1p: unable to spawn jobq slave process
Sun Sep 21 15:11:32 2008
Errors in file /db/oraprod01/app/oracle/admin/pcen/bdump/pcen_cjq0_21084.trc:

Sun Sep 21 15:11:32 2008
Process startup failed, error stack:
Sun Sep 21 15:11:32 2008
Errors in file /db/oraprod01/app/oracle/admin/pcen/bdump/pcen_psp0_21007.trc:
ORA-27300: OS system dependent operation:fork failed with status: 12
ORA-27301: OS failure message: Not enough space
ORA-27302: failure occurred at: skgpspawn3
Sun Sep 21 15:11:33 2008
Process J002 died, see its trace file
Sun Sep 21 15:11:33 2008
kkjcre1p: unable to spawn jobq slave process
Sun Sep 21 15:11:33 2008
Errors in file /db/oraprod01/app/oracle/admin/pcen/bdump/pcen_cjq0_21084.trc:

Sun Sep 21 15:11:42 2008
Errors in file /db/oraprod01/app/oracle/admin/pcen/udump/pcen_ora_4575.trc:
ORA-04030: out of process memory when trying to allocate 753136 bytes (pga heap,kco buffer)
ORA-07445: exception encountered: core dump [kghalf()+624] [SIGSEGV] [Address not mapped to object] [0xFFFFFFFFFFFFFFF0] [] []
ORA-27102: out of memory
HPUX-ia64 Error: 12: Not enough space
Additional information: 108
Additional information: 655360
Additional information: -444973056
Sun Sep 21 15:11:42 2008
Errors in file /db/oraprod01/app/oracle/admin/pcen/udump/pcen_ora_4575.trc:
ORA-00081: address range [0x60000000000A81E0, 0x60000000000A81E4) is not readable
ORA-07445: exception encountered: core dump [kghalf()+624] [SIGSEGV] [Address not mapped to object] [0xFFFFFFFFFFFFFFF0] [] []
ORA-27102: out of memory
HPUX-ia64 Error: 12: Not enough space
Additional information: 108
Additional information: 655360
Additional information: -444973056
Sun Sep 21 15:11:43 2008
Thread 1 advanced to log sequence 34405
Current log# 3 seq# 34405 mem# 0: +DATA/pcen/onlinelog/group_3.264.636531609
Current log# 3 seq# 34405 mem# 1: +FLASH/pcen/onlinelog/group_3.256.636531625
Sun Sep 21 15:11:43 2008
ERROR: Failed to connect with connect string:(DESCRIPTION=(ADDRESS=(PROTOCOL=beq)(PROGRAM=/db/oraprod01/app/oracle/product/10.2.0/db_1/bin/oracle)(ARGV0=oracle+ASM)(ENVS='ORACLE_HOME=/db/oraprod01/app/oracle/product/10.2.0/db_1,ORACLE_SID=+ASM')(ARGS='(DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))'))(enable=setuser))
Sun Sep 21 15:11:43 2008
Errors in file /db/oraprod01/app/oracle/admin/pcen/bdump/pcen_arc1_25220.trc:
ORA-19504: failed to create file "+FLASH/pcen/1_34404_592672654.dbf"
ORA-17502: ksfdcre:4 Failed to create file +FLASH/pcen/1_34404_592672654.dbf
ORA-15055: unable to connect to ASM instance
ORA-12549: TNS:operating system resource quota exceeded
ARC1: Error 19504 Creating archive log file to '+FLASH/pcen/1_34404_592672654.dbf'
ARC1: Failed to archive thread 1 sequence 34404 (19504)
ARCH: Archival stopped, error occurred. Will continue retrying
Sun Sep 21 15:11:43 2008
ORACLE Instance pcen - Archival Error
Sun Sep 21 15:11:43 2008
ORA-16038: log 2 sequence# 34404 cannot be archived
ORA-19504: failed to create file ""
ORA-00312: online log 2 thread 1: '+DATA/pcen/onlinelog/group_2.263.636531579'
ORA-00312: online log 2 thread 1: '+FLASH/pcen/onlinelog/group_2.257.636531595'
Sun Sep 21 15:11:43 2008
Errors in file /db/oraprod01/app/oracle/admin/pcen/bdump/pcen_arc1_25220.trc:
ORA-16038: log 2 sequence# 34404 cannot be archived
ORA-19504: failed to create file ""
ORA-00312: online log 2 thread 1: '+DATA/pcen/onlinelog/group_2.263.636531579'
ORA-00312: online log 2 thread 1: '+FLASH/pcen/onlinelog/group_2.257.636531595'
Sun Sep 21 15:11:44 2008
LNS: Standby redo logfile selected for thread 1 sequence 34405 for destination LOG_ARCHIVE_DEST_2
Sun Sep 21 15:11:51 2008
Errors in file /db/oraprod01/app/oracle/admin/pcen/udump/pcen_ora_3798.trc:
ORA-27102: out of memory
HPUX-ia64 Error: 12: Not enough space
Additional information: 108
Additional information: 131072
Sun Sep 21 15:11:51 2008
Errors in file /db/oraprod01/app/oracle/admin/pcen/udump/pcen_ora_3798.trc:
ORA-07445: exception encountered: core dump [kghalf()+624] [SIGSEGV] [Address not mapped to object] [0xFFFFFFFFFFFFFFF0] [] []
ORA-27102: out of memory
HPUX-ia64 Error: 12: Not enough space
Additional information: 108
Additional information: 131072
Sun Sep 21 15:12:00 2008
Errors in file /db/oraprod01/app/oracle/admin/pcen/bdump/pcen_j003_7035.trc:
ORA-27102: out of memory
HPUX-ia64 Error: 12: Not enough space
Additional information: 108
Additional information: 393216
Additional information: 2506208
Sun Sep 21 15:12:01 2008
Errors in file /db/oraprod01/app/oracle/admin/pcen/bdump/pcen_j001_14174.trc:
ORA-27102: out of memory
HPUX-ia64 Error: 12: Not enough space
Additional information: 108
Additional information: 131072
Additional information: 2506208
Sun Sep 21 15:12:02 2008
ARC0: Archiving not possible: No primary destinations
ARC0: Failed to archive thread 1 sequence 34404 (4)
ARCH: Archival stopped, error occurred. Will continue retrying
Sun Sep 21 15:12:02 2008
ORACLE Instance pcen - Archival Error
Sun Sep 21 15:12:02 2008
ORA-16014: log 2 sequence# 34404 not archived, no available destinations
ORA-00312: online log 2 thread 1: '+DATA/pcen/onlinelog/group_2.263.636531579'
ORA-00312: online log 2 thread 1: '+FLASH/pcen/onlinelog/group_2.257.636531595'
Sun Sep 21 15:12:02 2008
Errors in file /db/oraprod01/app/oracle/admin/pcen/bdump/pcen_arc0_25200.trc:
ORA-16014: log 2 sequence# 34404 not archived, no available destinations
ORA-00312: online log 2 thread 1: '+DATA/pcen/onlinelog/group_2.263.636531579'
ORA-00312: online log 2 thread 1: '+FLASH/pcen/onlinelog/group_2.257.636531595'
Sun Sep 21 15:12:07 2008
Errors in file /db/oraprod01/app/oracle/admin/pcen/udump/pcen_ora_6704.trc:
ORA-27102: out of memory
HPUX-ia64 Error: 12: Not enough space
Additional information: 108
Additional information: 524288
Additional information: -444973056
Sun Sep 21 15:12:09 2008
Errors in file /db/oraprod01/app/oracle/admin/pcen/udump/pcen_ora_6706.trc:
ORA-27102: out of memory
HPUX-ia64 Error: 12: Not enough space
Additional information: 108
Additional information: 131072
Additional information: -444973056
Sun Sep 21 15:12:11 2008
Errors in file /db/oraprod01/app/oracle/admin/pcen/bdump/pcen_j002_10066.trc:
ORA-12012: error on auto execute of job 157369
ORA-04030: out of process memory when trying to allocate bytes (,)
ORA-04030: out of process memory when trying to allocate ORA-04030: out of process memory when trying to allocate 589856 bytes (pga heap,drqlist_kfiopg)
bytes (,)
Sun Sep 21 15:12:24 2008
Errors in file /db/oraprod01/app/oracle/admin/pcen/udump/pcen_ora_3798.trc:
ORA-07445: exception encountered: core dump [kghalp()+496] [SIGSEGV] [Invalid permissions for mapped object] [0x00000003B] [] []
ORA-07445: exception encountered: core dump [kghalf()+624] [SIGSEGV] [Address not mapped to object] [0xFFFFFFFFFFFFFFF0] [] []
ORA-27102: out of memory
HPUX-ia64 Error: 12: Not enough space
Additional information: 108
Additional information: 131072
Sun Sep 21 15:12:26 2008
Errors in file /db/oraprod01/app/oracle/admin/pcen/bdump/pcen_j001_14174.trc:
ORA-00081: address range [0x60000000000A81E0, 0x60000000000A81E4) is not readable
ORA-27102: out of memory
HPUX-ia64 Error: 12: Not enough space
Additional information: 108
Additional information: 131072
Additional information: 2506208
Sun Sep 21 15:12:26 2008
Errors in file /db/oraprod01/app/oracle/admin/pcen/bdump/pcen_j003_7035.trc:
ORA-00081: address range [0x60000000000A81E0, 0x60000000000A81E4) is not readable
ORA-27102: out of memory
HPUX-ia64 Error: 12: Not enough space
Additional information: 108
Additional information: 393216
Additional information: 2506208
Sun Sep 21 15:12:30 2008
Process startup failed, error stack:
Sun Sep 21 15:12:30 2008
Errors in file /db/oraprod01/app/oracle/admin/pcen/bdump/pcen_psp0_21007.trc:
ORA-27300: OS system dependent operation:fork failed with status: 12
ORA-27301: OS failure message: Not enough space
ORA-27302: failure occurred at: skgpspawn3
Sun Sep 21 15:12:30 2008
Errors in file /db/oraprod01/app/oracle/admin/pcen/bdump/pcen_j002_10066.trc:
ORA-12012: error on auto execute of job 162389
ORA-04030: out of process memory when trying to allocate bytes (,)
ORA-04030: out of process memory when trying to allocate ORA-04030: out of process memory when trying to allocate 589856 bytes (pga heap,drqlist_kfiopg)
bytes (,)
Sun Sep 21 15:12:30 2008
Process J001 died, see its trace file
Sun Sep 21 15:12:30 2008
kkjcre1p: unable to spawn jobq slave process
Sun Sep 21 15:12:30 2008
Errors in file /db/oraprod01/app/oracle/admin/pcen/bdump/pcen_cjq0_21084.trc:

Sun Sep 21 15:12:30 2008
Errors in file /db/oraprod01/app/oracle/admin/pcen/bdump/pcen_j002_10066.trc:
ORA-12012: error on auto execute of job 140817
ORA-04030: out of process memory when trying to allocate bytes (,)
ORA-04030: out of process memory when trying to allocate ORA-04030: out of process memory when trying to allocate 589856 bytes (pga heap,drqlist_kfiopg)
bytes (,)
Sun Sep 21 15:13:03 2008
ARC1: Archiving not possible: No primary destinations
ARC1: Failed to archive thread 1 sequence 34404 (4)
Sun Sep 21 15:13:22 2008
Errors in file /db/oraprod01/app/oracle/admin/pcen/udump/pcen_ora_3798.trc:
ORA-04030: out of process memory when trying to allocate 753136 bytes (pga heap,kco buffer)
ORA-07445: exception encountered: core dump [kghalp()+496] [SIGSEGV] [Invalid permissions for mapped object] [0x00000003B] [] []
ORA-07445: exception encountered: core dump [kghalf()+624] [SIGSEGV] [Address not mapped to object] [0xFFFFFFFFFFFFFFF0] [] []
ORA-27102: out of memory
HPUX-ia64 Error: 12: Not enough space
Additional information: 108
Additional information: 131072
Sun Sep 21 15:14:03 2008
ARC0: Archiving not possible: No primary destinations
ARC0: Failed to archive thread 1 sequence 34404 (4)
Sun Sep 21 15:15:03 2008
ARC1: Archiving not possible: No primary destinations
ARC1: Failed to archive thread 1 sequence 34404 (4)
Sun Sep 21 15:15:52 2008
ARC1: Archiving not possible: No primary destinations
ARC1: Failed to archive thread 1 sequence 34404 (4)
Sun Sep 21 15:16:04 2008
ARC0: Archiving not possible: No primary destinations
ARC0: Failed to archive thread 1 sequence 34404 (4)
Sun Sep 21 15:17:47 2008
Archiver process freed from errors. No longer stopped
Sun Sep 21 15:24:01 2008
Errors in file /db/oraprod01/app/oracle/admin/pcen/udump/pcen_ora_12565.trc:
ORA-27102: out of memory
HPUX-ia64 Error: 12: Not enough space
Additional information: 108
Additional information: 196608
Additional information: -444973056
Sun Sep 21 15:24:08 2008
Errors in file /db/oraprod01/app/oracle/admin/pcen/udump/pcen_ora_11334.trc:
ORA-27102: out of memory
HPUX-ia64 Error: 12: Not enough space
Additional information: 108
Additional information: 589824
Additional information: -444973056
Sun Sep 21 15:24:53 2008
Errors in file /db/oraprod01/app/oracle/admin/pcen/udump/pcen_ora_11926.trc:
ORA-27102: out of memory
HPUX-ia64 Error: 12: Not enough space
Additional information: 108
Additional information: 786432
Sun Sep 21 15:31:33 2008
Process startup failed, error stack:
Sun Sep 21 15:31:33 2008
Errors in file /db/oraprod01/app/oracle/admin/pcen/bdump/pcen_psp0_21007.trc:
ORA-27300: OS system dependent operation:fork failed with status: 11
ORA-27301: OS failure message: Resource temporarily unavailable
ORA-27302: failure occurred at: skgpspawn5
Sun Sep 21 15:31:34 2008
Process J003 died, see its trace file
Sun Sep 21 15:31:34 2008
kkjcre1p: unable to spawn jobq slave process
Sun Sep 21 15:31:34 2008
Errors in file /db/oraprod01/app/oracle/admin/pcen/bdump/pcen_cjq0_21084.trc:

Sun Sep 21 15:31:39 2008
Process startup failed, error stack:
Sun Sep 21 15:31:39 2008
Errors in file /db/oraprod01/app/oracle/admin/pcen/bdump/pcen_psp0_21007.trc:
ORA-27300: OS system dependent operation:fork failed with status: 11
ORA-27301: OS failure message: Resource temporarily unavailable
ORA-27302: failure occurred at: skgpspawn5
Sun Sep 21 15:31:40 2008
Process J003 died, see its trace file
Sun Sep 21 15:31:40 2008
kkjcre1p: unable to spawn jobq slave process
Sun Sep 21 15:31:40 2008
Errors in file /db/oraprod01/app/oracle/admin/pcen/bdump/pcen_cjq0_21084.trc:

Sun Sep 21 15:31:59 2008
Errors in file /db/oraprod01/app/oracle/admin/pcen/udump/pcen_ora_12495.trc:
ORA-27102: out of memory
HPUX-ia64 Error: 12: Not enough space
Additional information: 108
Additional information: 917504
Additional information: -444973056
Sun Sep 21 15:31:59 2008
Errors in file /db/oraprod01/app/oracle/admin/pcen/udump/pcen_ora_21389.trc:
ORA-27102: out of memory
HPUX-ia64 Error: 12: Not enough space
Additional information: 108
Additional information: 1441792
Sun Sep 21 15:32:00 2008
Errors in file /db/oraprod01/app/oracle/admin/pcen/bdump/pcen_j000_13595.trc:
ORA-27102: out of memory
HPUX-ia64 Error: 12: Not enough space
Additional information: 108
Additional information: 458752
Additional information: 2506208
Sun Sep 21 15:32:01 2008
Errors in file /db/oraprod01/app/oracle/admin/pcen/bdump/pcen_j002_17863.trc:
ORA-12012: error on auto execute of job 11
ORA-04030: out of process memory when trying to allocate 589856 bytes (pga heap,drqlist_kfiopg)
Sun Sep 21 15:32:01 2008
Errors in file /db/oraprod01/app/oracle/admin/pcen/bdump/pcen_j001_17610.trc:
ORA-27102: out of memory
HPUX-ia64 Error: 12: Not enough space
Additional information: 108
Additional information: 655360
Sun Sep 21 15:32:11 2008
Errors in file /db/oraprod01/app/oracle/admin/pcen/udump/pcen_ora_11248.trc:
ORA-27102: out of memory
HPUX-ia64 Error: 12: Not enough space
Additional information: 108
Additional information: 589824
Additional information: -444973056
Sun Sep 21 15:32:11 2008
Errors in file /db/oraprod01/app/oracle/admin/pcen/udump/pcen_ora_11248.trc:
ORA-07445: exception encountered: core dump [kghalf()+624] [SIGSEGV] [Address not mapped to object] [0xFFFFFFFFFFFFFFF0] [] []
ORA-27102: out of memory
HPUX-ia64 Error: 12: Not enough space
Additional information: 108
Additional information: 589824
Additional information: -444973056
Sun Sep 21 15:32:26 2008
Errors in file /db/oraprod01/app/oracle/admin/pcen/bdump/pcen_j000_13595.trc:
ORA-00081: address range [0x60000000000A81E0, 0x60000000000A81E4) is not readable
ORA-27102: out of memory
HPUX-ia64 Error: 12: Not enough space
Additional information: 108
Additional information: 458752
Additional information: 2506208
Sun Sep 21 15:32:27 2008
Errors in file /db/oraprod01/app/oracle/admin/pcen/bdump/pcen_j001_17610.trc:
ORA-00081: address range [0x60000000000A81E0, 0x60000000000A81E4) is not readable
ORA-27102: out of memory
HPUX-ia64 Error: 12: Not enough space
Additional information: 108
Additional information: 655360
Sun Sep 21 15:32:28 2008
Errors in file /db/oraprod01/app/oracle/admin/pcen/udump/pcen_ora_21389.trc:
ORA-07445: exception encountered: core dump [kghalp()+496] [SIGSEGV] [Invalid permissions for mapped object] [0x00000003B] [] []
ORA-27102: out of memory
HPUX-ia64 Error: 12: Not enough space
Additional information: 108
Additional information: 1441792
Sun Sep 21 15:32:45 2008
Errors in file /db/oraprod01/app/oracle/admin/pcen/udump/pcen_ora_11248.trc:
ORA-00081: address range [0x60000000000A81E0, 0x60000000000A81E4) is not readable
ORA-07445: exception encountered: core dump [kghalf()+624] [SIGSEGV] [Address not mapped to object] [0xFFFFFFFFFFFFFFF0] [] []
ORA-27102: out of memory
HPUX-ia64 Error: 12: Not enough space
Additional information: 108
Additional information: 589824
Additional information: -444973056
Sun Sep 21 15:33:03 2008
Errors in file /db/oraprod01/app/oracle/admin/pcen/udump/pcen_ora_17422.trc:
ORA-27102: out of memory
HPUX-ia64 Error: 12: Not enough space
Additional information: 108
Additional information: 262144
Additional information: 21647360
Sun Sep 21 15:33:04 2008
Errors in file /db/oraprod01/app/oracle/admin/pcen/bdump/pcen_j000_17989.trc:
ORA-12012: error on auto execute of job 166160
ORA-04030: out of process memory when trying to allocate ORA-04030: out of process memory when trying to allocate 123416 bytes (QERHJ hash-joi,kllcqas:kllsltba)
ORA-06512: at "TERMINAL.LPM_TCH_EXTRACT", line 1424
ORA-06512: at line 5
bytes (,)
Sun Sep 21 15:33:21 2008
Errors in file /db/oraprod01/app/oracle/admin/pcen/udump/pcen_ora_17928.trc:
ORA-27102: out of memory
HPUX-ia64 Error: 12: Not enough space
Additional information: 108
Additional information: 1179648
Additional information: -444973056
Sun Sep 21 15:33:21 2008
Errors in file /db/oraprod01/app/oracle/admin/pcen/udump/pcen_ora_17928.trc:
ORA-07445: exception encountered: core dump [kghalf()+624] [SIGSEGV] [Address not mapped to object] [0xFFFFFFFFFFFFFFF0] [] []
ORA-27102: out of memory
HPUX-ia64 Error: 12: Not enough space
Additional information: 108
Additional information: 1179648
Additional information: -444973056
Sun Sep 21 15:33:22 2008
Errors in file /db/oraprod01/app/oracle/admin/pcen/udump/pcen_ora_21389.trc:
ORA-00081: address range [0x60000000000A81E0, 0x60000000000A81E4) is not readable
ORA-07445: exception encountered: core dump [kghalp()+496] [SIGSEGV] [Invalid permissions for mapped object] [0x00000003B] [] []
ORA-27102: out of memory
HPUX-ia64 Error: 12: Not enough space
Additional information: 108
Additional information: 1441792
Sun Sep 21 15:33:51 2008
Errors in file /db/oraprod01/app/oracle/admin/pcen/udump/pcen_ora_17928.trc:
ORA-07445: exception encountered: core dump [kghalp()+496] [SIGSEGV] [Invalid permissions for mapped object] [0x00000003B] [] []
ORA-07445: exception encountered: core dump [kghalf()+624] [SIGSEGV] [Address not mapped to object] [0xFFFFFFFFFFFFFFF0] [] []
ORA-27102: out of memory
HPUX-ia64 Error: 12: Not enough space
Additional information: 108
Additional information: 1179648
Additional information: -444973056
Sun Sep 21 15:34:08 2008
Thread 1 advanced to log sequence 34406
Current log# 1 seq# 34406 mem# 0: +DATA/pcen/onlinelog/group_1.262.636531547
Current log# 1 seq# 34406 mem# 1: +FLASH/pcen/onlinelog/group_1.258.636531565
Sun Sep 21 15:34:08 2008
ERROR: Failed to connect with connect string:(DESCRIPTION=(ADDRESS=(PROTOCOL=beq)(PROGRAM=/db/oraprod01/app/oracle/product/10.2.0/db_1/bin/oracle)(ARGV0=oracle+ASM)(ENVS='ORACLE_HOME=/db/oraprod01/app/oracle/product/10.2.0/db_1,ORACLE_SID=+ASM')(ARGS='(DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))'))(enable=setuser))
Sun Sep 21 15:34:08 2008
Errors in file /db/oraprod01/app/oracle/admin/pcen/bdump/pcen_arc0_25200.trc:
ORA-19504: failed to create file "+FLASH/pcen/1_34405_592672654.dbf"
ORA-17502: ksfdcre:4 Failed to create file +FLASH/pcen/1_34405_592672654.dbf
ORA-15055: unable to connect to ASM instance
ORA-12549: TNS:operating system resource quota exceeded
ARC0: Error 19504 Creating archive log file to '+FLASH/pcen/1_34405_592672654.dbf'
ARC0: Failed to archive thread 1 sequence 34405 (19504)
ARCH: Archival stopped, error occurred. Will continue retrying
Sun Sep 21 15:34:08 2008
ORACLE Instance pcen - Archival Error
Sun Sep 21 15:34:08 2008
ORA-16038: log 3 sequence# 34405 cannot be archived
ORA-19504: failed to create file ""
ORA-00312: online log 3 thread 1: '+DATA/pcen/onlinelog/group_3.264.636531609'
ORA-00312: online log 3 thread 1: '+FLASH/pcen/onlinelog/group_3.256.636531625'
Sun Sep 21 15:34:08 2008
Errors in file /db/oraprod01/app/oracle/admin/pcen/bdump/pcen_arc0_25200.trc:
ORA-16038: log 3 sequence# 34405 cannot be archived
ORA-19504: failed to create file ""
ORA-00312: online log 3 thread 1: '+DATA/pcen/onlinelog/group_3.264.636531609'
ORA-00312: online log 3 thread 1: '+FLASH/pcen/onlinelog/group_3.256.636531625'
Sun Sep 21 15:34:09 2008
ARC1: Archiving not possible: No primary destinations
ARC1: Failed to archive thread 1 sequence 34405 (4)
ARCH: Archival stopped, error occurred. Will continue retrying
Sun Sep 21 15:34:09 2008
ORACLE Instance pcen - Archival Error
Sun Sep 21 15:34:09 2008
ORA-16014: log 3 sequence# 34405 not archived, no available destinations
ORA-00312: online log 3 thread 1: '+DATA/pcen/onlinelog/group_3.264.636531609'
ORA-00312: online log 3 thread 1: '+FLASH/pcen/onlinelog/group_3.256.636531625'
Sun Sep 21 15:34:09 2008
Errors in file /db/oraprod01/app/oracle/admin/pcen/bdump/pcen_arc1_25220.trc:
ORA-16014: log 3 sequence# 34405 not archived, no available destinations
ORA-00312: online log 3 thread 1: '+DATA/pcen/onlinelog/group_3.264.636531609'
ORA-00312: online log 3 thread 1: '+FLASH/pcen/onlinelog/group_3.256.636531625'
Sun Sep 21 15:34:10 2008
LNS: Standby redo logfile selected for thread 1 sequence 34406 for destination LOG_ARCHIVE_DEST_2
Sun Sep 21 15:35:03 2008
Errors in file /db/oraprod01/app/oracle/admin/pcen/udump/pcen_ora_17901.trc:
ORA-27102: out of memory
HPUX-ia64 Error: 12: Not enough space
Additional information: 108
Additional information: 327680
Additional information: -1346033664
Sun Sep 21 15:35:10 2008
ARC0: Archiving not possible: No primary destinations
ARC0: Failed to archive thread 1 sequence 34405 (4)
Sun Sep 21 15:35:55 2008
ARC1: Archiving not possible: No primary destinations
ARC1: Failed to archive thread 1 sequence 34405 (4)
Sun Sep 21 15:36:10 2008
ARC1: Archiving not possible: No primary destinations
ARC1: Failed to archive thread 1 sequence 34405 (4)
Sun Sep 21 15:36:41 2008
Errors in file /db/oraprod01/app/oracle/admin/pcen/udump/pcen_ora_12750.trc:
ORA-00600: internal error code, arguments: [12240], [], [], [], [], [], [], []
Sun Sep 21 15:37:02 2008
Errors in file /db/oraprod01/app/oracle/admin/pcen/udump/pcen_ora_12750.trc:
ORA-07445: exception encountered: core dump [kcsgrsn()+400] [SIGSEGV] [Invalid permissions for mapped object] [0x0000006EC] [] []
ORA-00600: internal error code, arguments: [12240], [], [], [], [], [], [], []
Sun Sep 21 15:37:10 2008
ARC0: Archiving not possible: No primary destinations
ARC0: Failed to archive thread 1 sequence 34405 (4)
Sun Sep 21 15:37:35 2008
Errors in file /db/oraprod01/app/oracle/admin/pcen/udump/pcen_ora_12750.trc:
ORA-07445: exception encountered: core dump [kcsgrsn()+400] [SIGSEGV] [Invalid permissions for mapped object] [0x0000006EC] [] []
ORA-07445: exception encountered: core dump [kcsgrsn()+400] [SIGSEGV] [Invalid permissions for mapped object] [0x0000006EC] [] []
ORA-00600: internal error code, arguments: [12240], [], [], [], [], [], [], []
Sun Sep 21 15:38:11 2008
ARC1: Archiving not possible: No primary destinations
ARC1: Failed to archive thread 1 sequence 34405 (4)