Coredumps and instance terminated?
Alan3Dec 21 2012 — edited Jan 28 2013Database is 11.2.0.1 64bit running on physical Oracle Linux R5U8.
This is a test database that has been running without issue for quite some time.
Recently moved to new hardware and has been running ok until this past weekend...
On Friday I enabled archivelog mode for running routine hot backups.
Early Monday morning we got the first coredump.
Another Monday afternoon, then again Tuesday & Wednesday early morning.
I was suspicious of the logs being too small (3 @ 50MB ea) so Wednesday I raised them to 6 @ 100MB each (well more than it should need...)
Seemed to work ok as we made it through Wednesday & Thursday without incident, but it crashed again Thursday evening... same messages each time.
Looking at the alert it appears to be tied to one of our test queues, but not sure if that is a culprit or just a result of the crash.
I have no other alerts or error messages - this is it, so I'm at a bit of a loss where this is coming from.
Haven't been able to find much anything on the particular error...
Hoped someone here might have some idea...
Alert log excerpt follows....
----------------------------------------------
Archived Log entry 40117 added for thread 1 sequence 54083 ID 0x26a060e dest 1:
Thu Dec 20 17:11:20 2012
Exception [type: SIGSEGV, Address not mapped to object] [ADDR:0x9] [PC:0x4375C3D, kwqdlClnupCcoDl()+113] [flags: 0x0, count: 1]
Errors in file /home/oracle/diag/rdbms/devbpel/devbpel/trace/devbpel_ora_8278.trc (incident=135758):
ORA-07445: exception encountered: core dump [kwqdlClnupCcoDl()+113] [SIGSEGV] [ADDR:0x9] [PC:0x4375C3D] [Address not mapped to object] []
ORA-25228: timeout or end-of-fetch during message dequeue from HCBPELAQ.IP_IN_QUEUE
Incident details in: /home/oracle/diag/rdbms/devbpel/devbpel/incident/incdir_135758/devbpel_ora_8278_i135758.trc
Thu Dec 20 17:11:25 2012
Trace dumping is performing id=[cdmp_20121220171125]
Thu Dec 20 17:11:26 2012
Sweep [inc][135758]: completed
Sweep [inc2][135758]: completed
Thu Dec 20 17:12:24 2012
Exception [type: SIGSEGV, Address not mapped to object] [ADDR:0x9] [PC:0x4375C3D, kwqdlClnupCcoDl()+113] [flags: 0x0, count: 1]
Errors in file /home/oracle/diag/rdbms/devbpel/devbpel/trace/devbpel_pmon_7876.trc (incident=135350):
ORA-07445: exception encountered: core dump [kwqdlClnupCcoDl()+113] [SIGSEGV] [ADDR:0x9] [PC:0x4375C3D] [Address not mapped to object] []
Incident details in: /home/oracle/diag/rdbms/devbpel/devbpel/incident/incdir_135350/devbpel_pmon_7876_i135350.trc
Errors in file /home/oracle/diag/rdbms/devbpel/devbpel/trace/devbpel_pmon_7876.trc:
ORA-00602: internal programming exception
ORA-07445: exception encountered: core dump [kwqdlClnupCcoDl()+113] [SIGSEGV] [ADDR:0x9] [PC:0x4375C3D] [Address not mapped to object] []
PMON (ospid: 7876): terminating the instance due to error 602
Instance terminated by PMON, pid = 7876