Hello all,
Today our DB crashed with the below error in alert log running on linux 4.6 db : 10.2.0.3.0 (32bit)
Completed checkpoint up to RBA [0x1730.2.10], SCN: 5979396038739
Wed May 28 08:56:31 2014
Errors in file /oracle/clndb/10.2.0/admin/PROD_damacdb/udump/TESTcln_ora_6562.trc:
ORA-07445: exception encountered: core dump [kokcup()+97] [SIGSEGV] [Address not mapped to object] [0xC000095D] [] []
Wed May 28 08:56:32 2014
Errors in file /oracle/clndb/10.2.0/admin/PROD_damacdb/udump/TESTcln_ora_6562.trc:
ORA-07445: exception encountered: core dump [kghssgmm()+83] [SIGFPE] [Integer divide by zero] [0xAD03465] [] []
ORA-07445: exception encountered: core dump [kokcup()+97] [SIGSEGV] [Address not mapped to object] [0xC000095D] [] []
Wed May 28 08:59:55 2014
Beginning log switch checkpoint up to RBA [0x1731.2.10], SCN: 5979396047597
Thread 1 advanced to log sequence 5937
Current log# 2 seq# 5937 mem# 0: /d02/clndata/log02a.dbf
Current log# 2 seq# 5937 mem# 1: /d02/clndata/log02b.dbf
Wed May 28 09:02:25 2014
Errors in file /oracle/clndb/10.2.0/admin/PROD_damacdb/udump/TESTcln_ora_2032.trc:
ORA-07445: exception encountered: core dump [kghupr_flg()+551] [SIGSEGV] [Address not mapped to object] [0x10] [] []
Wed May 28 09:02:49 2014
Errors in file /oracle/clndb/10.2.0/admin/PROD_damacdb/bdump/TESTcln_pmon_23066.trc:
ORA-07445: exception encountered: core dump [kghupr_flg()+551] [SIGSEGV] [Address not mapped to object] [0x10] [] []
Wed May 28 09:02:54 2014
MMAN: terminating instance due to error 472
Instance terminated by MMAN, pid = 23070
Wed May 28 09:08:01 2014
Starting ORACLE instance (normal)
Wed May 28 09:08:01 2014
System parameter db_block_buffers enabled without VLM on.
LICENSE_MAX_SESSION = 0
LICENSE_SESSIONS_WARNING = 0
Picked latch-free SCN scheme 2
Using LOG_ARCHIVE_DEST_1 parameter default value as /oracle/clndb/10.2.0/dbs/arch
Autotune of undo retention is turned on.
IMODE=BR
ILAT =220
LICENSE_MAX_USERS = 0
SYS auditing is enabled
ksdpec: called for event 13740 prior to event group initialization
Starting up ORACLE RDBMS Version: 10.2.0.3.0.
System parameters with non-default values:
_trace_files_public = FALSE
processes = 1000
sessions = 2000
timed_statistics = TRUE
resource_limit = TRUE
shared_pool_size = 838860800
streams_pool_size = 50331648
shared_pool_reserved_size= 83886080
nls_language = american
nls_territory = america
nls_sort = binary
nls_date_format = DD-MON-RR
nls_numeric_characters = .,
nls_comp = binary
nls_length_semantics = BYTE
resource_manager_plan = INTERNAL_PLAN
control_files = /d02/clndata/cntrl01.dbf, /d02/clndata/cntrl02.dbf, /d02/clndata/cntrl03.dbf
db_block_buffers = 150000
db_block_checksum = TRUE
db_block_size = 8192
compatible = 10.2.0
log_buffer = 15276032
log_checkpoint_interval = 100000
log_checkpoint_timeout = 1200
db_files = 550
db_file_multiblock_read_count= 8
log_checkpoints_to_alert = TRUE
dml_locks = 10000
undo_management = AUTO
undo_tablespace = APPS_UNDOTS1
undo_retention = 7200
db_block_checking = FALSE
O7_DICTIONARY_ACCESSIBILITY= FALSE
remote_login_passwordfile= NONE
audit_sys_operations = TRUE
session_cached_cursors = 1200
utl_file_dir = /usr/tmp, /oracle/clndb/10.2.0/appsutil/outbound/PROD_damacdb
plsql_code_type = INTERPRETED
plsql_optimize_level = 2
job_queue_processes = 2
_system_trig_enabled = TRUE
cursor_sharing = EXACT
parallel_min_servers = 0
parallel_max_servers = 8
background_dump_dest = /oracle/clndb/10.2.0/admin/PROD_damacdb/bdump
user_dump_dest = /oracle/clndb/10.2.0/admin/PROD_damacdb/udump
max_dump_file_size = 20480
core_dump_dest = /oracle/clndb/10.2.0/admin/PROD_damacdb/cdump
session_max_open_files = 20
db_name = TESTCLN
open_cursors = 1500
os_authent_prefix =
_sort_elimination_cost_ratio= 5
sql92_security = TRUE
_b_tree_bitmap_plans = FALSE
_fast_full_scan_enabled = FALSE
_sqlexec_progression_cost= 2147483647
_like_with_bind_as_equality= TRUE
pga_aggregate_target = 4194304000
workarea_size_policy = AUTO
aq_tm_processes = 1
PMON started with pid=2, OS id=4457
PSP0 started with pid=3, OS id=4459
MMAN started with pid=4, OS id=4461
DBW0 started with pid=5, OS id=4464
LGWR started with pid=6, OS id=4466
CKPT started with pid=7, OS id=4468
SMON started with pid=8, OS id=4470
RECO started with pid=9, OS id=4472
CJQ0 started with pid=10, OS id=4474
MMON started with pid=11, OS id=4476
MMNL started with pid=12, OS id=4478
Wed May 28 09:08:03 2014
ALTER DATABASE MOUNT
Wed May 28 09:08:07 2014
Setting recovery target incarnation to 2
Wed May 28 09:08:07 2014
Successful mount of redo thread 1, with mount id 3266286451
Wed May 28 09:08:07 2014
Database mounted in Exclusive Mode
Completed: ALTER DATABASE MOUNT
Wed May 28 09:08:07 2014
ALTER DATABASE OPEN
Wed May 28 09:08:08 2014
Beginning crash recovery of 1 threads
parallel recovery started with 3 processes
Wed May 28 09:08:09 2014
Started redo scan
Wed May 28 09:08:09 2014
Completed redo scan
5831 redo blocks read, 402 data blocks need recovery
Wed May 28 09:08:09 2014
Started redo application at
Thread 1: logseq 5936, block 8590
Wed May 28 09:08:09 2014
Recovery of Online Redo Log: Thread 1 Group 1 Seq 5936 Reading mem 0
Mem# 0: /d02/clndata/log01a.dbf
Mem# 1: /d02/clndata/log01b.dbf
Wed May 28 09:08:10 2014
Recovery of Online Redo Log: Thread 1 Group 2 Seq 5937 Reading mem 0
Mem# 0: /d02/clndata/log02a.dbf
Mem# 1: /d02/clndata/log02b.dbf
Wed May 28 09:08:10 2014
Completed redo application
Wed May 28 09:08:10 2014
Completed crash recovery at
Thread 1: logseq 5937, block 4026, scn 5979396068960
402 data blocks read, 387 data blocks written, 5831 redo blocks read
Wed May 28 09:08:11 2014
Thread 1 advanced to log sequence 5938
Thread 1 opened at log sequence 5938
Current log# 1 seq# 5938 mem# 0: /d02/clndata/log01a.dbf
Current log# 1 seq# 5938 mem# 1: /d02/clndata/log01b.dbf
Successful open of redo thread 1
Wed May 28 09:08:11 2014
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Wed May 28 09:08:11 2014
Incremental checkpoint up to RBA [0x1732.3.0], current log tail at RBA [0x1732.3.0]
Wed May 28 09:08:13 2014
Successfully onlined Undo Tablespace 368.
Wed May 28 09:08:13 2014
SMON: enabling tx recovery
Wed May 28 09:08:13 2014
Database Characterset is UTF8
replication_dependency_tracking turned off (no async multimaster replication found)
Starting background process QMNC
QMNC started with pid=17, OS id=4490
Wed May 28 09:09:16 2014
Completed: ALTER DATABASE OPEN
Trace file
/oracle/clndb/10.2.0/admin/PROD_damacdb/udump/TESTcln_ora_6562.trc
Oracle Database 10g Enterprise Edition Release 10.2.0.3.0 - Production
With the Partitioning, OLAP and Data Mining options
ORACLE_HOME = /oracle/clndb/10.2.0
System name: Linux
Node name: TESTclonedb.damacerp.com
Release: 2.6.9-67.ELsmp
Version: #1 SMP Wed Nov 7 13:58:04 EST 2007
Machine: i686
Instance name: TESTCLN
Redo thread mounted by this instance: 1
Oracle process number: 317
Unix process pid: 6562, image: oracleTESTCLN@TESTclonedb.damacerp.com
*** 2014-05-28 08:56:31.820
*** ACTION NAME:(9287) 2014-05-28 08:56:31.696
*** MODULE NAME:(xdo.oa.template.server.TemplatesAM:R) 2014-05-28 08:56:31.696
*** SERVICE NAME:(TESTCLN) 2014-05-28 08:56:31.696
*** SESSION ID:(1783.20135) 2014-05-28 08:56:31.696
Exception signal: 11 (SIGSEGV), code: 1 (Address not mapped to object), addr: 0xc000095d, PC: [0x98b338b, kokcup()+97]
Registers:
%eax: 0x00000000 %ebx: 0x794034b4 %ecx: 0x794034b4
%edx: 0xc0000929 %edi: 0x0cd3a9e0 %esi: 0xb72da9e8
%esp: 0xbfffb9dc %ebp: 0xbfffb9e8 %eip: 0x098b338b
%efl: 0x00210246
kokcup()+86 (0x98b3380) mov 0x24(%ebx),%edx
kokcup()+89 (0x98b3383) testb $0x1,%dl
kokcup()+94 (0x98b3388) mov 0x18(%ebx),%edx
> kokcup()+97 (0x98b338b) movzw 0x34(%edx),%edx
kokcup()+101 (0x98b338f) test %edx,%edx
kokcup()+103 (0x98b3391) je 0x98b33a8
kokcup()+105 (0x98b3393) cmp $0xffff,%edx
kokcup()+111 (0x98b3399) jge 0x98b33a8
*** 2014-05-28 08:56:31.832
ksedmp: internal or fatal error
ORA-07445: exception encountered: core dump [kokcup()+97] [SIGSEGV] [Address not mapped to object] [0xC000095D] [] []
----- Call Stack Trace -----
calling call entry argument values in hex
location type point (? means dubious value)
-------------------- -------- -------------------- ----------------------------
ksedst()+27 call ksedst1() 1 ? 1 ?
ksedmp()+557 call ksedst() 1 ? 8168F10 ? B748BF3C ?
B80650 ? 10 ? B7491A90 ?
ssexhd()+882 call ksedmp() 3 ? 98B338B ? 636B6F6B ?
29287075 ? 37392B ? 0 ?
kokcup()+97 signal 00000000 B ? B748DC90 ? B748DD10 ?
kocdsfr()+460 call 00000000 CD3A9E0 ? B72DA9E8 ? 1 ?
B71627C8 ? B716152C ?
CCEAB18 ?