Oracle Database Discussions

Dear All,

I have a strange problem that the DB is shutting down automatically during specified timings in a day..

This is happening daily.

What is the problem behind this.

There was a DB crash before from which the DB has been restored. and this is a 11.5.10.2 EBS Production instance.

I have the alert log as mentioned below:

---------------------------------------

Wed Dec 16 17:45:29 2009
Errors in file /prod/oracle/proddb/9.2.0/admin/PROD_oraprod/bdump/prod_smon_22158.trc:
ORA-00600: internal error code, arguments: [kcbgtcr_12], [4], [], [], [], [], [], []
Wed Dec 16 17:45:29 2009
Non-fatal internal error happenned while SMON was doing temporary segment drop.
SMON encountered 98 out of maximum 100 non-fatal internal errors.
Wed Dec 16 17:45:39 2009
Errors in file /prod/oracle/proddb/9.2.0/admin/PROD_oraprod/bdump/prod_smon_22158.trc:
ORA-00600: internal error code, arguments: [kcbgtcr_12], [4], [], [], [], [], [], []
Wed Dec 16 17:45:40 2009
Non-fatal internal error happenned while SMON was doing temporary segment drop.
SMON encountered 99 out of maximum 100 non-fatal internal errors.
Wed Dec 16 17:45:50 2009
Errors in file /prod/oracle/proddb/9.2.0/admin/PROD_oraprod/bdump/prod_smon_22158.trc:
ORA-00600: internal error code, arguments: [kcbgtcr_12], [4], [], [], [], [], [], []
Wed Dec 16 17:45:51 2009
Non-fatal internal error happenned while SMON was doing temporary segment drop.
SMON encountered 100 out of maximum 100 non-fatal internal errors.
Wed Dec 16 17:46:01 2009
Errors in file /prod/oracle/proddb/9.2.0/admin/PROD_oraprod/bdump/prod_smon_22158.trc:
ORA-00600: internal error code, arguments: [kcbgtcr_12], [4], [], [], [], [], [], []
Wed Dec 16 17:46:02 2009
Non-fatal internal error happenned while SMON was doing temporary segment drop.
SMON exceeded the maximum limit of 100 internal error(s).
Wed Dec 16 17:46:02 2009
Errors in file /prod/oracle/proddb/9.2.0/admin/PROD_oraprod/bdump/prod_smon_22158.trc:
ORA-00600: internal error code, arguments: [kcbgtcr_12], [4], [], [], [], [], [], []
SMON: terminating instance due to error 600
Wed Dec 16 17:46:02 2009
Errors in file /prod/oracle/proddb/9.2.0/admin/PROD_oraprod/bdump/prod_pmon_22150.trc:
ORA-00600: internal error code, arguments: [], [], [], [], [], [], [], []
Instance terminated by SMON, pid = 22158
Wed Dec 16 18:06:42 2009
Starting ORACLE instance (normal)
LICENSE_MAX_SESSION = 0
LICENSE_SESSIONS_WARNING = 0
SCN scheme 2
Using log_archive_dest parameter default value
LICENSE_MAX_USERS = 0
SYS auditing is disabled
Starting up ORACLE RDBMS Version: 9.2.0.5.0.
System parameters with non-default values:
tracefiles_public = TRUE
processes = 200
sessions = 400
timed_statistics = TRUE
shared_pool_size = 301989888
shared_pool_reserved_size= 30000000
sharedpool_reserved_min_alloc= 4100
java_pool_size = 67108864
enqueue_resources = 32000
nls_language = american
nls_territory = america
nls_sort = binary
nls_date_format = DD-MON-RR
nls_numeric_characters = .,
nls_comp = binary
nls_length_semantics = BYTE
control_files = /proddata/datafiles/cntrl01.dbf, /proddata/datafiles/cntrl02.dbf, /proddata/datafiles/cntrl03.dbf
db_block_checksum = TRUE
db_block_size = 8192
db_cache_size = 167772160
compatible = 9.2.0
log_buffer = 10485760
log_checkpoint_interval = 100000
log_checkpoint_timeout = 1200
db_files = 512
db_file_multiblock_read_count= 8
log_checkpoints_to_alert = TRUE
dml_locks = 10000
row_locking = always
undo_management = AUTO
undo_tablespace = APPS_UNDOTS2
undo_suppress_errors = FALSE
db_block_checking = FALSE
max_enabled_roles = 100
O7_DICTIONARY_ACCESSIBILITY= FALSE
session_cached_cursors = 200
utl_file_dir = /usr/tmp, /prod/oracle/proddb/9.2.0/appsutil/outbound/PROD_oraprod
job_queue_processes = 2
systemtrig_enabled = TRUE
cursor_sharing = EXACT
parallel_min_servers = 0
parallel_max_servers = 8
background_dump_dest = /prod/oracle/proddb/9.2.0/admin/PROD_oraprod/bdump
user_dump_dest = /prod/oracle/proddb/9.2.0/admin/PROD_oraprod/udump
max_dump_file_size = 20480
core_dump_dest = /prod/oracle/proddb/9.2.0/admin/PROD_oraprod/cdump
optimizer_features_enable= 9.2.0
db_name = PROD
open_cursors = 600
ifile = /prod/oracle/proddb/9.2.0/dbs/PROD_oraprod_ifile.ora
sql_trace = FALSE
sortelimination_cost_ratio= 5
btree_bitmap_plans = FALSE
fastfull_scan_enabled = FALSE
optimizer_max_permutations= 2000
query_rewrite_enabled = true
indexjoin_enabled = FALSE
sqlexecprogression_cost= 2147483647
likewith_bind_as_equality= TRUE
pga_aggregate_target = 1073741824
workarea_size_policy = AUTO
aq_tm_processes = 1
olap_page_pool_size = 4194304
PMON started with pid=2
DBW0 started with pid=3
LGWR started with pid=4
CKPT started with pid=5
SMON started with pid=6
RECO started with pid=7
CJQ0 started with pid=8
QMN0 started with pid=9
Wed Dec 16 18:06:45 2009
ALTER DATABASE MOUNT
Wed Dec 16 18:06:49 2009
Successful mount of redo thread 1, with mount id 117640757
Wed Dec 16 18:06:49 2009
Database mounted in Exclusive Mode.
Completed: ALTER DATABASE MOUNT
Wed Dec 16 18:06:49 2009
ALTER DATABASE OPEN
Wed Dec 16 18:06:49 2009
Beginning crash recovery of 1 threads
Wed Dec 16 18:06:49 2009
Started redo scan
Wed Dec 16 18:06:49 2009
Completed redo scan
14511 redo blocks read, 364 data blocks need recovery
Wed Dec 16 18:06:49 2009
Started recovery at
Thread 1: logseq 3248, block 20031, scn 0.0
Recovery of Online Redo Log: Thread 1 Group 2 Seq 3248 Reading mem 0
Mem# 0 errs 0: /proddata/datafiles/log02a.dbf
Mem# 1 errs 0: /proddata/datafiles/log02b.dbf
Recovery of Online Redo Log: Thread 1 Group 1 Seq 3249 Reading mem 0
Mem# 0 errs 0: /proddata/datafiles/log01a.dbf
Mem# 1 errs 0: /proddata/datafiles/log01b.dbf
Wed Dec 16 18:06:50 2009
Completed redo application
Wed Dec 16 18:06:50 2009
Ended recovery at
Thread 1: logseq 3249, block 14067, scn 0.3237571566
364 data blocks read, 364 data blocks written, 14511 redo blocks read
Crash recovery completed successfully
Wed Dec 16 18:06:50 2009
Thread 1 advanced to log sequence 3250
Thread 1 opened at log sequence 3250
Current log# 2 seq# 3250 mem# 0: /proddata/datafiles/log02a.dbf
Current log# 2 seq# 3250 mem# 1: /proddata/datafiles/log02b.dbf
Successful open of redo thread 1
Wed Dec 16 18:06:50 2009
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Wed Dec 16 18:06:50 2009
SMON: enabling cache recovery
Wed Dec 16 18:06:51 2009
Undo Segment 12 Onlined
Undo Segment 13 Onlined
Undo Segment 14 Onlined
Undo Segment 15 Onlined
Undo Segment 16 Onlined
Undo Segment 17 Onlined
Undo Segment 18 Onlined
Undo Segment 19 Onlined
Undo Segment 20 Onlined
Undo Segment 21 Onlined
Successfully onlined Undo Tablespace 17.
Wed Dec 16 18:06:51 2009
SMON: enabling tx recovery
Wed Dec 16 18:06:51 2009
Database Characterset is AR8MSWIN1256
Wed Dec 16 18:06:51 2009
SMON: about to recover undo segment 22
SMON: mark undo segment 22 as available
SMON: about to recover undo segment 25
SMON: mark undo segment 25 as available
SMON: about to recover undo segment 29
SMON: mark undo segment 29 as available
Wed Dec 16 18:07:04 2009
replication_dependency_tracking turned off (no async multimaster replication found)
Completed: ALTER DATABASE OPEN
Wed Dec 16 18:10:12 2009
Beginning log switch checkpoint up to RBA [0xcb3.2.10], SCN: 0x0000.c0f983be
Thread 1 advanced to log sequence 3251
Current log# 1 seq# 3251 mem# 0: /proddata/datafiles/log01a.dbf
Current log# 1 seq# 3251 mem# 1: /proddata/datafiles/log01b.dbf
Wed Dec 16 18:10:27 2009
Thread 1 cannot allocate new log, sequence 3252
Checkpoint not complete
Current log# 1 seq# 3251 mem# 0: /proddata/datafiles/log01a.dbf
Current log# 1 seq# 3251 mem# 1: /proddata/datafiles/log01b.dbf
Wed Dec 16 18:10:29 2009
Completed checkpoint up to RBA [0xcb3.2.10], SCN: 0x0000.c0f983be
Wed Dec 16 18:10:30 2009
Beginning log switch checkpoint up to RBA [0xcb4.2.10], SCN: 0x0000.c0f98523
Thread 1 advanced to log sequence 3252
Current log# 2 seq# 3252 mem# 0: /proddata/datafiles/log02a.dbf
Current log# 2 seq# 3252 mem# 1: /proddata/datafiles/log02b.dbf
Wed Dec 16 18:17:42 2009
Errors in file /prod/oracle/proddb/9.2.0/admin/PROD_oraprod/udump/prod_ora_2589.trc:
ORA-00600: internal error code, arguments: [13011], [247630], [92371183], [182], [96562970], [0], [], []
Wed Dec 16 18:17:55 2009
Completed checkpoint up to RBA [0xcb4.2.10], SCN: 0x0000.c0f98523
Wed Dec 16 18:22:26 2009
Beginning log switch checkpoint up to RBA [0xcb5.2.10], SCN: 0x0000.c0f9ad8c
Thread 1 advanced to log sequence 3253
Current log# 1 seq# 3253 mem# 0: /proddata/datafiles/log01a.dbf
Current log# 1 seq# 3253 mem# 1: /proddata/datafiles/log01b.dbf
Wed Dec 16 18:41:54 2009
Completed checkpoint up to RBA [0xcb5.2.10], SCN: 0x0000.c0f9ad8c
Wed Dec 16 18:47:38 2009
Beginning log switch checkpoint up to RBA [0xcb6.2.10], SCN: 0x0000.c0fa05a7
Thread 1 advanced to log sequence 3254
Current log# 2 seq# 3254 mem# 0: /proddata/datafiles/log02a.dbf
Current log# 2 seq# 3254 mem# 1: /proddata/datafiles/log02b.dbf
Wed Dec 16 19:08:11 2009
Completed checkpoint up to RBA [0xcb6.2.10], SCN: 0x0000.c0fa05a7
Wed Dec 16 19:14:33 2009
Beginning log switch checkpoint up to RBA [0xcb7.2.10], SCN: 0x0000.c0fa5b8b
Thread 1 advanced to log sequence 3255
Current log# 1 seq# 3255 mem# 0: /proddata/datafiles/log01a.dbf
Current log# 1 seq# 3255 mem# 1: /proddata/datafiles/log01b.dbf
Wed Dec 16 19:34:12 2009
Completed checkpoint up to RBA [0xcb7.2.10], SCN: 0x0000.c0fa5b8b
Wed Dec 16 19:45:33 2009
Beginning log switch checkpoint up to RBA [0xcb8.2.10], SCN: 0x0000.c0fabbd5
Thread 1 advanced to log sequence 3256
Current log# 2 seq# 3256 mem# 0: /proddata/datafiles/log02a.dbf
Current log# 2 seq# 3256 mem# 1: /proddata/datafiles/log02b.dbf
Wed Dec 16 20:06:09 2009
Completed checkpoint up to RBA [0xcb8.2.10], SCN: 0x0000.c0fabbd5
Wed Dec 16 20:10:03 2009
Shutting down instance: further logons disabled
Shutting down instance (immediate)
License high water mark = 85
Wed Dec 16 20:10:04 2009
Errors in file /prod/oracle/proddb/9.2.0/admin/PROD_oraprod/bdump/prod_smon_2082.trc:
ORA-00600: internal error code, arguments: [kcbgtcr_12], [4], [], [], [], [], [], []
Wed Dec 16 20:10:05 2009
Non-fatal internal error happenned while SMON was doing temporary segment drop.
SMON encountered 1 out of maximum 100 non-fatal internal errors.
Wed Dec 16 20:10:06 2009
ALTER DATABASE CLOSE NORMAL
Wed Dec 16 20:10:07 2009
Errors in file /prod/oracle/proddb/9.2.0/admin/PROD_oraprod/bdump/prod_smon_2082.trc:
ORA-00600: internal error code, arguments: [kcbgtcr_12], [4], [], [], [], [], [], []
Non-fatal internal error happenned while SMON was doing temporary segment drop.
SMON encountered 2 out of maximum 100 non-fatal internal errors.
SMON: disabling tx recovery
SMON: disabling cache recovery
Wed Dec 16 20:10:08 2009
Shutting down archive processes
Archiving is disabled
Archive process shutdown avoided: 0 active
Thread 1 closed at log sequence 3256
Successful close of redo thread 1
Wed Dec 16 20:10:08 2009
Completed: ALTER DATABASE CLOSE NORMAL
Wed Dec 16 20:10:08 2009
ALTER DATABASE DISMOUNT
Completed: ALTER DATABASE DISMOUNT
ARCH: Archiving is disabled
Shutting down archive processes
Archiving is disabled
Archive process shutdown avoided: 0 active
ARCH: Archiving is disabled
Shutting down archive processes
Archiving is disabled
Archive process shutdown avoided: 0 active
Wed Dec 16 20:10:32 2009

----------------------

We were not able to identify what is the cause of this shutting down. Each time the DB has to be started again...

How can I rectify this...

Please update...

many thanks in advance......