Oracle Database Discussions

My instance is automatilly crashing after some time after startup by throwing following error, plz help me

ksdpec: called for event 13740 prior to event group initialization
Starting up ORACLE RDBMS Version: 10.2.0.1.0.
System parameters with non-default values:
processes = 150
__shared_pool_size = 331350016
__large_pool_size = 4194304
__java_pool_size = 8388608
__streams_pool_size = 0
sga_target = 612368384
control_files = /dev/vg_rac/rkumaondb_control1_raw, /dev/vg_rac/rkumaondb_control2_raw
db_block_size = 8192
__db_cache_size = 264241152
compatible = 10.2.0.1.0
db_file_multiblock_read_count= 16
cluster_database = TRUE
cluster_database_instances= 2
thread = 2
instance_number = 2
undo_management = AUTO
undo_tablespace = UNDOTBS2
remote_login_passwordfile= EXCLUSIVE
db_domain =
dispatchers = (PROTOCOL=TCP) (SERVICE=kumaondbXDB)
remote_listener = LISTENERS_KUMAONDB
job_queue_processes = 10
background_dump_dest = /u01/app/oracle/product/10.2.0/db_1/admin/kumaondb/bdump
user_dump_dest = /u01/app/oracle/product/10.2.0/db_1/admin/kumaondb/udump
core_dump_dest = /u01/app/oracle/product/10.2.0/db_1/admin/kumaondb/cdump
audit_file_dest = /u01/app/oracle/product/10.2.0/db_1/admin/kumaondb/adump
db_name = kumaondb
open_cursors = 300
pga_aggregate_target = 203423744
Cluster communication is configured to use the following interface(s) for this instance
80.0.0.2
Thu Jul 20 11:04:55 2006
cluster interconnect IPC version:Oracle UDP/IP
IPC Vendor 1 proto 2
PMON started with pid=2, OS id=25268
DIAG started with pid=3, OS id=25270
PSP0 started with pid=4, OS id=25272
LMON started with pid=5, OS id=25274
LMD0 started with pid=6, OS id=25276
LMS0 started with pid=7, OS id=25278
MMAN started with pid=8, OS id=25280
LGWR started with pid=10, OS id=25286
DBW0 started with pid=9, OS id=25282
CKPT started with pid=11, OS id=25313
SMON started with pid=12, OS id=25315
RECO started with pid=13, OS id=25317
CJQ0 started with pid=14, OS id=25319
MMON started with pid=15, OS id=25321
MMNL started with pid=16, OS id=25323
Thu Jul 20 11:04:56 2006
starting up 1 dispatcher(s) for network address '(ADDRESS=(PARTIAL=YES)(PROTOCOL=TCP))'...
starting up 1 shared server(s) ...
Thu Jul 20 11:04:57 2006
lmon registered with NM - instance id 2 (internal mem no 1)
Thu Jul 20 11:04:58 2006
Reconfiguration started (old inc 0, new inc 20)
pseudo shared rm latch used
List of nodes:
0 1
Global Resource Directory frozen
* allocate domain 0, invalid = TRUE
Communication channels reestablished
* domain 0 valid = 1 according to instance 0
Thu Jul 20 11:04:59 2006
Master broadcasted resource hash value bitmaps
Non-local Process blocks cleaned out
Thu Jul 20 11:04:59 2006
LMS 0: 0 GCS shadows cancelled, 0 closed
Set master node info
Submitted all remote-enqueue requests
Dwn-cvts replayed, VALBLKs dubious
All grantable enqueues granted
Thu Jul 20 11:04:59 2006
LMS 0: 0 GCS shadows traversed, 0 replayed
Thu Jul 20 11:04:59 2006
Submitted all GCS remote-cache requests
Fix write in gcs resources
Reconfiguration complete
LCK0 started with pid=19, OS id=25410
Thu Jul 20 11:05:00 2006
ALTER DATABASE MOUNT
Thu Jul 20 11:05:04 2006
Setting recovery target incarnation to 2
Thu Jul 20 11:05:04 2006
Successful mount of redo thread 2, with mount id 4110319156
Thu Jul 20 11:05:04 2006
Database mounted in Shared Mode (CLUSTER_DATABASE=TRUE)
Completed: ALTER DATABASE MOUNT
Thu Jul 20 11:05:05 2006
ALTER DATABASE OPEN
Picked broadcast on commit scheme to generate SCNs
Thu Jul 20 11:05:05 2006
Errors in file /u01/app/oracle/product/10.2.0/db_1/admin/kumaondb/bdump/kumaondb2_mmon_25321.trc:
ORA-00600: internal error code, arguments: [kmgs_parameter_update_timeout_1], [27091], [], [], [], [], [], []
ORA-27091: unable to queue I/O
ORA-27072: File I/O error
HP-UX Error: 9: Bad file number
Additional information: 4
Additional information: 4
Additional information: -1
Thu Jul 20 11:05:05 2006
Thread 2 opened at log sequence 76
Current log# 4 seq# 76 mem# 0: /dev/vg_rac/rkumaondb_redo2_2_raw
Successful open of redo thread 2
Thu Jul 20 11:05:05 2006
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Thu Jul 20 11:05:05 2006
SMON: enabling cache recovery
Thu Jul 20 11:05:05 2006
Successfully onlined Undo Tablespace 5.
Thu Jul 20 11:05:05 2006
SMON: enabling tx recovery
Thu Jul 20 11:05:06 2006
Database Characterset is WE8ISO8859P1
replication_dependency_tracking turned off (no async multimaster replication found)
Starting background process QMNC
QMNC started with pid=21, OS id=25577
Thu Jul 20 11:05:08 2006
Completed: ALTER DATABASE OPEN
Thu Jul 20 11:05:16 2006
Trace dumping is performing id=[cdmp_20060720110508]
Thu Jul 20 11:05:19 2006
ALTER SYSTEM SET service_names='kumaondb','orcl' SCOPE=MEMORY SID='kumaondb2';
Thu Jul 20 11:06:05 2006
Errors in file /u01/app/oracle/product/10.2.0/db_1/admin/kumaondb/bdump/kumaondb2_dbw0_25282.trc:
ORA-00600: internal error code, arguments: [kmgs_parameter_update_timeout_1], [27091], [], [], [], [], [], []
ORA-27091: unable to queue I/O
ORA-27072: File I/O error
HP-UX Error: 9: Bad file number
Additional information: 4
Additional information: 4
Additional information: -1
Thu Jul 20 11:06:06 2006
Errors in file /u01/app/oracle/product/10.2.0/db_1/admin/kumaondb/bdump/kumaondb2_dbw0_25282.trc:
ORA-00600: internal error code, arguments: [kmgs_parameter_update_timeout_1], [27091], [], [], [], [], [], []
ORA-27091: unable to queue I/O
ORA-27072: File I/O error
HP-UX Error: 9: Bad file number
Additional information: 4
Additional information: 4
Additional information: -1
Thu Jul 20 11:06:06 2006
DBW0: terminating instance due to error 471
Thu Jul 20 11:06:06 2006
Errors in file /u01/app/oracle/product/10.2.0/db_1/admin/kumaondb/bdump/kumaondb2_lms0_25278.trc:
ORA-00471: DBWR process terminated with error
Thu Jul 20 11:06:06 2006
Trace dumping is performing id=[cdmp_20060720110606]
Thu Jul 20 11:06:06 2006
Errors in file /u01/app/oracle/product/10.2.0/db_1/admin/kumaondb/bdump/kumaondb2_lmd0_25276.trc:
ORA-00471: DBWR process terminated with error
Thu Jul 20 11:06:06 2006
Errors in file /u01/app/oracle/product/10.2.0/db_1/admin/kumaondb/bdump/kumaondb2_lmon_25274.trc:
ORA-00471: DBWR process terminated with error
Thu Jul 20 11:06:12 2006
Instance terminated by DBW0, pid = 25282
Thu Jul 20 11:06:40 2006
Starting ORACLE instance (normal)
LICENSE_MAX_SESSION = 0
LICENSE_SESSIONS_WARNING = 0
Interface type 1 lan0 80.0.0.0 configured from OCR for use as a cluster interconnect
Interface type 1 lan900 198.100.0.0 configured from OCR for use as a public interface
Picked latch-free SCN scheme 1
Using LOG_ARCHIVE_DEST_1 parameter default value as /u01/app/oracle/product/10.2.0/db_1/dbs/arch
Autotune of undo retention is turned on.
LICENSE_MAX_USERS = 0
SYS auditing is disabled
ksdpec: called for event 13740 prior to event group initialization
Starting up ORACLE RDBMS Version: 10.2.0.1.0.
System parameters with non-default values:
processes = 150
__shared_pool_size = 331350016
__large_pool_size = 4194304
__java_pool_size = 8388608
__streams_pool_size = 0
sga_target = 612368384
control_files = /dev/vg_rac/rkumaondb_control1_raw, /dev/vg_rac/rkumaondb_control2_raw
db_block_size = 8192
__db_cache_size = 264241152
compatible = 10.2.0.1.0
db_file_multiblock_read_count= 16
cluster_database = TRUE
cluster_database_instances= 2
thread = 2
instance_number = 2
undo_management = AUTO
undo_tablespace = UNDOTBS2
remote_login_passwordfile= EXCLUSIVE
db_domain =
dispatchers = (PROTOCOL=TCP) (SERVICE=kumaondbXDB)
remote_listener = LISTENERS_KUMAONDB
job_queue_processes = 10
background_dump_dest = /u01/app/oracle/product/10.2.0/db_1/admin/kumaondb/bdump
user_dump_dest = /u01/app/oracle/product/10.2.0/db_1/admin/kumaondb/udump
core_dump_dest = /u01/app/oracle/product/10.2.0/db_1/admin/kumaondb/cdump
audit_file_dest = /u01/app/oracle/product/10.2.0/db_1/admin/kumaondb/adump
db_name = kumaondb
open_cursors = 300
pga_aggregate_target = 203423744
Cluster communication is configured to use the following interface(s) for this instance
80.0.0.2
Thu Jul 20 11:06:41 2006
cluster interconnect IPC version:Oracle UDP/IP
IPC Vendor 1 proto 2
PMON started with pid=2, OS id=27992
DIAG started with pid=3, OS id=27994
PSP0 started with pid=4, OS id=27996
LMON started with pid=5, OS id=27998
LMD0 started with pid=6, OS id=28000
LMS0 started with pid=7, OS id=28025
MMAN started with pid=8, OS id=28031
DBW0 started with pid=9, OS id=28033
CKPT started with pid=11, OS id=28037
LGWR started with pid=10, OS id=28035
SMON started with pid=12, OS id=28039
RECO started with pid=13, OS id=28041
CJQ0 started with pid=14, OS id=28043
MMON started with pid=15, OS id=28045
Thu Jul 20 11:06:42 2006
starting up 1 dispatcher(s) for network address '(ADDRESS=(PARTIAL=YES)(PROTOCOL=TCP))'...
MMNL started with pid=16, OS id=28047
Thu Jul 20 11:06:43 2006
starting up 1 shared server(s) ...
Thu Jul 20 11:06:43 2006
lmon registered with NM - instance id 2 (internal mem no 1)
Thu Jul 20 11:06:44 2006
Reconfiguration started (old inc 0, new inc 24)
pseudo shared rm latch used
List of nodes:
0 1
Global Resource Directory frozen
* allocate domain 0, invalid = TRUE
Communication channels reestablished
* domain 0 valid = 1 according to instance 0
Thu Jul 20 11:06:45 2006
Master broadcasted resource hash value bitmaps
Non-local Process blocks cleaned out
Thu Jul 20 11:06:45 2006
LMS 0: 0 GCS shadows cancelled, 0 closed
Set master node info
Submitted all remote-enqueue requests
Dwn-cvts replayed, VALBLKs dubious
All grantable enqueues granted
Thu Jul 20 11:06:45 2006
LMS 0: 0 GCS shadows traversed, 0 replayed
Thu Jul 20 11:06:45 2006
Submitted all GCS remote-cache requests
Fix write in gcs resources
Reconfiguration complete
LCK0 started with pid=19, OS id=28134
Thu Jul 20 11:06:46 2006
ALTER DATABASE MOUNT
Thu Jul 20 11:06:50 2006
Setting recovery target incarnation to 2
Thu Jul 20 11:06:50 2006
Successful mount of redo thread 2, with mount id 4110319156
Thu Jul 20 11:06:50 2006
Database mounted in Shared Mode (CLUSTER_DATABASE=TRUE)
Completed: ALTER DATABASE MOUNT
Thu Jul 20 11:06:51 2006
ALTER DATABASE OPEN
Picked broadcast on commit scheme to generate SCNs
Thu Jul 20 11:06:51 2006
Thread 2 opened at log sequence 77
Current log# 3 seq# 77 mem# 0: /dev/vg_rac/rkumaondb_redo2_1_raw
Successful open of redo thread 2
Thu Jul 20 11:06:51 2006
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Thu Jul 20 11:06:51 2006
SMON: enabling cache recovery
Thu Jul 20 11:06:51 2006
Errors in file /u01/app/oracle/product/10.2.0/db_1/admin/kumaondb/bdump/kumaondb2_mmon_28045.trc:
ORA-00600: internal error code, arguments: [kmgs_parameter_update_timeout_1], [27091], [], [], [], [], [], []
ORA-27091: unable to queue I/O
ORA-27072: File I/O error
HP-UX Error: 9: Bad file number
Additional information: 4
Additional information: 4
Additional information: -1
Thu Jul 20 11:06:51 2006
Successfully onlined Undo Tablespace 5.
Thu Jul 20 11:06:51 2006
SMON: enabling tx recovery
Thu Jul 20 11:06:51 2006
Database Characterset is WE8ISO8859P1
replication_dependency_tracking turned off (no async multimaster replication found)
Starting background process QMNC
QMNC started with pid=21, OS id=28328
Thu Jul 20 11:06:54 2006
Completed: ALTER DATABASE OPEN
Thu Jul 20 11:07:02 2006
Trace dumping is performing id=[cdmp_20060720110653]
Thu Jul 20 11:07:04 2006
ALTER SYSTEM SET service_names='kumaondb','orcl' SCOPE=MEMORY SID='kumaondb2';
Thu Jul 20 11:11:52 2006
Errors in file /u01/app/oracle/product/10.2.0/db_1/admin/kumaondb/bdump/kumaondb2_dbw0_28033.trc:
ORA-00600: internal error code, arguments: [kmgs_parameter_update_timeout_1], [27091], [], [], [], [], [], []
ORA-27091: unable to queue I/O
ORA-27072: File I/O error
HP-UX Error: 9: Bad file number
Additional information: 4
Additional information: 4
Additional information: -1
Thu Jul 20 11:11:53 2006
Errors in file /u01/app/oracle/product/10.2.0/db_1/admin/kumaondb/bdump/kumaondb2_dbw0_28033.trc:
ORA-00600: internal error code, arguments: [kmgs_parameter_update_timeout_1], [27091], [], [], [], [], [], []
ORA-27091: unable to queue I/O
ORA-27072: File I/O error
HP-UX Error: 9: Bad file number
Additional information: 4
Additional information: 4
Additional information: -1
Thu Jul 20 11:11:53 2006
DBW0: terminating instance due to error 471
Thu Jul 20 11:11:53 2006
Errors in file /u01/app/oracle/product/10.2.0/db_1/admin/kumaondb/bdump/kumaondb2_smon_28039.trc:
ORA-00471: DBWR process terminated with error
Thu Jul 20 11:11:53 2006
Errors in file /u01/app/oracle/product/10.2.0/db_1/admin/kumaondb/bdump/kumaondb2_lms0_28025.trc:
ORA-00471: DBWR process terminated with error
Thu Jul 20 11:11:53 2006
Trace dumping is performing id=[cdmp_20060720111153]
Thu Jul 20 11:11:59 2006
Instance terminated by DBW0, pid = 28033