Oracle Database Discussions

Hi , I´ve an oracle database 9.2.0.7 64 bits in cluster mode, in the cluster I´ve two instances more, but this error only appears in one, this error is appears three or four times in the same week, but always is in node2, but other instance in node2 haven´t any error. I don´t know what is happening, any idea for resolve this?

the result of alert.log is:
Thu Mar 17 10:13:04 2011
Errors in file :
ORA-07445: exception encountered: core dump [kpufdesc()+258] [SIGSEGV] [Address not mapped to object] [0x000000010] [] []
Thu Mar 17 10:17:15 2011
Errors in file /sinee1/admin/sinee9i/udump/sinee9i_ora_3524.trc:
ORA-07445: exception encountered: core dump [kcbgtcr()+1049] [SIGSEGV] [Address not mapped to object] [0x395EE79B8] [] []
Thu Mar 17 10:20:54 2011
Errors in file /sinee1/admin/sinee9i/udump/sinee9i_ora_2691.trc:
ORA-07445: exception encountered: core dump [kcbgtcr()+1049] [SIGSEGV] [Address not mapped to object] [0x395EE79B8] [] []
Thu Mar 17 10:22:54 2011
Errors in file /sinee1/admin/sinee9i/udump/sinee9i_ora_14148.trc:
ORA-07445: exception encountered: core dump [kcbgtcr()+1049] [SIGSEGV] [Address not mapped to object] [0x395EE79B8] [] []
Thu Mar 17 10:23:19 2011
Errors in file /sinee1/admin/sinee9i/udump/sinee9i_ora_16225.trc:
ORA-07445: exception encountered: core dump [kcbgtcr()+1049] [SIGSEGV] [Address not mapped to object] [0x395EE79B8] [] []
Thu Mar 17 10:29:22 2011
Errors in file /sinee1/admin/sinee9i/udump/sinee9i_ora_23638.trc:
ORA-07445: exception encountered: core dump [kcbgtcr()+1049] [SIGSEGV] [Address not mapped to object] [0x395EE79B8] [] []
Thu Mar 17 10:51:03 2011
Errors in file /sinee1/admin/sinee9i/udump/sinee9i_ora_5224.trc:
ORA-07445: exception encountered: core dump [kcbgtcr()+1049] [SIGSEGV] [Address not mapped to object] [0x395EE79B8] [] []
Thu Mar 17 10:51:03 2011
Errors in file /sinee1/admin/sinee9i/udump/sinee9i_ora_5224.trc:
ORA-07445: exception encountered: core dump [kgidmp()+774] [SIGSEGV] [unknown code] [0x000000000] [] []
ORA-07445: exception encountered: core dump [kcbgtcr()+1049] [SIGSEGV] [Address not mapped to object] [0x395EE79B8] [] []
Thu Mar 17 10:51:03 2011
Errors in file /sinee1/admin/sinee9i/udump/sinee9i_ora_5224.trc:
ORA-07445: exception encountered: core dump [kgidmp()+774] [SIGSEGV] [unknown code] [0x000000000] [] []
ORA-07445: exception encountered: core dump [kgidmp()+774] [SIGSEGV] [unknown code] [0x000000000] [] []
ORA-07445: exception encountered: core dump [kcbgtcr()+1049] [SIGSEGV] [Address not mapped to object] [0x395EE79B8] [] []
Thu Mar 17 10:51:09 2011
Errors in file /sinee1/admin/sinee9i/udump/sinee9i_ora_27937.trc:
ORA-07445: exception encountered: core dump [kcbgtcr()+1049] [SIGSEGV] [unknown code] [0x000000000] [] []
Thu Mar 17 10:53:36 2011
Errors in file /sinee1/admin/sinee9i/bdump/sinee9i_ckpt_13811.trc:
ORA-07445: exception encountered: core dump [kcbkzx()+908] [SIGSEGV] [Address not mapped to object] [0x000000011] [] []
Thu Mar 17 10:53:38 2011
Errors in file /sinee1/admin/sinee9i/bdump/sinee9i_pmon_13801.trc:
ORA-00470: LGWR process terminated with error
Thu Mar 17 10:53:38 2011
PMON: terminating instance due to error 470
Instance terminated by PMON, pid = 13801
Thu Mar 17 10:55:48 2011
Starting ORACLE instance (normal)
LICENSE_MAX_SESSION = 0
LICENSE_SESSIONS_WARNING = 0
SCN scheme 3
Using log_archive_dest parameter default value
LICENSE_MAX_USERS = 0
SYS auditing is disabled
Starting up ORACLE RDBMS Version: 9.2.0.7.0.
System parameters with non-default values:
processes = 300
timed_statistics = TRUE
shared_pool_size = 2315255808
sga_max_size = 4299135488
large_pool_size = 33554432
java_pool_size = 251658240
spfile = /sinee1/dbs/spfilesinee9i.ora
control_files = /sinee1/oradata/sinee9i/control01.ctl, /sinee1/oradata/sinee9i/control02.ctl, /sinee1/oradata/sinee9i/control03.ctl
db_block_size = 8192
db_cache_size = 1073741824
compatible = 9.2.0.0.0
log_archive_start = TRUE
log_archive_dest_1 = LOCATION=/sinee_log/archive
log_archive_format = %t_%s.dbf
db_file_multiblock_read_count= 16
fast_start_mttr_target = 300
undo_management = AUTO
undo_tablespace = UNDOTBS1
undo_retention = 10800
remote_login_passwordfile= EXCLUSIVE
db_domain =
instance_name = sinee9i
dispatchers = (PROTOCOL=TCP) (SERVICE=sinee9iXDB)
session_cached_cursors = 250
job_queue_processes = 10
hash_join_enabled = TRUE
background_dump_dest = /sinee1/admin/sinee9i/bdump
user_dump_dest = /sinee1/admin/sinee9i/udump
core_dump_dest = /sinee1/admin/sinee9i/cdump
sort_area_size = 524288
db_name = sinee9i
open_cursors = 500
star_transformation_enabled= FALSE
query_rewrite_enabled = FALSE
pga_aggregate_target = 2147483648
aq_tm_processes = 1
PMON started with pid=2
DBW0 started with pid=3
DBW1 started with pid=4
DBW2 started with pid=5
LGWR started with pid=6
CKPT started with pid=7
SMON started with pid=8
RECO started with pid=9
CJQ0 started with pid=10
Thu Mar 17 10:55:48 2011
starting up 1 shared server(s) ...
QMN0 started with pid=11
Thu Mar 17 10:55:48 2011
starting up 1 dispatcher(s) for network address '(ADDRESS=(PARTIAL=YES)(PROTOCOL=TCP))'...
ARCH: STARTING ARCH PROCESSES
ARC0 started with pid=14
ARC0: Archival started
ARC1 started with pid=15
Thu Mar 17 10:55:49 2011
ARCH: STARTING ARCH PROCESSES COMPLETE
Thu Mar 17 10:55:49 2011
ARC0: Becoming the 'no FAL' ARCH
Thu Mar 17 10:55:49 2011
ARC1: Archival started
Thu Mar 17 10:55:49 2011
ARC0: Becoming the 'no FAL' ARCHARC0: Thread not mounted
Thu Mar 17 10:55:49 2011
ARC1: Becoming the heartbeat ARCH
ARC1: Becoming the heartbeat ARCHARC1: Thread not mounted
Thu Mar 17 10:55:49 2011
ALTER DATABASE MOUNT
Thu Mar 17 10:55:53 2011
Successful mount of redo thread 1, with mount id 3567685621
Thu Mar 17 10:55:53 2011
Database mounted in Exclusive Mode.
Completed: ALTER DATABASE MOUNT
Thu Mar 17 10:55:53 2011
ALTER DATABASE OPEN
Thu Mar 17 10:55:53 2011
Beginning crash recovery of 1 threads
Thu Mar 17 10:55:53 2011
Started redo scan
Thu Mar 17 10:55:53 2011
Completed redo scan
2917 redo blocks read, 166 data blocks need recovery
Thu Mar 17 10:55:54 2011
Started recovery at
Thread 1: logseq 2625, block 20639, scn 0.0
Thu Mar 17 10:55:54 2011
Recovery of Online Redo Log: Thread 1 Group 3 Seq 2625 Reading mem 0
Mem# 0 errs 0: /sinee_log/redo/redo03.log
Thu Mar 17 10:55:54 2011
Completed redo application
Thu Mar 17 10:55:54 2011
Ended recovery at
Thread 1: logseq 2625, block 23556, scn 1254.610554787
166 data blocks read, 162 data blocks written, 2917 redo blocks read
Crash recovery completed successfully
Thu Mar 17 10:55:54 2011
LGWR: Primary database is in CLUSTER CONSISTENT mode
Thread 1 advanced to log sequence 2626
Thread 1 opened at log sequence 2626
Current log# 1 seq# 2626 mem# 0: /sinee_log/redo/redo01.log
Successful open of redo thread 1
Thu Mar 17 10:55:54 2011
ARC1: Evaluating archive log 3 thread 1 sequence 2625
ARC1: Beginning to archive log 3 thread 1 sequence 2625
Creating archive destination LOG_ARCHIVE_DEST_1: '/sinee_log/archive/1_2625.dbf'
Thu Mar 17 10:55:54 2011
ARC0: Evaluating archive log 3 thread 1 sequence 2625
Thu Mar 17 10:55:54 2011
ARC0: Unable to archive log 3 thread 1 sequence 2625
Log actively being archived by another process
Thu Mar 17 10:55:54 2011
SMON: enabling cache recovery
Thu Mar 17 10:55:55 2011
ARC1: Completed archiving log 3 thread 1 sequence 2625
Thu Mar 17 10:55:55 2011
Successfully onlined Undo Tablespace 1.
Thu Mar 17 10:55:55 2011
SMON: enabling tx recovery
Thu Mar 17 10:55:58 2011
Database Characterset is WE8ISO8859P1
replication_dependency_tracking turned off (no async multimaster replication found)
Completed: ALTER DATABASE OPEN
Thu Mar 17 17:00:06 2011
Thread 1 advanced to log sequence 2627
Current log# 2 seq# 2627 mem# 0: /sinee_log/redo/redo02.log
Thu Mar 17 17:00:06 2011
ARC0: Evaluating archive log 1 thread 1 sequence 2626
ARC0: Beginning to archive log 1 thread 1 sequence 2626
Creating archive destination LOG_ARCHIVE_DEST_1: '/sinee_log/archive/1_2626.dbf'
Thu Mar 17 17:00:06 2011
ARCH: Evaluating archive log 1 thread 1 sequence 2626
ARCH: Unable to archive log 1 thread 1 sequence 2626
Log actively being archived by another process
Thu Mar 17 17:00:06 2011
ARC0: Completed archiving log 1 thread 1 sequence 2626
Thu Mar 17 17:06:19 2011
Thread 1 advanced to log sequence 2628
Current log# 3 seq# 2628 mem# 0: /sinee_log/redo/redo03.log
Thu Mar 17 17:06:19 2011
ARC0: Evaluating archive log 2 thread 1 sequence 2627
ARC0: Beginning to archive log 2 thread 1 sequence 2627
Creating archive destination LOG_ARCHIVE_DEST_1: '/sinee_log/archive/1_2627.dbf'
Thu Mar 17 17:06:19 2011
ARCH: Evaluating archive log 2 thread 1 sequence 2627
ARCH: Unable to archive log 2 thread 1 sequence 2627
Log actively being archived by another process
Thu Mar 17 17:06:19 2011
ARC0: Completed archiving log 2 thread 1 sequence 2627
Thu Mar 17 17:07:27 2011
ALTER DATABASE BACKUP CONTROLFILE TO '/var/opt/omni/tmp/ctrl_sinee9i.dbf' REUSE
Thu Mar 17 17:07:27 2011
Completed: ALTER DATABASE BACKUP CONTROLFILE TO '/var/opt/omn
Thu Mar 17 21:30:23 2011
Thread 1 advanced to log sequence 2629
Current log# 1 seq# 2629 mem# 0: /sinee_log/redo/redo01.log
Thu Mar 17 21:30:23 2011
ARC0: Evaluating archive log 3 thread 1 sequence 2628
ARC0: Beginning to archive log 3 thread 1 sequence 2628
Creating archive destination LOG_ARCHIVE_DEST_1: '/sinee_log/archive/1_2628.dbf'
ARC0: Completed archiving log 3 thread 1 sequence 2628
Thu Mar 17 21:31:21 2011
ARC0: Evaluating archive log 1 thread 1 sequence 2629
Thu Mar 17 21:31:21 2011
Thread 1 advanced to log sequence 2630
Thu Mar 17 21:31:21 2011
Current log# 2 seq# 2630 mem# 0: /sinee_log/redo/redo02.log
Thu Mar 17 21:31:21 2011
ARC0: Beginning to archive log 1 thread 1 sequence 2629
Creating archive destination LOG_ARCHIVE_DEST_1: '/sinee_log/archive/1_2629.dbf'
ARC0: Completed archiving log 1 thread 1 sequence 2629





the result of messages is:

Mar 17 10:13:04 blake6 kernel: oracle[2595]: segfault at 0000000095feb028 rip 0000003d727cb2b1 rsp 0000000095feb028 error 4
Mar 17 10:17:16 blake6 kernel: oracle[3524]: segfault at 0000000095feb028 rip 0000003d727cb2b1 rsp 0000000095feb028 error 4
Mar 17 10:20:54 blake6 kernel: oracle[2691]: segfault at 0000000095feb028 rip 0000003d727cb2b1 rsp 0000000095feb028 error 4
Mar 17 10:22:55 blake6 kernel: oracle[14148]: segfault at 0000000095feb028 rip 0000003d727cb2b1 rsp 0000000095feb028 error 4
Mar 17 10:23:20 blake6 kernel: oracle[16225]: segfault at 0000000095feb028 rip 0000003d727cb2b1 rsp 0000000095feb028 error 4
Mar 17 10:29:22 blake6 kernel: oracle[23638]: segfault at 0000000095feb028 rip 0000003d727cb2b1 rsp 0000000095feb028 error 4
Mar 17 10:38:40 blake6 snmpd[1755]: Received SNMP packet(s) from 10.18.2.139
Mar 17 10:51:03 blake6 kernel: oracle[5224]: segfault at 0000000095fe9c28 rip 0000003d727cb2b1 rsp 0000000095fe9c28 error 4
Mar 17 10:51:10 blake6 kernel: oracle[27937]: segfault at 0000000095feb028 rip 0000003d727cb2b1 rsp 0000000095feb028 error 4
Mar 17 10:53:36 blake6 kernel: oracle[13811]: segfault at 0000000095feb028 rip 0000003d727cb2b1 rsp 0000000095feb028 error 4
Mar 17 10:54:02 blake6 cmcld[31310]: Service sinee9i_service terminated due to an exit(1).
Mar 17 10:54:02 blake6 cmcld[31310]: Automatically restarted service sinee9i_service for the 1st time after failure.
Mar 17 10:54:36 blake6 cmcld[31310]: Service sinee9i_service terminated due to an exit(1).
Mar 17 10:54:36 blake6 cmcld[31310]: Automatically restarted service sinee9i_service for the 2nd time after failure.
Mar 17 10:55:10 blake6 cmcld[31310]: Service sinee9i_service terminated due to an exit(1).
Mar 17 10:55:10 blake6 cmcld[31310]: Automatically restarted service sinee9i_service for the 3rd time after failure.
Mar 17 10:55:40 blake6 cmcld[31310]: Service sinee9i_service terminated due to an exit(1).
Mar 17 10:55:40 blake6 cmcld[31310]: Service sinee9i_service in package sinee9i has gone down.
Mar 17 10:55:40 blake6 cmcld[31310]: Disabled node blake6 from running package sinee9i.
Mar 17 10:55:40 blake6 cmcld[31310]: Failing package sinee9i on node blake6 due to service failure.
Mar 17 10:55:40 blake6 cmcld[31310]: Request from node blake6 to fail package sinee9i on node blake6.
Mar 17 10:55:40 blake6 cmcld[31310]: Executing '/usr/local/cmcluster/conf/sinee9i/sinee9i.cntl stop' for package sinee9i, as service PKG*43779.
Mar 17 10:55:40 blake6 cmhaltserv: cmhaltserv sinee9i_service
Mar 17 10:55:41 blake6 su(pam_unix)[27335]: session opened for user oracle by (uid=0)
Mar 17 10:55:41 blake6 su(pam_unix)[27335]: session closed for user oracle
Mar 17 10:55:42 blake6 cmmodnet: cmmodnet -r -i 10.18.3.137 10.18.3.0
Mar 17 10:55:43 blake6 udevd[3071]: udev done!
Mar 17 10:55:44 blake6 last message repeated 2 times
Mar 17 10:55:44 blake6 cmcld[31310]: Service PKG*43779 terminated due to an exit(0).
Mar 17 10:55:44 blake6 cmcld[31310]: Halted package sinee9i on node blake6.
Mar 17 10:56:00 blake6 cmcld[31310]: (blake5) Started package sinee9i on node blake5.


Can You help me, please?

thanks