Oracle Database Discussions

Dear All,

Currently my client is using oracle database 10.2.0.4.0 (RAC). However occasionally during processing (the application trying to write/modify data), they experience multiple ORA-00481 errors and then the database crashed. But I would say it's sort of automatic restart as the log said so.

Trying to search on some articles about this, and found one saying this is because of internal bug, and problem fixed with patch 3659289.

So far, I couldn't find much information about this patch (3659289) and I need some help here.

1. I'm wondering if this is the case from my client's database?.
2. How to check if this specific patch (3659289) has been applied?
3. Would be appreciate if you have access to this patch and post the details here.

Advises, Suggestion or jokes will be appreciated. Thanks in advance!

Thanks and Best Regards,
CT



Some additional information:
1. Oracle database version 10.2.0.4.0

Connected to:
Oracle Database 10g Release 10.2.0.4.0 - 64bit Production
With the Real Application Clusters option

2. RAC with two nodes.

3. Running on Solaris 10 sparc infrastructure.

4. What error log said:

Sun Dec 4 00:51:09 2011
Errors in file /opt/app/oracle/admin/xxxxx/bdump/xxxxx1_lms1_16541.trc:
Sun Dec 4 00:51:34 2011
Trace dumping is performing id=[cdmp_20111204003639]
Sun Dec 4 00:52:03 2011
kkjcre1p: unable to spawn jobq slave process
Sun Dec 4 00:52:04 2011
Errors in file /opt/app/oracle/admin/xxxxx/bdump/xxxxx1_cjq0_16598.trc:

Sun Dec 4 00:54:58 2011
Errors in file /opt/app/oracle/admin/xxxxx/bdump/xxxxx1_lmon_16525.trc:
ORA-29740: evicted by member 1, group incarnation 74
Sun Dec 4 00:54:59 2011
Errors in file /opt/app/oracle/admin/xxxxx/bdump/xxxxx1_j002_14521.trc:
ORA-00481: LMON process terminated with error
Sun Dec 4 00:54:59 2011
Errors in file /opt/app/oracle/admin/xxxxx/bdump/xxxxx1_j000_29799.trc:
ORA-00481: LMON process terminated with error
Sun Dec 4 00:54:59 2011
WARNING: inbound connection timed out (ORA-3136)
Sun Dec 4 00:54:59 2011
Errors in file /opt/app/oracle/admin/xxxxx/bdump/xxxxx1_j003_15093.trc:
ORA-00481: LMON process terminated with error
Sun Dec 4 00:54:59 2011
Errors in file /opt/app/oracle/admin/xxxxx/bdump/xxxxx1_lck0_16841.trc:
ORA-00481: LMON process terminated with error
Sun Dec 4 00:54:59 2011
Errors in file /opt/app/oracle/admin/xxxxx/bdump/xxxxx1_pmon_16519.trc:
ORA-00481: LMON process terminated with error
Sun Dec 4 00:54:59 2011
Errors in file /opt/app/oracle/admin/xxxxx/bdump/xxxxx1_lms3_16549.trc:
ORA-00481: LMON process terminated with error
Sun Dec 4 00:54:59 2011
Errors in file /opt/app/oracle/admin/xxxxx/bdump/xxxxx1_psp0_16523.trc:
ORA-00481: LMON process terminated with error
Sun Dec 4 00:55:00 2011
Errors in file /opt/app/oracle/admin/xxxxx/bdump/xxxxx1_lms5_16557.trc:
ORA-00481: LMON process terminated with error
Sun Dec 4 00:55:00 2011
Errors in file /opt/app/oracle/admin/xxxxx/bdump/xxxxx1_lms4_16553.trc:
ORA-00481: LMON process terminated with error
Sun Dec 4 00:55:01 2011
Errors in file /opt/app/oracle/admin/xxxxx/bdump/xxxxx1_lmd0_16527.trc:
ORA-00481: LMON process terminated with error
Sun Dec 4 00:55:01 2011
Errors in file /opt/app/oracle/admin/xxxxx/bdump/xxxxx1_lms0_16537.trc:
ORA-00481: LMON process terminated with error
Sun Dec 4 00:55:01 2011
Errors in file /opt/app/oracle/admin/xxxxx/bdump/xxxxx1_lms1_16541.trc:
ORA-00481: LMON process terminated with error
Sun Dec 4 00:55:01 2011
RECO: terminating instance due to error 481
Sun Dec 4 00:55:07 2011
Instance terminated by RECO, pid = 16588
Sun Dec 4 00:55:18 2011
Starting ORACLE instance (normal)
LICENSE_MAX_SESSION = 0
LICENSE_SESSIONS_WARNING = 0
Interface type 1 bge2 10.0.2.0 configured from OCR for use as a cluster interconnect
Interface type 1 bge0 202.76.5.0 configured from OCR for use as a public interface
Picked latch-free SCN scheme 3
Autotune of undo retention is turned on.
LICENSE_MAX_USERS = 0
SYS auditing is disabled
ksdpec: called for event 13740 prior to event group initialization
Starting up ORACLE RDBMS Version: 10.2.0.4.0.
System parameters with non-default values:
processes = 300
sessions = 500
sga_max_size = 5368709120
__shared_pool_size = 1879048192
shared_pool_size = 1073741824
__large_pool_size = 16777216
large_pool_size = 16777216
__java_pool_size = 134217728
java_pool_size = 134217728
__streams_pool_size = 0
shared_pool_reserved_size= 142606336
spfile = +DATA/xxxxx/spfilexxxxx.ora
sga_target = 5368709120
control_files = +DATA/xxxxx/controlfile/current.258.702223445
db_block_size = 8192
__db_cache_size = 3288334336
db_cache_size = 3221225472
compatible = 10.2.0.1.0
log_archive_dest = +REDOARCH/xxxxx/archlog
log_buffer = 30543360
db_file_multiblock_read_count= 16
cluster_database = TRUE
cluster_database_instances= 2
db_create_file_dest = +DATA
db_create_online_log_dest_1= +REDOARCH
db_recovery_file_dest_size= 2147483648
thread = 1
instance_number = 1
undo_management = AUTO
undo_tablespace = UNDOTBS1
undo_retention = 14400
remote_login_passwordfile= EXCLUSIVE
db_domain =
dispatchers = (PROTOCOL=TCP) (SERVICE=xxxxxXDB)
local_listener = LISTENER_xxxxx1
remote_listener = LISTENERS_xxxxx
session_cached_cursors = 100
job_queue_processes = 10
cursor_sharing = SIMILAR
background_dump_dest = /opt/app/oracle/admin/xxxxx/bdump
user_dump_dest = /opt/app/oracle/admin/xxxxx/udump
core_dump_dest = /opt/app/oracle/admin/xxxxx/cdump
audit_file_dest = /opt/app/oracle/admin/xxxxx/adump
db_name = xxxxx
open_cursors = 500
pga_aggregate_target = 1073741824
Cluster communication is configured to use the following interface(s) for this instance
10.0.2.235
Sun Dec 4 00:55:25 2011
cluster interconnect IPC version:Oracle UDP/IP (generic)
IPC Vendor 1 proto 2
PMON started with pid=2, OS id=19662
DIAG started with pid=3, OS id=19664
PSP0 started with pid=4, OS id=19666
LMON started with pid=5, OS id=19668
LMD0 started with pid=6, OS id=19670
LMS0 started with pid=7, OS id=19672
LMS1 started with pid=8, OS id=19676
LMS2 started with pid=9, OS id=19688
LMS3 started with pid=10, OS id=19692
LMS4 started with pid=11, OS id=19696
LMS5 started with pid=12, OS id=19700
MMAN started with pid=13, OS id=19704
DBW0 started with pid=14, OS id=19706
DBW1 started with pid=15, OS id=19708
DBW2 started with pid=16, OS id=19718
LGWR started with pid=17, OS id=19720
CKPT started with pid=18, OS id=19722
SMON started with pid=19, OS id=19724
RECO started with pid=20, OS id=19726
CJQ0 started with pid=21, OS id=19728
MMON started with pid=22, OS id=19730
Sun Dec 4 00:55:28 2011
starting up 1 dispatcher(s) for network address '(ADDRESS=(PARTIAL=YES)(PROTOCOL=TCP))'...
MMNL started with pid=23, OS id=19732
Sun Dec 4 00:55:28 2011
starting up 1 shared server(s) ...
Sun Dec 4 00:55:28 2011
DISM started, OS id=19746
Sun Dec 4 00:55:52 2011
lmon registered with NM - instance id 1 (internal mem no 0)
Sun Dec 4 00:55:53 2011
Reconfiguration started (old inc 0, new inc 78)
List of nodes:
0 1
Global Resource Directory frozen
* allocate domain 0, invalid = TRUE
Communication channels reestablished
* domain 0 valid according to instance 1
* domain 0 valid = 1 according to instance 1
Sun Dec 4 00:55:53 2011
Master broadcasted resource hash value bitmaps
Non-local Process blocks cleaned out
Sun Dec 4 00:55:53 2011
LMS 0: 0 GCS shadows cancelled, 0 closed
Sun Dec 4 00:55:53 2011
LMS 4: 0 GCS shadows cancelled, 0 closed
Sun Dec 4 00:55:53 2011
LMS 3: 0 GCS shadows cancelled, 0 closed
Sun Dec 4 00:55:53 2011
LMS 1: 0 GCS shadows cancelled, 0 closed
Sun Dec 4 00:55:53 2011
LMS 5: 0 GCS shadows cancelled, 0 closed
Sun Dec 4 00:55:53 2011
LMS 2: 0 GCS shadows cancelled, 0 closed
Set master node info
Submitted all remote-enqueue requests
Dwn-cvts replayed, VALBLKs dubious
All grantable enqueues granted
Sun Dec 4 00:55:53 2011
LMS 5: 0 GCS shadows traversed, 0 replayed
Sun Dec 4 00:55:53 2011
LMS 3: 0 GCS shadows traversed, 0 replayed
Sun Dec 4 00:55:53 2011
LMS 2: 0 GCS shadows traversed, 0 replayed
Sun Dec 4 00:55:53 2011
LMS 0: 0 GCS shadows traversed, 0 replayed
Sun Dec 4 00:55:53 2011
LMS 4: 0 GCS shadows traversed, 0 replayed
Sun Dec 4 00:55:53 2011
LMS 1: 0 GCS shadows traversed, 0 replayed
Sun Dec 4 00:55:53 2011
Submitted all GCS remote-cache requests
Post SMON to start 1st pass IR
Fix write in gcs resources
Reconfiguration complete
LCK0 started with pid=26, OS id=19984
Sun Dec 4 00:55:55 2011
Oracle Data Guard is not available in this edition of Oracle.
Sun Dec 4 00:55:55 2011
ALTER DATABASE MOUNT
Sun Dec 4 00:55:56 2011
Starting background process ASMB
ASMB started with pid=28, OS id=19995
Starting background process RBAL
RBAL started with pid=29, OS id=19999
Sun Dec 4 00:55:59 2011
SUCCESS: diskgroup DATA was mounted
Sun Dec 4 00:56:03 2011
Setting recovery target incarnation to 7
Sun Dec 4 00:56:04 2011
Successful mount of redo thread 1, with mount id 3288132717
Sun Dec 4 00:56:04 2011
Database mounted in Shared Mode (CLUSTER_DATABASE=TRUE)
Completed: ALTER DATABASE MOUNT
Sun Dec 4 00:56:05 2011
ALTER DATABASE OPEN
Picked broadcast on commit scheme to generate SCNs
Sun Dec 4 00:56:06 2011
LGWR: STARTING ARCH PROCESSES
ARC0 started with pid=33, OS id=20135
Sun Dec 4 00:56:06 2011
ARC0: Archival started
ARC1: Archival started
LGWR: STARTING ARCH PROCESSES COMPLETE
ARC1 started with pid=34, OS id=20137
Sun Dec 4 00:56:06 2011
SUCCESS: diskgroup REDOARCH was mounted
Thread 1 opened at log sequence 23830
Current log# 3 seq# 23830 mem# 0: +REDOARCH/xxxxx/onlinelog/group_3.261.709928305
Successful open of redo thread 1
Sun Dec 4 00:56:07 2011
SMON: enabling cache recovery
Sun Dec 4 00:56:07 2011
ARC0: Becoming the 'no FAL' ARCH
ARC0: Becoming the 'no SRL' ARCH
Sun Dec 4 00:56:07 2011
ARC1: Becoming the heartbeat ARCH
Sun Dec 4 00:56:09 2011
Successfully onlined Undo Tablespace 1.
Sun Dec 4 00:56:09 2011
SMON: enabling tx recovery
Sun Dec 4 00:56:09 2011
Database Characterset is UTF8
replication_dependency_tracking turned off (no async multimaster replication found)
Starting background process QMNC
QMNC started with pid=35, OS id=20185
Sun Dec 4 00:56:15 2011
Errors in file /opt/app/oracle/admin/xxxxx/udump/xxxxx1_ora_20121.trc:
ORA-00604: error occurred at recursive SQL level 1
ORA-12663: Services required by client not available on the server
ORA-36961: Oracle OLAP is not available.
ORA-06512: at "SYS.OLAPIHISTORYRETENTION", line 1
ORA-06512: at line 15
Sun Dec 4 00:56:15 2011
Completed: ALTER DATABASE OPEN