Hi expert ,
oracle 11g.2 ASM RAC under OL 5.6
tonight the network team have an operation with the network but today i see more email from monitoring system everything is done
2 database that we have and application server is down also for 20 minute and then it's go back and became UP
and the below i found it in alert log files
Wed Feb 20 02:40:51 2013
SKGXP: ospid 27658: network interface with IP address 169.254.210.63 no longer running (check cable)
Wed Feb 20 02:41:10 2013
NOTE: ASMB terminating
Errors in file /home/app/oracle/diag/rdbms/absnl/ABSNL2/trace/ABSNL2_asmb_27692.trc:
ORA-15064: communication failure with ASM instance
ORA-03113: end-of-file on communication channel
Process ID:
Session ID: 50 Serial number: 5
Errors in file /home/app/oracle/diag/rdbms/absnl/ABSNL2/trace/ABSNL2_asmb_27692.trc:
ORA-15064: communication failure with ASM instance
ORA-03113: end-of-file on communication channel
Process ID:
Session ID: 50 Serial number: 5
ASMB (ospid: 27692): terminating the instance due to error 15064
Instance terminated by ASMB, pid = 27692
Wed Feb 20 02:47:58 2013
Starting ORACLE instance (normal)
LICENSE_MAX_SESSION = 0
LICENSE_SESSIONS_WARNING = 0
Private Interface 'eth1:1' configured from GPnP for use as a private interconnect.
[name='eth1:1', type=1, ip=169.254.210.63, mac=1c-c1-de-26-2c-7e, net=169.254.0.0/16, mask=255.255.0.0, use=haip:cluster_interconnect/62]
Public Interface 'eth0' configured from GPnP for use as a public interface.
[name='eth0', type=1, ip=10.200.200.111, mac=1c-c1-de-26-2c-7c, net=10.200.200.0/24, mask=255.255.255.0, use=public/1]
Public Interface 'eth0:1' configured from GPnP for use as a public interface.
[name='eth0:1', type=1, ip=10.200.200.116, mac=1c-c1-de-26-2c-7c, net=10.200.200.0/24, mask=255.255.255.0, use=public/1]
Picked latch-free SCN scheme 3
and other database i found the below :
Wed Feb 20 02:47:41 2013
ALTER DATABASE MOUNT /* db agent *//* {2:2554:2} */
NOTE: Loaded library: System
SUCCESS: diskgroup DATA was mounted
NOTE: dependency between database MEDI and diskgroup resource ora.DATA.dg is established
Successful mount of redo thread 2, with mount id 3613558669
Database mounted in Shared Mode (CLUSTER_DATABASE=TRUE)
Lost write protection disabled
Completed: ALTER DATABASE MOUNT /* db agent *//* {2:2554:2} */
ALTER DATABASE OPEN /* db agent *//* {2:2554:2} */
Picked broadcast on commit scheme to generate SCNs
ARCH: STARTING ARCH PROCESSES
Wed Feb 20 02:47:47 2013
ARC0 started with pid=38, OS id=9898
ARC0: Archival started
ARCH: STARTING ARCH PROCESSES COMPLETE
ARC0: STARTING ARCH PROCESSES
Wed Feb 20 02:47:48 2013
ARC1 started with pid=39, OS id=9902
Wed Feb 20 02:47:48 2013
ARC2 started with pid=40, OS id=9904
ARC1: Archival started
ARC2: Archival started
Wed Feb 20 02:47:48 2013
ARC3 started with pid=41, OS id=9906
ARC1: Becoming the 'no FAL' ARCH
ARC1: Becoming the 'no SRL' ARCH
ARC2: Becoming the heartbeat ARCH
Thread 2 opened at log sequence 11315
Current log# 4 seq# 11315 mem# 0: +DATA/medi/onlinelog/group_4.266.763062925
Successful open of redo thread 2
SMON: enabling cache recovery
[9770] Successfully onlined Undo Tablespace 4.
Undo initialization finished serial:0 start:4294406380 end:4294406830 diff:450 (4 seconds)
Verifying file header compatibility for 11g tablespace encryption..
Verifying 11g file header compatibility for tablespace encryption completed
Wed Feb 20 02:47:49 2013
SMON: enabling tx recovery
Database Characterset is WE8MSWIN1252
Errors in file /home/app/oracle/diag/rdbms/medi/MEDI2/trace/MEDI2_smon_9734.trc (incident=26585):
ORA-00600: internal error code, arguments: [ktsxtffs2], [3], [1398030667], [], [], [], [], [], [], [], [], []
Incident details in: /home/app/oracle/diag/rdbms/medi/MEDI2/incident/incdir_26585/MEDI2_smon_9734_i26585.trc
No Resource Manager plan active
Starting background process GTX0
Wed Feb 20 02:47:50 2013
GTX0 started with pid=42, OS id=9912
replication_dependency_tracking turned off (no async multimaster replication found)
ARC3: Archival started
ARC0: STARTING ARCH PROCESSES COMPLETE
Starting background process QMNC
Wed Feb 20 02:47:50 2013
QMNC started with pid=43, OS id=9917
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Errors in file /home/app/oracle/diag/rdbms/medi/MEDI2/trace/MEDI2_smon_9734.trc (incident=26586):
ORA-00600: internal error code, arguments: [kghstack_free1], [sort - search in file #s], [], [], [], [], [], [], [], [], [], []
ORA-00600: internal error code, arguments: [ktsxtffs2], [3], [1398030667], [], [], [], [], [], [], [], [], []
Incident details in: /home/app/oracle/diag/rdbms/medi/MEDI2/incident/incdir_26586/MEDI2_smon_9734_i26586.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Wed Feb 20 02:47:51 2013
Dumping diagnostic data in directory=[cdmp_20130220024751], requested by (instance=2, osid=9734 (SMON)), summary=[incident=26585].
Starting background process SMCO
Wed Feb 20 02:47:51 2013
SMCO started with pid=44, OS id=9924
Dumping diagnostic data in directory=[cdmp_20130220024751], requested by (instance=2, osid=9734 (SMON)), summary=[incident=26586].
Wed Feb 20 02:47:51 2013
Completed: ALTER DATABASE OPEN /* db agent *//* {2:2554:2} */
Wed Feb 20 02:47:52 2013
ALTER SYSTEM SET local_listener='(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=10.200.200.131)(PORT=1521))))' SCOPE=MEMORY SID='MEDI2';
Wed Feb 20 02:47:53 2013
minact-scn: Inst 2 is a slave inc#:8 mmon proc-id:9742 status:0x2
minact-scn status: grec-scn:0x0000.00000000 gmin-scn:0x0000.00000000 gcalc-scn:0x0000.00000000
Wed Feb 20 02:47:53 2013
Starting background process CJQ0
Wed Feb 20 02:47:53 2013
CJQ0 started with pid=47, OS id=9960
Wed Feb 20 02:48:42 2013
Sweep [inc][26586]: completed
Sweep [inc][26585]: completed
Sweep [inc2][26586]: completed
Sweep [inc2][26585]: completed
[oracle@medi02 trace]$
please i need your advice what i do in this situation (stop listener,...) or any idea that we found it is helpful
for DBA.