WARNING: ASM communication error: op 0 state 0x0
Hi,
Oracle 11.2.0.3 Enterprise edition
Windows 2008R2 64bit
I'm trying to troubleshoot a random problem that I get in my Oracle RAC (see alert log snippet below).
Wed Aug 29 12:04:09 2012
ORA-25253 encountered when generating server alert SMG-3503
Wed Aug 29 12:05:10 2012
ORA-25253 encountered when generating server alert SMG-3503
Wed Aug 29 12:09:38 2012
WARNING: ASM communication error: op 0 state 0x0 (15055)
ERROR: direct connection failure with ASM
WARNING: ASM communication error: op 0 state 0x0 (15055)
ERROR: direct connection failure with ASM
WARNING: ASM communication error: op 0 state 0x0 (15055)
ERROR: direct connection failure with ASM
WARNING: ASM communication error: op 0 state 0x0 (15055)
ERROR: direct connection failure with ASM
WARNING: ASM communication error: op 0 state 0x0 (15055)
ERROR: direct connection failure with ASM
Wed Aug 29 12:12:22 2012
***********************************************************************
Fatal NI connect error 12537, connecting to:
(LOCAL=NO)
VERSION INFORMATION:
TNS for 64-bit Windows: Version 11.2.0.3.0 - Production
Oracle Bequeath NT Protocol Adapter for 64-bit Windows: Version 11.2.0.3.0 - Production
Windows NT TCP/IP NT Protocol Adapter for 64-bit Windows: Version 11.2.0.3.0 - Production
Time: 29-AUG-2012 12:12:22
Tracing not turned on.
Tns error struct:
ns main err code: 12537
TNS-12537: TNS:connection closed
ns secondary err code: 12560
nt main err code: 0
nt secondary err code: 0
nt OS err code: 0
opiodr aborting process unknown ospid (504) as a result of ORA-609
Wed Aug 29 12:12:22 2012
***********************************************************************
Fatal NI connect error 12537, connecting to:
(LOCAL=NO)
VERSION INFORMATION:
TNS for 64-bit Windows: Version 11.2.0.3.0 - Production
Oracle Bequeath NT Protocol Adapter for 64-bit Windows: Version 11.2.0.3.0 - Production
Windows NT TCP/IP NT Protocol Adapter for 64-bit Windows: Version 11.2.0.3.0 - Production
Time: 29-AUG-2012 12:12:22
Tracing not turned on.
Tns error struct:
ns main err code: 12537
TNS-12537: TNS:connection closed
ns secondary err code: 12560
nt main err code: 0
nt secondary err code: 0
nt OS err code: 0
opiodr aborting process unknown ospid (9856) as a result of ORA-609
Wed Aug 29 12:12:22 2012
***********************************************************************
***********************************************************************
Fatal NI connect error 12537, connecting to:
(LOCAL=NO)
VERSION INFORMATION:
TNS for 64-bit Windows: Version 11.2.0.3.0 - Production
Oracle Bequeath NT Protocol Adapter for 64-bit Windows: Version 11.2.0.3.0 - Production
Windows NT TCP/IP NT Protocol Adapter for 64-bit Windows: Version 11.2.0.3.0 - Production
Time: 29-AUG-2012 12:12:22
Tracing not turned on.
Tns error struct:
ns main err code: 12537
TNS-12537: TNS:connection closed
ns secondary err code: 12560
nt main err code: 0
nt secondary err code: 0
nt OS err code: 0
opiodr aborting process unknown ospid (9388) as a result of ORA-609
Wed Aug 29 12:12:22 2012
***********************************************************************
Fatal NI connect error 12537, connecting to:
(LOCAL=NO)
VERSION INFORMATION:
TNS for 64-bit Windows: Version 11.2.0.3.0 - Production
Oracle Bequeath NT Protocol Adapter for 64-bit Windows: Version 11.2.0.3.0 - Production
Windows NT TCP/IP NT Protocol Adapter for 64-bit Windows: Version 11.2.0.3.0 - Production
Time: 29-AUG-2012 12:12:22
Tracing not turned on.
Tns error struct:
ns main err code: 12537
TNS-12537: TNS:connection closed
ns secondary err code: 12560
nt main err code: 0
nt secondary err code: 0
nt OS err code: 0
opiodr aborting process unknown ospid (7320) as a result of ORA-609
Wed Aug 29 12:12:22 2012
***********************************************************************
Fatal NI connect error 12537, connecting to:
(LOCAL=NO)
VERSION INFORMATION:
TNS for 64-bit Windows: Version 11.2.0.3.0 - Production
Oracle Bequeath NT Protocol Adapter for 64-bit Windows: Version 11.2.0.3.0 - Production
Windows NT TCP/IP NT Protocol Adapter for 64-bit Windows: Version 11.2.0.3.0 - Production
Time: 29-AUG-2012 12:12:22
Tracing not turned on.
Tns error struct:
ns main err code: 12537
TNS-12537: TNS:connection closed
ns secondary err code: 12560
nt main err code: 0
nt secondary err code: 0
nt OS err code: 0
opiodr aborting process unknown ospid (9360) as a result of ORA-609
Wed Aug 29 12:12:22 2012
***********************************************************************
Fatal NI connect error 12537, connecting to:
(LOCAL=NO)
VERSION INFORMATION:
TNS for 64-bit Windows: Version 11.2.0.3.0 - Production
Oracle Bequeath NT Protocol Adapter for 64-bit Windows: Version 11.2.0.3.0 - Production
Windows NT TCP/IP NT Protocol Adapter for 64-bit Windows: Version 11.2.0.3.0 - Production
Time: 29-AUG-2012 12:12:22
Tracing not turned on.
Tns error struct:
ns main err code: 12537
TNS-12537: TNS:connection closed
ns secondary err code: 12560
nt main err code: 0
nt secondary err code: 0
nt OS err code: 0
opiodr aborting process unknown ospid (8496) as a result of ORA-609
Wed Aug 29 12:12:22 2012
***********************************************************************
Fatal NI connect error 12537, connecting to:
(LOCAL=NO)
VERSION INFORMATION:
TNS for 64-bit Windows: Version 11.2.0.3.0 - Production
Oracle Bequeath NT Protocol Adapter for 64-bit Windows: Version 11.2.0.3.0 - Production
Windows NT TCP/IP NT Protocol Adapter for 64-bit Windows: Version 11.2.0.3.0 - Production
Time: 29-AUG-2012 12:12:22
Tracing not turned on.
Tns error struct:
ns main err code: 12537
TNS-12537: TNS:connection closed
ns secondary err code: 12560
nt main err code: 0
nt secondary err code: 0
nt OS err code: 0
opiodr aborting process unknown ospid (4732) as a result of ORA-609
Wed Aug 29 12:12:22 2012
***********************************************************************
Fatal NI connect error 12537, connecting to:
(LOCAL=NO)
VERSION INFORMATION:
TNS for 64-bit Windows: Version 11.2.0.3.0 - Production
Oracle Bequeath NT Protocol Adapter for 64-bit Windows: Version 11.2.0.3.0 - Production
Windows NT TCP/IP NT Protocol Adapter for 64-bit Windows: Version 11.2.0.3.0 - Production
Time: 29-AUG-2012 12:12:22
Tracing not turned on.
Tns error struct:
ns main err code: 12537
TNS-12537: TNS:connection closed
ns secondary err code: 12560
nt main err code: 0
nt secondary err code: 0
nt OS err code: 0
opiodr aborting process unknown ospid (8936) as a result of ORA-609
Wed Aug 29 12:12:22 2012
***********************************************************************
Fatal NI connect error 12537, connecting to:
(LOCAL=NO)
VERSION INFORMATION:
TNS for 64-bit Windows: Version 11.2.0.3.0 - Production
Oracle Bequeath NT Protocol Adapter for 64-bit Windows: Version 11.2.0.3.0 - Production
Windows NT TCP/IP NT Protocol Adapter for 64-bit Windows: Version 11.2.0.3.0 - Production
Time: 29-AUG-2012 12:12:22
Tracing not turned on.
Tns error struct:
ns main err code: 12537
TNS-12537: TNS:connection closed
ns secondary err code: 12560
nt main err code: 0
nt secondary err code: 0
nt OS err code: 0
opiodr aborting process unknown ospid (9316) as a result of ORA-609
Wed Aug 29 12:12:23 2012
***********************************************************************
Fatal NI connect error 12537, connecting to:
(LOCAL=NO)
VERSION INFORMATION:
TNS for 64-bit Windows: Version 11.2.0.3.0 - Production
Oracle Bequeath NT Protocol Adapter for 64-bit Windows: Version 11.2.0.3.0 - Production
Windows NT TCP/IP NT Protocol Adapter for 64-bit Windows: Version 11.2.0.3.0 - Production
Time: 29-AUG-2012 12:12:23
Tracing not turned on.
Tns error struct:
ns main err code: 12537
TNS-12537: TNS:connection closed
ns secondary err code: 12560
nt main err code: 0
nt secondary err code: 0
nt OS err code: 0
opiodr aborting process unknown ospid (9768) as a result of ORA-609
Wed Aug 29 12:12:23 2012
***********************************************************************
Fatal NI connect error 12537, connecting to:
(LOCAL=NO)
VERSION INFORMATION:
TNS for 64-bit Windows: Version 11.2.0.3.0 - Production
Oracle Bequeath NT Protocol Adapter for 64-bit Windows: Version 11.2.0.3.0 - Production
Windows NT TCP/IP NT Protocol Adapter for 64-bit Windows: Version 11.2.0.3.0 - Production
Time: 29-AUG-2012 12:12:23
Tracing not turned on.
Tns error struct:
ns main err code: 12537
TNS-12537: TNS:connection closed
ns secondary err code: 12560
nt main err code: 0
nt secondary err code: 0
nt OS err code: 0
opiodr aborting process unknown ospid (5848) as a result of ORA-609
Wed Aug 29 12:24:41 2012
WARNING: ASM communication error: op 0 state 0x0 (15055)
ERROR: direct connection failure with ASM
WARNING: ASM communication error: op 0 state 0x0 (15055)
ERROR: direct connection failure with ASM
WARNING: ASM communication error: op 0 state 0x0 (15055)
ERROR: direct connection failure with ASM
WARNING: ASM communication error: op 0 state 0x0 (15055)
ERROR: direct connection failure with ASM
WARNING: ASM communication error: op 0 state 0x0 (15055)
ERROR: direct connection failure with ASM
Wed Aug 29 12:31:42 2012
Thread 2 advanced to log sequence 2813 (LGWR switch)
Current log# 3 seq# 2813 mem# 0: +REDO/qcare00/onlinelog/group_3.259.780247037
Wed Aug 29 12:31:42 2012
Archived Log entry 6684 added for thread 2 sequence 2812 ID 0x6cba185b dest 1:
Wed Aug 29 12:32:18 2012
ORA-25253 encountered when generating server alert SMG-3503
Wed Aug 29 12:35:18 2012
ORA-25253 encountered when generating server alert SMG-3503
Wed Aug 29 12:38:15 2012
***********************************************************************
Fatal NI connect error 12638, connecting to:
(DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))
VERSION INFORMATION:
TNS for 64-bit Windows: Version 11.2.0.3.0 - Production
Oracle Bequeath NT Protocol Adapter for 64-bit Windows: Version 11.2.0.3.0 - Production
Time: 29-AUG-2012 12:38:15
Tracing not turned on.
Tns error struct:
ns main err code: 12638
TNS-12638: Credential retrieval failed
ns secondary err code: 0
nt main err code: 0
nt secondary err code: 0
nt OS err code: 0
When we get the error this is the status of the cluster:
Microsoft Windows [Version 6.1.7601]
Copyright (c) 2009 Microsoft Corporation. All rights reserved.
C:\Windows\system32>crsctl stat res -t
--------------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.dg
ONLINE ONLINE s0060
ONLINE ONLINE s0061
ora.FRA.dg
ONLINE ONLINE s0060
ONLINE ONLINE s0061
ora.GRID.dg
ONLINE ONLINE s0060
ONLINE ONLINE s0061
ora.LISTENER.lsnr
ONLINE ONLINE s0060
ONLINE INTERMEDIATE s0061 CHECK TIMED OUT
ora.REDO.dg
ONLINE ONLINE s0060
ONLINE ONLINE s0061
ora.asm
ONLINE ONLINE s0060 Started
ONLINE ONLINE s0061 Started
ora.gsd
OFFLINE OFFLINE s0060
OFFLINE OFFLINE s0061
ora.net1.network
ONLINE ONLINE s0060
ONLINE ONLINE s0061
ora.ons
ONLINE ONLINE s0060
ONLINE ONLINE s0061
ora.registry.acfs
ONLINE ONLINE s0060
ONLINE ONLINE s0061
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE s0060
ora.LISTENER_SCAN2.lsnr
1 ONLINE INTERMEDIATE s0061 CHECK TIMED OUT
ora.cvu
1 ONLINE ONLINE s0061
ora.s0060.vip
1 ONLINE ONLINE s0060
ora.s0061.vip
1 ONLINE ONLINE s0061
ora.oc4j
1 ONLINE OFFLINE
ora.qc00.db
1 ONLINE ONLINE s0060 Open
2 ONLINE ONLINE s0061 Open
ora.qc00.qsc00.svc
2 ONLINE ONLINE s0060
ora.qc01.db
1 ONLINE ONLINE s0060 Open
2 ONLINE ONLINE s0061 Open
ora.qc01.qsc01.svc
2 ONLINE ONLINE s0061
ora.scan1.vip
1 ONLINE ONLINE s0060
ora.scan2.vip
1 ONLINE ONLINE s0061
As soon as we reboot the node with problems the database is working again.
Thanks in advace.
Edited by: user12996279 on 29-ago-2012 7:48