Skip to Main Content

Listener down and crs_stat -t hangs

John-MKJun 8 2012 — edited Nov 6 2013
Hi,

2 Node RAC Cluster. DB=10.2.0.4, Clusterware 11g, OS=SLES.

Listener went down on Node 2 (dbb02px), and crs_stat -t hangs for approx. 10 minutes. I have to start the node 2 listener from node 1. The problem occured second time in 2 months time. Please help as I am not able to understand the problem.
grid@dbb01px [ec]: /home/grid > srvctl start listener -n dbb02px
PRCC-1015 : LISTENER was already running on dbb02px

grid@dbb01px [ec]: /home/grid > srvctl stop listener -n dbb02px
grid@dbb01px [ec]: /home/grid > srvctl start listener -n dbb02px
Till now I only found the following.
# Cluster alertlog: /oracle/app/11.2.0/grid/log/dbb02px/alertdbb02px.log
2012-06-06 06:40:30.460
[crsd(17805)]CRS-5825:Agent '/oracle/app/11.2.0/grid/bin/oraagent_oracle' is unresponsive and will be restarted. Details at (:CRSAGF00131:) in /oracle/app/11.2.0/grid/log/dbb02px/crsd/crsd.log.
2012-06-06 06:43:03.334
[crsd(17805)]CRS-5825:Agent '/oracle/app/11.2.0/grid/bin/oraagent_grid' is unresponsive and will be restarted. Details at (:CRSAGF00131:) in /oracle/app/11.2.0/grid/log/dbb02px/crsd/crsd.log.
## /oracle/app/11.2.0/grid/log/dbb02px/crsd/crsd.log

2012-06-06 06:37:02.897: [ CRSCOMM][2961041152][FFAIL] Couldnt clscreceive message, no message: 11
2012-06-06 06:37:02.897: [ CRSCOMM][2961041152] Client disconnected.
2012-06-06 06:37:02.897: [ CRSCOMM][2961041152][FFAIL] Listener got clsc error 11 for memNum. 5
2012-06-06 06:37:02.923: [ CRSCOMM][2961041152] IPC listener connection to member 5 has been removed
2012-06-06 06:37:02.923: [CLSFRAME][2961041152] Removing IPC Member:{Relative|Node:0|Process:5|Type:3}
2012-06-06 06:37:02.934: [CLSFRAME][2961041152] Disconnected from AGENT process: {Relative|Node:0|Process:5|Type:3}
2012-06-06 06:37:02.998: [    AGFW][2954737408] Agfw Proxy Server received process disconnected notification, count=1
2012-06-06 06:37:02.998: [    AGFW][2954737408] /oracle/app/11.2.0/grid/bin/oraagent_grid disconnected.
2012-06-06 06:37:02.998: [    AGFW][2954737408] Restarting the agent /oracle/app/11.2.0/grid/bin/oraagent_grid
2012-06-06 06:37:02.998: [    AGFW][2954737408] Agent /oracle/app/11.2.0/grid/bin/oraagent_grid[27724] stopped!
2012-06-06 06:37:02.998: [ CRSCOMM][2954737408] removeConnection: Member 5 does not exist.
2012-06-06 06:37:02.998: [    AGFW][2954737408] Starting the agent: /oracle/app/11.2.0/grid/bin/oraagent with user id: grid and incarnation:4
2012-06-06 06:37:03.019: [    AGFW][2954737408] Starting the HB [Interval =  30000, misscount = 6kill allowed=1] for agent: /oracle/app/11.2.0/grid/bin/oraagent_grid
2012-06-06 06:37:03.269: [CLSFRAME][2961041152] New IPC Member:{Relative|Node:0|Process:6|Type:3}:AGENT
2012-06-06 06:37:03.269: [CLSFRAME][2961041152] New process connected to us ID:{Relative|Node:0|Process:6|Type:3} Info:AGENT
2012-06-06 06:37:03.287: [    AGFW][2954737408] Agfw Proxy Server received the message: AGENT_HANDSHAKE[Proxy] ID 20484:14
2012-06-06 06:37:03.287: [    AGFW][2954737408] Agent /oracle/app/11.2.0/grid/bin/oraagent_grid with pid:6135 connected to server.
2012-06-06 06:37:03.288: [    AGFW][2954737408] Agfw Proxy Server sending message: RESTYPE_ADD[ora.asm.type] ID 8196:14416771 to the agent /oracle/app/11.2.0/grid/bin/oraagent_grid
2012-06-06 06:37:03.302: [    AGFW][2954737408] Agfw Proxy Server sending message: RESTYPE_ADD[ora.database.type] ID 8196:14416775 to the agent /oracle/app/11.2.0/grid/bin/oraagent_grid
2012-06-06 06:37:03.302: [    AGFW][2954737408] Could not find the resource type [ ora.discoveryserver.type ]
2012-06-06 06:37:03.302: [    AGFW][2954737408] Agfw Proxy Server sending message: RESTYPE_ADD[ora.diskgroup.type] ID 8196:14416777 to the agent /oracle/app/11.2.0/grid/bin/oraagent_grid
2012-06-06 06:37:03.302: [    AGFW][2954737408] Agfw Proxy Server sending message: RESTYPE_ADD[ora.eons.type] ID 8196:14416779 to the agent /oracle/app/11.2.0/grid/bin/oraagent_grid
2012-06-06 06:37:03.302: [    AGFW][2954737408] Could not find the resource type [ ora.evm.type ]
2012-06-06 06:37:03.302: [    AGFW][2954737408] Could not find the resource type [ ora.gipc.type ]
2012-06-06 06:37:03.302: [    AGFW][2954737408] Could not find the resource type [ ora.gpnp.type ]
2012-06-06 06:37:03.303: [    AGFW][2954737408] Agfw Proxy Server sending message: RESTYPE_ADD[ora.gsd.type] ID 8196:14416781 to the agent /oracle/app/11.2.0/grid/bin/oraagent_grid
2012-06-06 06:37:03.303: [    AGFW][2954737408] Agfw Proxy Server sending message: RESTYPE_ADD[ora.listener.type] ID 8196:14416783 to the agent /oracle/app/11.2.0/grid/bin/oraagent_grid
2012-06-06 06:37:03.303: [    AGFW][2954737408] Could not find the resource type [ ora.mdns.type ]
2012-06-06 06:37:03.303: [    AGFW][2954737408] Agfw Proxy Server sending message: RESTYPE_ADD[ora.ons.type] ID 8196:14416785 to the agent /oracle/app/11.2.0/grid/bin/oraagent_grid
2012-06-06 06:37:03.304: [    AGFW][2954737408] Agfw Proxy Server sending message: RESTYPE_ADD[ora.scan_listener.type] ID 8196:14416787 to the agent /oracle/app/11.2.0/grid/bin/oraagent_grid
2012-06-06 06:37:03.304: [    AGFW][2954737408] Could not find the resource type [ ora.service.type ]
2012-06-06 06:37:03.304: [    AGFW][2954737408] Could not find the resource type [ ora.supercluster.type ]
2012-06-06 06:37:03.309: [    AGFW][2954737408] Agfw Proxy Server sending message: RESOURCE_ADD[ora.DATA1.dg dbb02px 1] ID 4356:14416789 to the agent /oracle/app/11.2.0/grid/bin/oraagent_grid
2012-06-06 06:37:03.310: [    AGFW][2954737408] Agfw Proxy Server sending message: RESOURCE_ADD[ora.LISTENER.lsnr dbb02px 1] ID 4356:14416793 to the agent /oracle/app/11.2.0/grid/bin/oraagent_grid
2012-06-06 06:37:03.310: [    AGFW][2954737408] Agfw Proxy Server sending message: RESOURCE_ADD[ora.REDO1.dg dbb02px 1] ID 4356:14416795 to the agent /oracle/app/11.2.0/grid/bin/oraagent_grid
2012-06-06 06:37:03.310: [    AGFW][2954737408] Agfw Proxy Server sending message: RESOURCE_ADD[ora.asm dbb02px 1] ID 4356:14416797 to the agent /oracle/app/11.2.0/grid/bin/oraagent_grid
2012-06-06 06:37:03.310: [    AGFW][2954737408] Agfw Proxy Server sending message: RESOURCE_ADD[ora.eons dbb02px 1] ID 4356:14416799 to the agent /oracle/app/11.2.0/grid/bin/oraagent_grid
2012-06-06 06:37:03.310: [    AGFW][2954737408] Agfw Proxy Server sending message: RESOURCE_ADD[ora.gsd dbb02px 1] ID 4356:14416801 to the agent /oracle/app/11.2.0/grid/bin/oraagent_grid
2012-06-06 06:37:03.311: [    AGFW][2954737408] Agfw Proxy Server sending message: RESOURCE_ADD[ora.ons dbb02px 1] ID 4356:14416803 to the agent /oracle/app/11.2.0/grid/bin/oraagent_grid
2012-06-06 06:37:03.311: [    AGFW][2954737408] Agfw Proxy Server replying to the message: AGENT_HANDSHAKE[Proxy] ID 20484:14
/oracle/app/11.2.0/grid/log/dbb02px/agent/crsd/oraagent_grid/oraagent_grid.log

2012-06-06 07:24:18.733: [ora.LISTENER.lsnr][402650880] [check]
(:CLSN00010:)TNS-12541: TNS:no listener

2012-06-06 07:24:18.733: [ora.LISTENER.lsnr][402650880] [check]
(:CLSN00010:)  TNS-00511: No listener

2012-06-06 07:24:18.733: [ora.LISTENER.lsnr][402650880] [check]
(:CLSN00010:)TNS-12541: TNS:no listener

2012-06-06 07:24:18.733: [ora.LISTENER.lsnr][402650880] [check] return val (clsagfwret) =
1
2012-06-06 07:24:18.733: [ora.LISTENER.lsnr][402650880] [check] LsnrAgent::check }
2012-06-06 07:24:18.733: [    AGFW][402650880] check for resource: ora.LISTENER.lsnr
dbb02px 1 completed with status: OFFLINE
2012-06-06 07:24:18.734: [    AGFW][528475904] ora.LISTENER.lsnr dbb02px 1 state changed
from: ONLINE to: OFFLINE
2012-06-06 07:24:18.734: [    AGFW][528475904] Agent sending last reply for:
RESOURCE_STOP[ora.LISTENER.lsnr dbb02px 1] ID 4099:14418276
2012-06-06 07:24:19.385: [ USRTHRD][369080064] event type is CRS_RESOURCE_STATE_CHANGE
2012-06-06 07:24:19.385: [ USRTHRD][369080064] bodylen = 657
2012-06-06 07:24:19.385: [ USRTHRD][369080064] -----------BodyBlock----------
2012-06-06 07:24:19.385: [ USRTHRD][369080064] Seg:11.2.0.1.0
2012-06-06 07:24:19.385: [ USRTHRD][369080064]  USER=grid
2012-06-06 07:24:19.385: [ USRTHRD][369080064]  REASON=USER_ACTION
2012-06-06 07:24:19.385: [ USRTHRD][369080064]  TIMESTAMP=2012-06-06 07:24:18
2012-06-06 07:24:19.385: [ USRTHRD][369080064]  NAME=ora.LISTENER.lsnr
2012-06-06 07:24:19.385: [ USRTHRD][369080064]  RESOURCE_INSTANCE_ID=ora.LISTENER.lsnr
dbb02px 1
2012-06-06 07:24:19.385: [ USRTHRD][369080064]  TYPE=ora.listener.type
2012-06-06 07:24:19.385: [ USRTHRD][369080064]  PREVIOUS_STATE=ONLINE
2012-06-06 07:24:19.385: [ USRTHRD][369080064]  CURRENT_STATE=OFFLINE
2012-06-06 07:24:19.385: [ USRTHRD][369080064]  DERIVED_STATE=OFFLINE
2012-06-06 07:24:19.385: [ USRTHRD][369080064]  STATE_DETAILS=EmPtY
2012-06-06 07:24:19.385: [ USRTHRD][369080064]  PREV_STATE_DETAILS=EmPtY
I would appreciate if somebody could help why crs_stat -t command hangs after listener went down. Yes, the two concerns why the listener goes down and why the crs_stat -t commands hangs?

Thanks a lot

Regards.

Edited by: 910385 on 08.06.2012 06:17

Edited by: 910385 on 08.06.2012 06:45
Comments
Post Details
Added on Jun 8 2012
4 comments
2,808 views