Hi,
Since we did a RAC database migration we have a listener that goes down, on one node only.
OS is oracle linux 5
oracle version 10.2.0.4
First I thought it was due to contention on the system. Than I thought it was rman backup which used 3 channels. I modified it to 1 channel. But this did not solve the issue.
The listener log files shows:
24-OCT-2014 02:45:47 * ping * 12547
TNS-12547: TNS:lost contact
TNS-12560: TNS:protocol adapter error
TNS-00517: Lost contact
Linux Error: 32: Broken pipe
24-OCT-2014 02:45:47 * ping * 0
24-OCT-2014 02:45:47 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=hostname..com)(USER=oracle))(COMMAND=services)(ARGUMENTS=64)(SERVICE=(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=10.**.**.7)(PORT=1521))))(VERSION=169870336)) * services * 0
24-OCT-2014 02:45:47 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=__jdbc__)(USER=))(SERVICE_NAME=EX..com)) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.**.**.5)(PORT=26441)) * establish * EX..com * 0
24-OCT-2014 02:45:47 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=hostname..com)(USER=oracle))(COMMAND=status)(ARGUMENTS=64)(SERVICE=LISTENER_hostname)(VERSION=169870336)) * status * 0
No longer listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=EXTPROC1)))
No longer listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=10.**.**.7)(PORT=1521)))
No longer listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=10.**.**.5)(PORT=1521)))
Listener completed notification to CRS on stop
and
20-OCT-2014 02:07:59 * ping * 12547
TNS-12547: TNS:lost contact
TNS-12560: TNS:protocol adapter error
TNS-00517: Lost contact
Linux Error: 32: Broken pipe
20-OCT-2014 02:07:59 * ping * 0
20-OCT-2014 02:07:59 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=__jdbc__)(USER=))(SERVICE_NAME=EX..com)) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.**.**.5)(PORT=25180)) * establish * EX.**.com * 0
20-OCT-2014 02:07:59 * (CONNECT_DATA=(SERVER=DEDICATED)(SERVICE_NAME=EX.**.com)(FAILOVER_MODE=(TYPE=SELECT)(METHOD=BASIC)(RETRIES=180)(DELAY=5))(CID=(PROGRAM=C:\H2)(USER=Administrator))) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.**.**.222)(PORT=51520)) * establish * EX..com * 0
20-OCT-2014 02:07:59 * (CONNECT_DATA=(SERVER=DEDICATED)(SERVICE_NAME=EX..com)(FAILOVER_MODE=(TYPE=SELECT)(METHOD=BASIC)(RETRIES=180)(DELAY=5))(CID=(PROGRAM=C:\er.exe)(HOST=-13)(USER=Administrator))) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.**.**.23)(PORT=58005)) * establish * EX..com * 0
20-OCT-2014 02:07:59 * (CONNECT_DATA=(SERVER=DEDICATED)(SERVICE_NAME=EX..com)(FAILOVER_MODE=(TYPE=SELECT)(METHOD=BASIC)(RETRIES=180)(DELAY=5))(CID=(PROGRAM=C:\er.exe)(HOST-05)(USER=Administrator))(INSTANCE_NAME=EX1)) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.**.**.15)(PORT=62958)) * establish * EX..com * 12518
TNS-12518: TNS:listener could not hand off client connection
TNS-12571: TNS:packet writer failure
TNS-12560: TNS:protocol adapter error
TNS-00530: Protocol adapter error
Linux Error: 104: Connection reset by peer
20-OCT-2014 02:07:59 * (CONNECT_DATA=(SERVER=DEDICATED)(SERVICE_NAME=EX..com)(FAILOVER_MODE=(TYPE=SELECT)(METHOD=BASIC)(RETRIES=180)(DELAY=5))(CID=(PROGRAM=C:\MServer.exe)(HOST=MEDIASTOR-02)(USER=Administrator))) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.**.**.12)(PORT=64954)) * establish * EX..com * 12518
TNS-12518: TNS:listener could not hand off client connection
TNS-12571: TNS:packet writer failure
TNS-12560: TNS:protocol adapter error
TNS-00530: Protocol adapter error
Linux Error: 104: Connection reset by peer
20-OCT-2014 02:07:59 * (CONNECT_DATA=(SERVER=DEDICATED)(SERVICE_NAME=EX..com)(FAILOVER_MODE=(TYPE=SELECT)(METHOD=BASIC)(RETRIES=180)(DELAY=5))(CID=(PROGRAM=C:\Mnge.exe)(HOST=-14)(USER=Administrator))) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.**.**.24)(PORT=49566)) * establish * EX..com * 12518
TNS-12518: TNS:listener could not hand off client connection
TNS-12571: TNS:packet writer failure
TNS-12560: TNS:protocol adapter error
TNS-00530: Protocol adapter error
My crsd.log shows:
2014-10-22 02:04:57.670: [ CRSEVT][687806784]0CAAMonitorHandler :: 0:Action Script /u01/crs/bin/racgwrap(check) timed out for ora.amcmstvdb01.vip! (timeout=60)
2014-10-22 02:04:57.671: [ CRSAPP][687806784]0CheckResource error for ora.amcmstvdb01.vip error code = -2
2014-10-24 02:19:13.940: [ CRSAPP][685705536]0CheckResource error for ora.amcmstvdb01.LISTENER_AMCMSTVDB01.lsnr error code = 1
2014-10-24 02:19:13.967: [ CRSRES][685705536]0In stateChanged, ora.amcmstvdb01.LISTENER_AMCMSTVDB01.lsnr target is ONLINE
2014-10-24 02:19:13.969: [ CRSRES][685705536]0ora.amcmstvdb01.LISTENER_AMCMSTVDB01.lsnr on amcmstvdb01 went OFFLINE unexpectedly
2014-10-24 02:19:13.973: [ CRSRES][685705536]0StopResource: setting CLI values
2014-10-24 02:19:13.981: [ CRSRES][685705536]0Attempting to stop `ora.amcmstvdb01.LISTENER_AMCMSTVDB01.lsnr` on member `amcmstvdb01`
2014-10-24 02:28:16.221: [ CRSRES][685705536]0Stop of `ora.amcmstvdb01.LISTENER_AMCMSTVDB01.lsnr` on member `amcmstvdb01` succeeded.
2014-10-24 02:28:16.223: [ CRSRES][685705536]0ora.amcmstvdb01.LISTENER_AMCMSTVDB01.lsnr RESTART_COUNT=0 RESTART_ATTEMPTS=5
2014-10-24 02:28:16.224: [ CRSRES][685705536]0Restarting ora.amcmstvdb01.LISTENER_AMCMSTVDB01.lsnr on amcmstvdb01
2014-10-24 02:28:16.227: [ CRSRES][685705536]0startRunnable: setting CLI values
2014-10-24 02:28:16.228: [ CRSRES][685705536]0Attempting to start `ora.amcmstvdb01.LISTENER_AMCMSTVDB01.lsnr` on member `amcmstvdb01`
2014-10-24 02:45:03.791: [ CRSEVT][685705536]0CAAMonitorHandler :: 0:Could not join /oracle/10.2.0/db_1/bin/racgwrap(start)
category: 1234, operation: scls_process_join, loc: childcrash, OS error: 0, other: Abnormal termination of the child
2014-10-24 02:45:03.791: [ CRSEVT][685705536]0CAAMonitorHandler :: 0:Action Script /oracle/10.2.0/db_1/bin/racgwrap(start) timed out for ora.amcmstvdb01.LISTENER_AMCMSTVDB01.lsnr! (timeout=600)
2014-10-24 02:45:03.792: [ CRSAPP][685705536]0StartResource error for ora.amcmstvdb01.LISTENER_AMCMSTVDB01.lsnr error code = -2
2014-10-24 02:45:21.220: [ CRSEVT][687806784]0CAAMonitorHandler :: 0:Could not join /oracle/10.2.0/db_1/bin/racgwrap(check)
category: 1234, operation: scls_process_join, loc: childcrash, OS error: 0, other: Abnormal termination of the child
2014-10-24 02:45:21.222: [ CRSEVT][687806784]0CAAMonitorHandler :: 0:Action Script /oracle/10.2.0/db_1/bin/racgwrap(check) timed out for ora.EX.EX1.inst! (timeout=600)
2014-10-24 02:45:21.223: [ CRSAPP][687806784]0CheckResource error for ora.EX.EX1.inst error code = -2
2014-10-24 02:45:50.080: [ CRSRES][685705536]0Start of `ora.amcmstvdb01.LISTENER_AMCMSTVDB01.lsnr` on member `amcmstvdb01` failed.
2014-10-24 02:45:50.084: [ CRSRES][685705536]0ora.amcmstvdb01.LISTENER_AMCMSTVDB01.lsnr failed on amcmstvdb01 relocating.
2014-10-24 02:45:50.103: [ CRSRES][685705536]0Cannot relocate ora.amcmstvdb01.LISTENER_AMCMSTVDB01.lsnrStopping dependents
2014-10-24 02:45:50.111: [ CRSRES][685705536]0StopResource: setting CLI values
2014-10-24 11:23:09.036: [ CRSRES][687806784]0ora.amcmstvdb01.LISTENER_AMCMSTVDB01.lsnr target set to OFFLINE before stop action
2014-10-24 11:23:09.037: [ CRSRES][687806784]0StopResource: setting CLI values
2014-10-24 11:23:09.046: [ CRSRES][687806784]0Target set to OFFLINE for `ora.amcmstvdb01.LISTENER_AMCMSTVDB01.lsnr`
2014-10-24 11:23:09.115: [ CRSRES][687806784]0ora.amcmstvdb01.ons target set to OFFLINE before stop action
2014-10-24 11:23:09.116: [ CRSRES][687806784]0StopResource: setting CLI values
2014-10-24 11:23:09.124: [ CRSRES][687806784]0Attempting to stop `ora.amcmstvdb01.ons` on member `amcmstvdb01`
2014-10-24 11:23:10.142: [ CRSRES][687806784]0Stop of `ora.amcmstvdb01.ons` on member `amcmstvdb01` succeeded.
2014-10-24 11:23:10.212: [ CRSRES][687806784]0ora.amcmstvdb01.vip target set to OFFLINE before stop action
2014-10-24 11:23:10.213: [ CRSRES][687806784]0StopResource: setting CLI values
2014-10-24 11:23:10.221: [ CRSRES][687806784]0Attempting to stop `ora.amcmstvdb01.vip` on member `amcmstvdb01`
2014-10-24 11:23:11.232: [ CRSRES][687806784]0Stop of `ora.amcmstvdb01.vip` on member `amcmstvdb01` succeeded.
2014-10-24 11:23:11.268: [ CRSRES][687806784]0ora.amcmstvdb01.gsd target set to OFFLINE before stop action
2014-10-24 11:23:11.269: [ CRSRES][687806784]0StopResource: setting CLI values
2014-10-24 11:23:11.277: [ CRSRES][687806784]0Attempting to stop `ora.amcmstvdb01.gsd` on member `amcmstvdb01`
2014-10-24 11:23:12.284: [ CRSRES][687806784]0Stop of `ora.amcmstvdb01.gsd` on member `amcmstvdb01` succeeded.
2014-10-24 11:23:23.714: [ CRSRES][687806784]0startRunnable: setting CLI values
2014-10-24 11:23:23.723: [ CRSRES][687806784]0Attempting to start `ora.amcmstvdb01.gsd` on member `amcmstvdb01`
2014-10-24 11:23:24.740: [ CRSRES][687806784]0Start of `ora.amcmstvdb01.gsd` on member `amcmstvdb01` succeeded.
2014-10-24 11:23:24.840: [ CRSRES][687806784]0startRunnable: setting CLI values
2014-10-24 11:23:24.847: [ CRSRES][687806784]0Attempting to start `ora.amcmstvdb01.vip` on member `amcmstvdb01`
2014-10-24 11:23:26.857: [ CRSRES][687806784]0Start of `ora.amcmstvdb01.vip` on member `amcmstvdb01` succeeded.
2014-10-24 11:23:26.962: [ CRSRES][687806784]0startRunnable: setting CLI values
2014-10-24 11:23:26.971: [ CRSRES][687806784]0Attempting to start `ora.amcmstvdb01.ons` on member `amcmstvdb01`
2014-10-24 11:23:27.017: [ OCRUTL][683604288]u_freem: mem passed is null
2014-10-24 11:23:28.983: [ CRSRES][687806784]0Start of `ora.amcmstvdb01.ons` on member `amcmstvdb01` succeeded.
2014-10-24 11:23:29.093: [ CRSRES][687806784]0startRunnable: setting CLI values
2014-10-24 11:23:29.099: [ CRSRES][687806784]0Attempting to start `ora.amcmstvdb01.LISTENER_AMCMSTVDB01.lsnr` on member `amcmstvdb01`
2014-10-24 11:23:30.110: [ CRSRES][687806784]0Start of `ora.amcmstvdb01.LISTENER_AMCMSTVDB01.lsnr` on member `amcmstvdb01` succeeded.
2014-10-27 03:07:00.415: [ CRSAPP][687806784]0CheckResource error for ora.amcmstvdb01.LISTENER_AMCMSTVDB01.lsnr error code = 1
2014-10-27 03:07:00.421: [ CRSRES][687806784]0In stateChanged, ora.amcmstvdb01.LISTENER_AMCMSTVDB01.lsnr target is ONLINE
2014-10-27 03:07:00.423: [ CRSRES][687806784]0ora.amcmstvdb01.LISTENER_AMCMSTVDB01.lsnr on amcmstvdb01 went OFFLINE unexpectedly
2014-10-27 03:07:00.424: [ CRSRES][687806784]0StopResource: setting CLI values
2014-10-27 03:07:00.430: [ CRSRES][687806784]0Attempting to stop `ora.amcmstvdb01.LISTENER_AMCMSTVDB01.lsnr` on member `amcmstvdb01`
2014-10-27 03:07:02.437: [ CRSRES][687806784]0Stop of `ora.amcmstvdb01.LISTENER_AMCMSTVDB01.lsnr` on member `amcmstvdb01` succeeded.
2014-10-27 03:07:02.439: [ CRSRES][687806784]0ora.amcmstvdb01.LISTENER_AMCMSTVDB01.lsnr RESTART_COUNT=0 RESTART_ATTEMPTS=5
2014-10-27 03:07:02.440: [ CRSRES][687806784]0Restarting ora.amcmstvdb01.LISTENER_AMCMSTVDB01.lsnr on amcmstvdb01
2014-10-27 03:07:02.443: [ CRSRES][687806784]0startRunnable: setting CLI values
2014-10-27 03:07:02.444: [ CRSRES][687806784]0Attempting to start `ora.amcmstvdb01.LISTENER_AMCMSTVDB01.lsnr` on member `amcmstvdb01`
2014-10-27 03:07:04.564: [ CRSRES][687806784]0Start of `ora.amcmstvdb01.LISTENER_AMCMSTVDB01.lsnr` on member `amcmstvdb01` succeeded.
2014-10-27 03:07:04.566: [ CRSRES][687806784]0Successfully restarted ora.amcmstvdb01.LISTENER_AMCMSTVDB01.lsnr on amcmstvdb01, RESTART_COUNT=1
2014-10-27 03:07:04.571: [ CRSRES][687806784]0ora.amcmstvdb01.LISTENER_AMCMSTVDB01.lsnr Updated LAST_RESTART time in ocr
2014-10-28 02:17:25.278: [ CRSEVT][875006272]0CAAMonitorHandler :: 0:Could not join /u01/crs/bin/racgwrap(check)
category: 1234, operation: scls_process_join, loc: childcrash, OS error: 0, other: Abnormal termination of the child
Might this be related:
https://support.oracle.com/epmos/faces/DocumentDisplay?_afrLoop=380194821865447&id=985170.1&displayIndex=2&_afrWindowMod…
https://support.oracle.com/epmos/faces/SearchDocDisplay?_adf.ctrl-state=drbucptmm_436&_afrLoop=385944517854402#SYMPTOM
I hope you can give me a hint or two on how to solve this issue.
Kind regards and thank you in advance.