This is an Oracle Net question I believe but I couldn't find a specific discussion group for that any more so I apologize for putting it here and please redirect to the correct discussion group.
Starting a couple of days ago I started seeing repeated connection timeout messages in the alert log on one of my Production database servers. As seen below the timeouts reference two (2) specific client IP addresses, xx.xx.40.67 and xx.xx.161.69. The 40.67 is the Production database server and the 161.69 is the Tomcat server.
The database server is in a Data Guard environment to a Physical Standby on a database server in the DR site but that server IP is not referenced in any of the alert log messages. I mention that only because I did re-create the Physical Standby just before the timeout messages started appearing. I have checked the status of the Standby and it is in perfect sync with the Primary and I see no connection or timeout issues between those two servers.
Here is a section of the alert log showing a timeout from both the above clients:
63,8507d8462
Thu Oct 13 08:28:04 2016
***********************************************************************
Fatal NI connect error 12170.
VERSION INFORMATION:
TNS for Solaris: Version 11.2.0.4.0 - Production
Oracle Bequeath NT Protocol Adapter for Solaris: Version 11.2.0.4.0 - Production
TCP/IP NT Protocol Adapter for Solaris: Version 11.2.0.4.0 - Production
Time: 13-OCT-2016 08:28:04
Tracing not turned on.
Tns error struct:
ns main err code: 12535
TNS-12535: TNS:operation timed out
ns secondary err code: 12609
nt main err code: 0
nt secondary err code: 0
nt OS err code: 0
Client address: (ADDRESS=(PROTOCOL=tcp)(HOST=xx.xx.40.67)(PORT=62789))
Thu Oct 13 08:28:20 2016
***********************************************************************
Fatal NI connect error 12170.
VERSION INFORMATION:
TNS for Solaris: Version 11.2.0.4.0 - Production
Oracle Bequeath NT Protocol Adapter for Solaris: Version 11.2.0.4.0 - Production
TCP/IP NT Protocol Adapter for Solaris: Version 11.2.0.4.0 - Production
Time: 13-OCT-2016 08:28:20
Tracing not turned on.
Tns error struct:
ns main err code: 12535
TNS-12535: TNS:operation timed out
ns secondary err code: 12606
nt main err code: 0
nt secondary err code: 0
nt OS err code: 0
Client address: (ADDRESS=(PROTOCOL=tcp)(HOST=xx.xx.161.69)(PORT=65240))
WARNING: inbound connection timed out (ORA-3136)
Here is the section of the listener log around the time of the timeouts seen above:
<msg time='2016-10-13T08:24:21.738-04:00' org_id='oracle' comp_id='tnslsnr'
type='UNKNOWN' level='16' host_id='apspdba'
host_addr='10.182.192.67'>
<txt>13-OCT-2016 08:24:21 * (CONNECT_DATA=(SID=APSKTLP1)(CID=(PROGRAM=perl@apspdba)(HOST=apspdba)(USER=oracle))) * (ADDRESS=(PROTOCOL=tcp)(HOST=xx.xx.40.67)(PORT=63297)) * establish * APSKTLP1 * 0
</txt>
</msg>
<msg time='2016-10-13T08:24:22.022-04:00' org_id='oracle' comp_id='tnslsnr'
type='UNKNOWN' level='16' host_id='apspdba'
host_addr='xx.xx.192.67'>
<txt>13-OCT-2016 08:24:22 * (CONNECT_DATA=(SID=APSKTLP1)(CID=(PROGRAM=perl@apspdba)(HOST=apspdba)(USER=oracle))) * (ADDRESS=(PROTOCOL=tcp)(HOST=xx.xx.40.67)(PORT=63300)) * establish * APSKTLP1 * 0
</txt>
</msg>
<msg time='2016-10-13T08:24:22.307-04:00' org_id='oracle' comp_id='tnslsnr'
type='UNKNOWN' level='16' host_id='apspdba'
host_addr='xx.xx.192.67'>
<txt>13-OCT-2016 08:24:22 * (CONNECT_DATA=(SID=APSKTLP1)(CID=(PROGRAM=perl@apspdba)(HOST=apspdba)(USER=oracle))) * (ADDRESS=(PROTOCOL=tcp)(HOST=xx.xx.40.67)(PORT=63301)) * establish * APSKTLP1 * 0
</txt>
</msg>
<msg time='2016-10-13T08:25:17.356-04:00' org_id='oracle' comp_id='tnslsnr'
type='UNKNOWN' level='16' host_id='apspdba'
host_addr='xx.xx.192.67'>
<txt>13-OCT-2016 08:25:17 * (CONNECT_DATA=(SID=APSKTLP1)(CID=(PROGRAM=perl@apspdba)(HOST=apspdba)(USER=oracle))) * (ADDRESS=(PROTOCOL=tcp)(HOST=xx.xx.40.67)(PORT=63412)) * establish * APSKTLP1 * 0
</txt>
</msg>
<msg time='2016-10-13T08:26:13.441-04:00' org_id='oracle' comp_id='tnslsnr'
type='UNKNOWN' level='16' host_id='apspdba'
host_addr='xx.xx.192.67'>
<txt>13-OCT-2016 08:26:13 * (CONNECT_DATA=(SID=APSKTLP1)(CID=(PROGRAM=perl@apspdba)(HOST=apspdba)(USER=oracle))) * (ADDRESS=(PROTOCOL=tcp)(HOST=xx.xx.40.67)(PORT=63477)) * establish * APSKTLP1 * 0
</txt>
</msg>
<msg time='2016-10-13T08:26:37.457-04:00' org_id='oracle' comp_id='tnslsnr'
type='UNKNOWN' level='16' host_id='apspdba'
host_addr='xx.xx.192.67'>
<txt>13-OCT-2016 08:26:37 * (CONNECT_DATA=(SID=APSKTLP1)(CID=(PROGRAM=perl@apspdba)(HOST=apspdba)(USER=oracle))) * (ADDRESS=(PROTOCOL=tcp)(HOST=xx.xx.40.67)(PORT=63498)) * establish * APSKTLP1 * 0
</txt>
</msg>
<msg time='2016-10-13T08:28:04.228-04:00' org_id='oracle' comp_id='tnslsnr'
type='UNKNOWN' level='16' host_id='apspdba'
host_addr='xx.xx.192.67'>
<txt>13-OCT-2016 08:28:04 * (CONNECT_DATA=(SID=APSKTLP1)(CID=(PROGRAM=emagent)(HOST=apspdba)(USER=oracle))) * (ADDRESS=(PROTOCOL=tcp)(HOST=xx.xx.40.67)(PORT=63589)) * establish * APSKTLP1 * 0
</txt>
</msg>
<msg time='2016-10-13T08:28:04.488-04:00' org_id='oracle' comp_id='tnslsnr'
type='UNKNOWN' level='16' host_id='apspdba'
host_addr='xx.xx.192.67'>
<txt>13-OCT-2016 08:28:04 * (CONNECT_DATA=(SID=APSKTLP1)(CID=(PROGRAM=perl@apspdba)(HOST=apspdba)(USER=oracle))) * (ADDRESS=(PROTOCOL=tcp)(HOST=xx.xx.40.67)(PORT=63591)) * establish * APSKTLP1 * 0
</txt>
</msg>
<msg time='2016-10-13T08:28:04.774-04:00' org_id='oracle' comp_id='tnslsnr'
type='UNKNOWN' level='16' host_id='apspdba'
host_addr='xx.xx.192.67'>
<txt>13-OCT-2016 08:28:04 * (CONNECT_DATA=(SID=APSKTLP1)(CID=(PROGRAM=perl@apspdba)(HOST=apspdba)(USER=oracle))) * (ADDRESS=(PROTOCOL=tcp)(HOST=xx.xx.40.67)(PORT=63592)) * establish * APSKTLP1 * 0
</txt>
</msg>
<msg time='2016-10-13T08:28:18.928-04:00' org_id='oracle' comp_id='tnslsnr'
type='UNKNOWN' level='16' host_id='apspdba'
host_addr='xx.xx.192.67'>
<txt>13-OCT-2016 08:28:18 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=apspdba)(USER=oracle))(COMMAND=services)(ARGUMENTS=64)(SERVICE=(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=xx.xx.40.67)(PORT=1517))))(VERSION=186647552)) * services * 0
</txt>
</msg>
<msg time='2016-10-13T08:28:19.788-04:00' org_id='oracle' comp_id='tnslsnr'
type='UNKNOWN' level='16' host_id='apspdba'
host_addr='xx.xx.192.67'>
<txt>WARNING: Subscription for node down event still pending
</txt>
</msg>
<msg time='2016-10-13T08:28:19.790-04:00' org_id='oracle' comp_id='tnslsnr'
type='UNKNOWN' level='16' host_id='apspdba'
host_addr='xx.xx.192.67'>
<txt>13-OCT-2016 08:28:19 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=apspdba)(USER=oracle))(COMMAND=status)(ARGUMENTS=64)(SERVICE=(ADDRESS=(PROTOCOL=TCP)(HOST=xx.xx.40.67)(PORT=1517)))(VERSION=186647552)) * status * 0
</txt>
</msg>
<msg time='2016-10-13T08:29:14.647-04:00' org_id='oracle' comp_id='tnslsnr'
type='UNKNOWN' level='16' host_id='apspdba'
host_addr='xx.xx.192.67'>
<txt>13-OCT-2016 08:29:14 * ping * 0
</txt>
</msg>
<msg time='2016-10-13T08:29:15.138-04:00' org_id='oracle' comp_id='tnslsnr'
type='UNKNOWN' level='16' host_id='apspdba'
host_addr='xx.xx.192.67'>
<txt>WARNING: Subscription for node down event still pending
</txt>
</msg>
<msg time='2016-10-13T08:29:15.140-04:00' org_id='oracle' comp_id='tnslsnr'
type='UNKNOWN' level='16' host_id='apspdba'
host_addr='xx.xx.192.67'>
<txt>13-OCT-2016 08:29:15 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=apspdba)(USER=oracle))(COMMAND=status)(ARGUMENTS=64)(SERVICE=(ADDRESS=(PROTOCOL=TCP)(HOST=xx.xx.40.67)(PORT=1517)))(VERSION=186647552)) * status * 0
</txt>
The database server has two (2) interface cards in it, the application interface is the xx.xx.192.67 IP seen above and the management interface is the xx.xx.40.67 IP. All application servers are connecting via the application interface and user connections (TOAD, SQL*Developer, etc.) are connecting via the management interface.
I see two (2) IP's referenced as 'client' IP's, one being the database server and the other the Tomcat server.
Why would the database server be referenced as a 'client' other than for the Data Guard environment ?
What would be causing the timeouts between the Tomcat server and the database server and why would they have started all of a sudden ?
Any ideas are very welcome and if you need additional information, please advise.