Skip to Main Content
Forums

Java Database Connectivity (JDBC)

Announcement

For appeals, questions and feedback about Oracle Forums, please email oracle-forums-moderators_us@oracle.com. Technical questions should be asked in the appropriate category. Thank you!

Interested in getting your voice heard by members of the Developer Marketing team at Oracle? Check out this post for AppDev or this post for AI focus group information.

Getting 'java.net.SocketException: Connection timed out' after long running process

User_CIP5OJan 25 2019 — edited Jan 28 2019

Hi team,

We're getting 'java.net.SocketException: Connection timed out' in our application.

Below is the application stacktrace:

...
Caused by: java.sql.SQLRecoverableException: Closed Connection
        at oracle.jdbc.driver.PhysicalConnection.getAutoCommit(PhysicalConnection.java:2254) ~[ojdbc7.jar:12.1.0.2.0]
        at oracle.jdbc.driver.PhysicalConnection.rollback(PhysicalConnection.java:2424) ~[ojdbc7.jar:12.1.0.2.0]
        at weblogic.jdbc.wrapper.Connection.rollback(Connection.java:1785) ~[com.bea.core.datasource6.jar:12.2.1.2]
        at org.hibernate.resource.jdbc.internal.AbstractLogicalConnectionImplementor.rollback(AbstractLogicalConnectionImplementor.java:115) ~[hibernate-core-5.1.1.Final.jar:5.1.1.Final]
        at org.hibernate.resource.transaction.backend.jdbc.internal.JdbcResourceLocalTransactionCoordinatorImpl$TransactionDriverControlImpl.rollback(JdbcResourceLocalTransactionCoordina
torImpl.java:250) ~[hibernate-core-5.1.1.Final.jar:5.1.1.Final]
        at org.hibernate.engine.transaction.internal.TransactionImpl.rollback(TransactionImpl.java:89) ~[hibernate-core-5.1.1.Final.jar:5.1.1.Final]
        at org.hibernate.jpa.internal.TransactionImpl.rollback(TransactionImpl.java:101) ~[hibernate-entitymanager-5.1.1.Final.jar:5.1.1.Final]
        at org.springframework.orm.jpa.JpaTransactionManager.doRollback(JpaTransactionManager.java:563) ~[spring-orm-5.0.6.RELEASE.jar:5.0.6.RELEASE]
        ... 18 more

Further digging we found the following stacktrace related to the SocketException:

Caused by: java.sql.SQLRecoverableException: IO Error: Connection timed out
at oracle.jdbc.driver.T4CPreparedStatement.executeForDescribe(T4CPreparedStatement.java:790)
at oracle.jdbc.driver.OracleStatement.executeMaybeDescribe(OracleStatement.java:925)
at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1111)
at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:4798)
at oracle.jdbc.driver.OraclePreparedStatement.executeQuery(OraclePreparedStatement.java:4845)
at oracle.jdbc.driver.OraclePreparedStatementWrapper.executeQuery(OraclePreparedStatementWrapper.java:1501)
at weblogic.jdbc.wrapper.PreparedStatement.executeQuery(PreparedStatement.java:144)
at org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.extract(ResultSetReturnImpl.java:70)
... 47 more
Caused by: java.net.SocketException: Connection timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:170)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at oracle.net.ns.Packet.receive(Packet.java:311)
at oracle.net.ns.DataPacket.receive(DataPacket.java:105)
at oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:305)
at oracle.net.ns.NetInputStream.read(NetInputStream.java:249)
at oracle.net.ns.NetInputStream.read(NetInputStream.java:171)
at oracle.net.ns.NetInputStream.read(NetInputStream.java:89)
at oracle.jdbc.driver.T4CSocketInputStreamWrapper.readNextPacket(T4CSocketInputStreamWrapper.java:123)
at oracle.jdbc.driver.T4CSocketInputStreamWrapper.read(T4CSocketInputStreamWrapper.java:79)
at oracle.jdbc.driver.T4CMAREngineStream.unmarshalUB1(T4CMAREngineStream.java:429)
at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:397)
at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:257)
at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:587)
at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:225)
at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:53)
at oracle.jdbc.driver.T4CPreparedStatement.executeForDescribe(T4CPreparedStatement.java:774)

Below is the fact about the application:

1. It's deployed in Weblogic 12.2.1.2.0 running on Linux 64bit.

2. The error occurs only after long running process (>20mins).

3. The WL & DB server are in different subnet.

4. No idle timeout is specified in the firewall between WL & DB server.

5. Tcp dump shows that client sends FIN at the time the error occurs.

6. We've specified 'Test connection on reserved' with test freq. 120secs in WL.

The error happens when app tries to query using connection after leaving it idle for some time.

Any help would be greatly appreciated.

Regards,

James
This post has been answered by Zlatko Sirotic on Jan 27 2019
Jump to Answer

Comments

Zlatko Sirotic

Because you said this:

> 3. The WL & DB server are in different subnet.

I would first suspect the firewall settings, but you also said this:

> 4. No idle timeout is specified in the firewall between WL & DB server.

so the firewall should not be a problem.

I assume you are using a connection pool, but I do not know which (perhaps Oracle UCP).

Check the setting that defines the maximum time after which the connection is returned to the connection pool, if the user is inactive.

Regards,

Zlatko

Zlatko Sirotic

Check:

Inactive Connection Timeout

The number of seconds of inactivity after which reserved connections will forcibly be released back into the pool. When set to 0 (the default), this feature is disabled.

Regards,

Zlatko

User_CIP5O

Hi Zlatko,

We've checked the setting you mentioned and it's the default (0).

And I forgot to mention that not all long running process ends up with this error.

Regards,

James

Zlatko Sirotic

> And I forgot to mention that not all long running process ends up with this error.

Do you use Connection Harvesting?

https://docs.oracle.com/middleware/1212/wls/JDBCA/ds_harvesting.htm#JDBCA658

You can specify a number of reserved connections to be released when a data source reaches a specified number of available connections.

Harvesting helps to ensure that a specified number of connections are always available in the pool and improves performance by minimizing connection initialization ...

Regards,

Zlatko

User_CIP5O

Hi Zlato,

If using it means we have to change the setting other than default then the answer is no.

Regards,

James

Zlatko Sirotic

It's possible that the problem is related to Hibernate settings!?

Unfortunately, I do not know almost anything about Hibernate

(except that it, as JDBC in general, has AutoCommit = true as default, which in most cases is bad for data integrity).

Regards,

Zlatko

User_CIP5O

Hi Zlatko,

I doubt that Hibernate has something to do with it since it sits far above.

We'll write standalone straight JDBC snippet and see if we can reproduce the errors.

Regards,

James

Zlatko Sirotic

> I doubt that Hibernate has something to do with it since it sits far above.

I agree but, for example, Hibernate supports the javax.persistence.query.timeout query hint to call the setQueryTimeout() method on the JDBC Statement.

I do not want to say that setQueryTimeout() is a "culprit" for closing a connection - this method does not close the connection (unless there is another code that explicitly does this, perhaps in exception handler).

Regards,

Zlatko

User_CIP5O

Hi Zlatko,

Thank you for your insight.

Anyway, I'm still open for possibilities including Hibernate, I will look into it.

Regards,

James

User_CIP5O

Hi Zlatko,

I have written & run small snippets just to connect to DB and run query 'select 1 from DUAL'. The snippet will run the query then sleep for 1 hour after which it will run the same query again using the same connection.

Below is the result:

1. On my laptop to local XE DB -> SUCCESS.

2. On our test environment where WL & DB server are in the same subnet -> SUCCESS.

3. On production servers, where WL & DB are in different subnet -> FAILED.

#3 failed with the following stacktrace:

java.sql.SQLRecoverableException: IO Error: Connection timed out
        at oracle.jdbc.driver.T4CPreparedStatement.executeForDescribe(T4CPreparedStatement.java:779) ~[ojdbc8-18.3.0.0.0.jar:18.3.0.0.0]
        at oracle.jdbc.driver.OracleStatement.executeMaybeDescribe(OracleStatement.java:921) ~[ojdbc8-18.3.0.0.0.jar:18.3.0.0.0]
        at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1099) ~[ojdbc8-18.3.0.0.0.jar:18.3.0.0.0]
        at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3640) ~[ojdbc8-18.3.0.0.0.jar:18.3.0.0.0]
        at oracle.jdbc.driver.T4CPreparedStatement.executeInternal(T4CPreparedStatement.java:1384) ~[ojdbc8-18.3.0.0.0.jar:18.3.0.0.0]
        at oracle.jdbc.driver.OraclePreparedStatement.executeQuery(OraclePreparedStatement.java:3687) ~[ojdbc8-18.3.0.0.0.jar:18.3.0.0.0]
        at oracle.jdbc.driver.OraclePreparedStatementWrapper.executeQuery(OraclePreparedStatementWrapper.java:1165) ~[ojdbc8-18.3.0.0.0.jar:18.3.0.0.0]
...
Caused by: java.io.IOException: Connection timed out
        at sun.nio.ch.FileDispatcherImpl.read0(Native Method) ~[?:1.8.0_71]
        at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) ~[?:1.8.0_71]
        at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) ~[?:1.8.0_71]
        at sun.nio.ch.IOUtil.read(IOUtil.java:197) ~[?:1.8.0_71]
        at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380) ~[?:1.8.0_71]
        at oracle.net.nt.TimeoutSocketChannel.read(TimeoutSocketChannel.java:178) ~[ojdbc8-18.3.0.0.0.jar:18.3.0.0.0]
        at oracle.net.ns.NSProtocolNIO.doSocketRead(NSProtocolNIO.java:544) ~[ojdbc8-18.3.0.0.0.jar:18.3.0.0.0]
        at oracle.net.ns.NIOPacket.readHeader(NIOPacket.java:234) ~[ojdbc8-18.3.0.0.0.jar:18.3.0.0.0]
        at oracle.net.ns.NIOPacket.readPacketFromSocketChannel(NIOPacket.java:174) ~[ojdbc8-18.3.0.0.0.jar:18.3.0.0.0]
        at oracle.net.ns.NIOPacket.readFromSocketChannel(NIOPacket.java:122) ~[ojdbc8-18.3.0.0.0.jar:18.3.0.0.0]
        at oracle.net.ns.NIOPacket.readFromSocketChannel(NIOPacket.java:100) ~[ojdbc8-18.3.0.0.0.jar:18.3.0.0.0]
        at oracle.net.ns.NIONSDataChannel.readDataFromSocketChannel(NIONSDataChannel.java:86) ~[ojdbc8-18.3.0.0.0.jar:18.3.0.0.0]
        at oracle.jdbc.driver.T4CMAREngineNIO.prepareForUnmarshall(T4CMAREngineNIO.java:762) ~[ojdbc8-18.3.0.0.0.jar:18.3.0.0.0]
        at oracle.jdbc.driver.T4CMAREngineNIO.unmarshalUB1(T4CMAREngineNIO.java:427) ~[ojdbc8-18.3.0.0.0.jar:18.3.0.0.0]
        at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:394) ~[ojdbc8-18.3.0.0.0.jar:18.3.0.0.0]
        at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:255) ~[ojdbc8-18.3.0.0.0.jar:18.3.0.0.0]
        at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:610) ~[ojdbc8-18.3.0.0.0.jar:18.3.0.0.0]
        at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:253) ~[ojdbc8-18.3.0.0.0.jar:18.3.0.0.0]
        at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:86) ~[ojdbc8-18.3.0.0.0.jar:18.3.0.0.0]
        at oracle.jdbc.driver.T4CPreparedStatement.executeForDescribe(T4CPreparedStatement.java:765) ~[ojdbc8-18.3.0.0.0.jar:18.3.0.0.0]
        ... 8 more

This is either network related issue or DB setting.

Regards,

James

Zlatko Sirotic
Answer

It would be good to check the settings on the firewall once again.

It has happened to me several times that network engineers have said that the firewall does not interrupt the connection.

After an unsuccessful search for another culprit (database, connection pool, JDBC driver, application server ...), they checked in more detail and saw that they had made a mistake.

Regards,

Zlatko

Marked as Answer by User_CIP5O · Jan 27 2019
User_CIP5O

Hi Zlato,

I've made the network admin check the firewall, this time he added additional parameter that finally nailed the problem.

Regards,

James

User_7ZWP1

Hi James,
I'm facing similar issue while connecting to DB server from WebLogic server via data source connection intermittently, both servers are in different subnet. Could you please help me with the network settings done on firewall to resolve.
Regards,
Ricky

shaz-ali

In my case network guy is saying that packets are dropped intermittently by giving the reason -"TCP segment with urgent pointer. Urgent data indication was stripped" from WebLogic server to DB server.

@zlatko-sirotic do u have any suggestions on this.

1 - 14

Post Details

Added on Jan 25 2019
#java-database-connectivity-jdbc
14 comments
20,412 views
1 mention