Hi,
We have ords 24 configured on some of our internal servers and we have this issue where we lose connectivity to ORDS and the database sessions can't reestablish. Basically every night we drop a pdb and it is rebuilt using the same name from a template. twice out of the last 3 weeks we have an issue in the morning where no new ords connections can establish to the database.
We can see from the below log that the databases was closed at 03:04 last night.
2025-07-04T03:04:45.208314+01:00
Pluggable database PDB_NAME closed
When i look at the portal server logs i can see at 03:07 it failed to get a connection from the database at 03:07 which is fine since the databases was shutdown.
####<4 Jul 2025, 03:07:05,269 British Summer Time> <oracle.dbtools.rest> <<y7pTwWXA-tqWOwM87TQ_fw> HEAD hostname /pls/coreportal_pdb_name 571 The connection pool named: |pdb_name|lo|, encountered a network operation error: UCP-29: Failed to get a connection
DatabaseConnectionException [statusCode=571, logLevel=SEVERE, reasons=[The connection pool named: |pdb_name|lo|, encountered a network operation error: UCP-29: Failed to get a connection]]
at oracle.dbtools.http.errors.ServletResponseExceptionMapper.mapError(ServletResponseExceptionMapper.java:69)
at oracle.dbtools.http.errors.ErrorLogger.log(ErrorLogger.java:27)
at oracle.dbtools.http.errors.ErrorPageFilter.doFilter(ErrorPageFilter.java:111)
at oracle.dbtools.http.filters.HttpFilter.doFilter(HttpFilter.java:47)
at oracle.dbtools.http.filters.FilterChainImpl.doFilter(FilterChainImpl.java:64)
at oracle.dbtools.http.secure.ForceHttpsFilter.doFilter(ForceHttpsFilter.java:74)
at oracle.dbtools.http.filters.HttpFilter.doFilter(HttpFilter.java:47)
at oracle.dbtools.http.filters.FilterChainImpl.doFilter(FilterChainImpl.java:64)
at oracle.dbtools.http.auth.ForceAuthFilter.doFilter(ForceAuthFilter.java:44)
at oracle.dbtools.http.filters.HttpFilter.doFilter(HttpFilter.java:47)
at oracle.dbtools.http.filters.FilterChainImpl.doFilter(FilterChainImpl.java:64)
at oracle.dbtools.http.filters.Filters.filter(Filters.java:67)
at oracle.dbtools.http.entrypoint.EntryPoint.service(EntryPoint.java:70)
at oracle.dbtools.http.entrypoint.EntryPointServlet.service(EntryPointServlet.java:130)
at oracle.dbtools.entrypoint.WebApplicationRequestEntryPoint.service(WebApplicationRequestEntryPoint.java:50)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:750)
at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:295)
at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:260)
at weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:137)
at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:353)
at weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:25)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:82)
at weblogic.servlet.internal.RequestEventsFilter.doFilter(RequestEventsFilter.java:32)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:82)
at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3866)
at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3829)
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:344)
at weblogic.security.service.SecurityManager.runAsForUserCode(SecurityManager.java:197)
at weblogic.servlet.provider.WlsSecurityProvider.runAsForUserCode(WlsSecurityProvider.java:203)
at weblogic.servlet.provider.WlsSubjectHandle.run(WlsSubjectHandle.java:71)
at weblogic.servlet.internal.WebAppServletContext.processSecuredExecute(WebAppServletContext.java:2502)
at weblogic.servlet.internal.WebAppServletContext.doSecuredExecute(WebAppServletContext.java:2351)
at weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2326)
at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2304)
at weblogic.servlet.internal.ServletRequestImpl.runInternal(ServletRequestImpl.java:1779)
at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1733)
at weblogic.servlet.provider.ContainerSupportProviderImpl$WlsRequestExecutor.run(ContainerSupportProviderImpl.java:272)
at weblogic.invocation.ComponentInvocationContextManager._runAs(ComponentInvocationContextManager.java:352)
at weblogic.invocation.ComponentInvocationContextManager.runAs(ComponentInvocationContextManager.java:337)
at weblogic.work.LivePartitionUtility.doRunWorkUnderContext(LivePartitionUtility.java:57)
at weblogic.work.PartitionUtility.runWorkUnderContext(PartitionUtility.java:41)
at weblogic.work.SelfTuningWorkManagerImpl.runWorkUnderContext(SelfTuningWorkManagerImpl.java:651)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:420)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:360)
Caused By: oracle.dbtools.url.mapping.TargetNotAvailableException: The connection pool named: |pdb_name|lo|, encountered a network operation error: UCP-29: Failed to get a connection
The database was back online at 03:20
PDB_NAME (4):Successfully created internal service PDB_NAME at open
2025-07-04T03:20:01.563874+01:00
****************************************************************
Post plug operations are now complete.
Pluggable database PDB_NAME with pdb id - 4 is now marked as NEW.
But the below alert was generating in the portal logs until the following morning. The database was back online, no accounts were locked, we hadn't hit the max number of sessions since none could reestablish after the database was brought back online so i have no idea why this would happen twice in 3 weeks even though we repeat the same steps every night.
Anyone familiar with this and any idea what i need to look at? We had to restart ords both times to fix this.
####<4 Jul 2025, 09:08:12,465 British Summer Time> <oracle.dbtools.rest> <oracle.dbtools.common.jdbc.ConnectionPoolConfigurationException: ORDS was unable to make a connection to the database. This can occur if the database is unavailable, the maximum number of sessions has been reached or the pool is not correctly configured. The connection pool named: |PDB_NAME|lo| had the following error(s): ORA-01109: database not open
https://docs.oracle.com/error-help/db/ora-01109/
java.util.concurrent.ExecutionException: oracle.dbtools.common.jdbc.ConnectionPoolConfigurationException: ORDS was unable to make a connection to the database. This can occur if the database is unavailable, the maximum number of sessions has been reached or the pool is not correctly configured. The connection pool named: |pdb_name|lo| had the following error(s): ORA-01109: database not open
https://docs.oracle.com/error-help/db/ora-01109/
at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191)
at oracle.dbtools.common.config.db.RefreshablePoolDataSource$RefreshState.pool(RefreshablePoolDataSource.java:425)
at oracle.dbtools.common.config.db.RefreshablePoolDataSource.delegate(RefreshablePoolDataSource.java:203)
at oracle.dbtools.common.config.db.RefreshablePoolDataSource.isValid(RefreshablePoolDataSource.java:173)
at oracle.dbtools.common.config.db.DatabasePool.isValid(DatabasePool.java:95)
at oracle.dbtools.jdbc.pools.JDBCPool.isValid(JDBCPool.java:52)
at oracle.dbtools.jdbc.pools.JDBCPoolCache.pool(JDBCPoolCache.java:147)
at oracle.dbtools.jdbc.pools.JDBCPoolCache.pool(JDBCPoolCache.java:81)
at oracle.dbtools.jdbc.pools.JDBCPoolResolver.resolve(JDBCPoolResolver.java:53)
at oracle.dbtools.jdbc.pools.JDBCSchemaTarget.lambda$new$0(JDBCSchemaTarget.java:49)
at oracle.dbtools.common.functions.MaybeValidImpl.load(MaybeValidImpl.java:206)
at oracle.dbtools.common.functions.MaybeValidImpl.(MaybeValidImpl.java:31)
at oracle.dbtools.common.functions.MaybeValidImpl$LoaderImpl.load(MaybeValidImpl.java:269)
at oracle.dbtools.common.functions.MaybeValidImpl$LoaderImpl.load(MaybeValidImpl.java:241)
at oracle.dbtools.jdbc.pools.JDBCSchemaTarget.(JDBCSchemaTarget.java:49)
at oracle.dbtools.jdbc.pools.JDBCSchemaTarget$Builder.build(JDBCSchemaTarget.java:283)
at oracle.dbtools.jdbc.pools.JDBCSchemaTarget.schemaTarget(JDBCSchemaTarget.java:243)
at oracle.dbtools.jdbc.pools.JDBCPoolURLMapping.(JDBCPoolURLMapping.java:34)
at oracle.dbtools.jdbc.pools.JDBCTenantMappings.mapping(JDBCTenantMappings.java:78)
at oracle.dbtools.jdbc.pools.JDBCTenantMappings.mapping(JDBCTenantMappings.java:61)
at oracle.dbtools.jdbc.pools.local.DefaultLocalTenantMapping.mapUrlPath(DefaultLocalTenantMapping.java:236)
at oracle.dbtools.jdbc.pools.local.DefaultLocalTenantMapping.doFilter(DefaultLocalTenantMapping.java:104)
at oracle.dbtools.url.mapping.tenant.TenantMappingDispatcher.dispatch(TenantMappingDispatcher.java:59)
at oracle.dbtools.url.mapping.tenant.TenantMappingFilter.doFilter(TenantMappingFilter.java:91)
at oracle.dbtools.http.filters.HttpFilter.doFilter(HttpFilter.java:47)
at oracle.dbtools.http.filters.FilterChainImpl.doFilter(FilterChainImpl.java:64)
at oracle.dbtools.http.forwarding.ForwardingFailedFilter.doFilter(ForwardingFailedFilter.java:41)
at oracle.dbtools.http.filters.HttpFilter.doFilter(HttpFilter.java:47)
at oracle.dbtools.http.filters.FilterChainImpl.doFilter(FilterChainImpl.java:64)
at oracle.dbtools.http.auth.external.ExternalSessionFilter.doFilter(ExternalSessionFilter.java:59)
at oracle.dbtools.http.filters.HttpFilter.doFilter(HttpFilter.java:47)
at oracle.dbtools.http.filters.FilterChainImpl.doFilter(FilterChainImpl.java:64)
at oracle.dbtools.apex.support.auth.ApexSessionQueryRewriteFilter.doFilter(ApexSessionQueryRewriteFilter.java:58)
at oracle.dbtools.http.filters.HttpFilter.doFilter(HttpFilter.java:47)
at oracle.dbtools.http.filters.FilterChainImpl.doFilter(FilterChainImpl.java:64)
at oracle.dbtools.http.cors.CORSResponseFilter.doFilter(CORSResponseFilter.java:90)
at oracle.dbtools.http.filters.HttpResponseFilter.doFilter(HttpResponseFilter.java:45)
at oracle.dbtools.http.filters.FilterChainImpl.doFilter(FilterChainImpl.java:64)
at oracle.dbtools.http.filters.AbsoluteLocationFilter.doFilter(AbsoluteLocationFilter.java:65)
at oracle.dbtools.http.filters.HttpResponseFilter.doFilter(HttpResponseFilter.java:45)
at oracle.dbtools.http.filters.FilterChainImpl.doFilter(FilterChainImpl.java:64)
at oracle.dbtools.http.auth.external.ExternalAccessValidationFilter.doFilter(ExternalAccessValidationFilter.java:59)
at oracle.dbtools.http.filters.HttpFilter.doFilter(HttpFilter.java:47)
at oracle.dbtools.http.filters.FilterChainImpl.doFilter(FilterChainImpl.java:64)
at oracle.dbtools.http.forwarding.TrailingSlashes.checkForMissingTrailingSlash(TrailingSlashes.java:48)
at oracle.dbtools.http.errors.ErrorPageFilter.checkForMissingTrailingSlash(ErrorPageFilter.java:142)
at oracle.dbtools.http.errors.ErrorPageFilter.doFilter(ErrorPageFilter.java:108)
at oracle.dbtools.http.filters.HttpFilter.doFilter(HttpFilter.java:47)
at oracle.dbtools.http.filters.FilterChainImpl.doFilter(FilterChainImpl.java:64)
at oracle.dbtools.http.secure.ForceHttpsFilter.doFilter(ForceHttpsFilter.java:74)
at oracle.dbtools.http.filters.HttpFilter.doFilter(HttpFilter.java:47)
at oracle.dbtools.http.filters.FilterChainImpl.doFilter(FilterChainImpl.java:64)
at oracle.dbtools.http.auth.ForceAuthFilter.doFilter(ForceAuthFilter.java:44)
at oracle.dbtools.http.filters.HttpFilter.doFilter(HttpFilter.java:47)
at oracle.dbtools.http.filters.FilterChainImpl.doFilter(FilterChainImpl.java:64)
at oracle.dbtools.http.filters.Filters.filter(Filters.java:67)
at oracle.dbtools.http.entrypoint.EntryPoint.service(EntryPoint.java:70)
at oracle.dbtools.http.entrypoint.EntryPointServlet.service(EntryPointServlet.java:130)
at oracle.dbtools.entrypoint.WebApplicationRequestEntryPoint.service(WebApplicationRequestEntryPoint.java:50)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:750)
at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:295)
at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:260)
at weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:137)
at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:353)
at weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:25)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:82)
at weblogic.servlet.internal.RequestEventsFilter.doFilter(RequestEventsFilter.java:32)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:82)
at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3866)
at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3829)
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:344)
at weblogic.security.service.SecurityManager.runAsForUserCode(SecurityManager.java:197)
at weblogic.servlet.provider.WlsSecurityProvider.runAsForUserCode(WlsSecurityProvider.java:203)
at weblogic.servlet.provider.WlsSubjectHandle.run(WlsSubjectHandle.java:71)
at weblogic.servlet.internal.WebAppServletContext.processSecuredExecute(WebAppServletContext.java:2502)
at weblogic.servlet.internal.WebAppServletContext.doSecuredExecute(WebAppServletContext.java:2351)
at weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2326)
at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2304)
at weblogic.servlet.internal.ServletRequestImpl.runInternal(ServletRequestImpl.java:1779)
at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1733)
at weblogic.servlet.provider.ContainerSupportProviderImpl$WlsRequestExecutor.run(ContainerSupportProviderImpl.java:272)
at weblogic.invocation.ComponentInvocationContextManager._runAs(ComponentInvocationContextManager.java:352)
at weblogic.invocation.ComponentInvocationContextManager.runAs(ComponentInvocationContextManager.java:337)
at weblogic.work.LivePartitionUtility.doRunWorkUnderContext(LivePartitionUtility.java:57)
at weblogic.work.PartitionUtility.runWorkUnderContext(PartitionUtility.java:41)
at weblogic.work.SelfTuningWorkManagerImpl.runWorkUnderContext(SelfTuningWorkManagerImpl.java:651)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:420)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:360)
Caused By: oracle.dbtools.common.jdbc.ConnectionPoolConfigurationException: ORDS was unable to make a connection to the database. This can occur if the database is unavailable, the maximum number of sessions has been reached or the pool is not correctly configured. The connection pool named: |pdb_name|lo| had the following error(s): ORA-01109: database not open