Hi good folks,
this maybe is not issue but still need some advice.
One of our customers has ORDS with Tomcat and it has been crashed. I asked for some logs and found couple of things. Error on windows server indicating low virtual memory. And couple of errors in Tomcat logs:
The pool named: |apex|pu| is reporting no APEX workspaces are enabled for RESTful Services, but the pool named: |apex|al| is reporting 1 workspaces are enabled.
This implies APEX was installed after Oracle REST Data Services. Please run the following command and restart Oracle REST Data Services:
java -jar ords.war validate --database apex
APEX based RESTful Services, including those used to serve static resources will not function until this has been done.
However start up message indicates that everything has been configured OK:
03-Aug-2018 17:04:11.464 INFO [localhost-startStop-1] . Using configuration folder: E:\Oracle\REST\ords
03-Aug-2018 17:04:11.495 INFO [localhost-startStop-1] . Validating pool: |apex||
03-Aug-2018 17:04:12.136 INFO [localhost-startStop-1] . Pool: |apex|| is correctly configured
03-Aug-2018 17:04:12.136 INFO [localhost-startStop-1] . Validating pool: |apex|al|
03-Aug-2018 17:04:12.152 INFO [localhost-startStop-1] . Pool: |apex|al| is correctly configured
03-Aug-2018 17:04:12.167 INFO [localhost-startStop-1] . Validating pool: |apex|pu|
03-Aug-2018 17:04:12.183 INFO [localhost-startStop-1] . Pool: |apex|pu| is correctly configured
03-Aug-2018 17:04:12.183 INFO [localhost-startStop-1] . Validating pool: |apex|rt|
03-Aug-2018 17:04:12.214 INFO [localhost-startStop-1] . Pool: |apex|rt| is correctly configured
03-Aug-2018 17:04:12.464 INFO [localhost-startStop-1] . Oracle REST Data Services initialized
Also every time tomcat service has been stopped next has been logged:
WARNING [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [ords] appears to have started a thread named [UCP-worker-thread-4] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
And :
WARNING [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [ords] appears to have started a thread named [oracle.jdbc.driver.BlockSource.ThreadedCachingBlockSource.BlockReleaser] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.lang.Object.wait(Native Method)
oracle.jdbc.driver.BlockSource$ThreadedCachingBlockSource$BlockReleaser.run(BlockSource.java:327)
I tried to move ojdbc7-12.1.0.2-19030178-18642808.jar file from e:\TomcatAPEX\webapps\ords\WEB-INF\lib\ to e:\TomcatAPEX\lib\ as I found suggestion on net but even though ORDS works error message is still there.
Environment:
- Tomcat 8.0.50
- ORDS 3.0.12.263.15.32
- Windows 2008R2
- APEX 5.1.1.00.08
Conditions:
- Server is serving 2 more instances of Tomcat with some other applications without any issues (memory pools same as ORDS one)
- JAVA memory is set as 1024 MB for initial memory pool and 2048 MB for maximum memory pool
- Pictures and web pages are showing perfectly normal (no issues with 404 or such errors)
Questions:
- How can I get rid of warning messages or I should ignore them.
- Are errors in tomcat log directed by tomcat or passed from ORDS, and how can I increase error log level for ORDS? Log configuration from defaults.xml in ORDS folder seems does nothing.
- Should the manipulation of Windows Virtual memory and JAVA pools on Tomcat service be only solution?
Kind regards,