Hi,
I have my Oracle Enterprise Manager then an agent deployed to one of the target servers. The agent always crashes. After successfully restarting it (using emctl start agent), it again crash. No error on start but it always crash.
I checked this document EM 12c: Enterprise Manager 12c Cloud Control Agent Restarts Due to Out Of Memory Exception: java.lang.OutOfMemoryError: unable to create new native thread (Doc ID 1499576.1)
however it does not explain why my agent on the secondary server (its a dataguard setup, my primary sever has the problematic agent) does not crash despite having the same ulimit values as compared with the primary server.
Below is the error from emagent.nohup
May 16, 2018 1:39:53 PM oracle.eons.proxy.impl.ConnectionManagerImpl readFormFactor
WARNING: unable to locate formfactor file - /u01/app/oracle/agent12c/core/12.1.0.5.0/eons/conf/.formfactor
Agent is going down due to an OutOfMemoryError
----- Wed May 16 13:41:30 2018::69265::Checking status of EMAgent : 33680 -----
----- Wed May 16 13:41:30 2018::69265::EMAgent exited at Wed May 16 13:41:30 2018 with return value 57. -----
----- Wed May 16 13:41:30 2018::69265::EMAgent will be restarted because of an Out of Memory Exception. -----
----- Wed May 16 13:41:30 2018::69265::writeAbnormalExitTimestampToAgntStmp: exitCause=OOM : restartRequired=1 -----
----- Wed May 16 13:41:30 2018::69265::Restarting EMAgent. -----
----- Wed May 16 13:41:32 2018::69265::Auto tuning the agent at time Wed May 16 13:41:32 2018 -----
inMemoryLoggingSize=6291456
_SchedulePersistTimer=30
MaxThreads=10
agentJavaDefines=-Xmx185M -XX:MaxPermSize=96M
SchedulerRandomSpreadMins=5
UploadMaxNumberXML=5000
UploadMaxMegaBytesXML=50.0
Auto tuning was successful
----- Wed May 16 13:41:38 2018::69265::Finished auto tuning the agent at time Wed May 16 13:41:38 2018 -----
----- Wed May 16 13:41:38 2018::69265::Launching the JVM with following options: -Xmx185M -XX:MaxPermSize=96M -server -Djava.security.egd=file:///dev/./urandom -Dsun.lang.ClassLoader.allowArraySyntax=true -XX:+UseLinuxPosixThreadCPUClocks -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -XX:+UseCompressedOops -----
----- Wed May 16 13:41:38 2018::69265::Agent Launched with PID 37836 at time Wed May 16 13:41:38 2018 -----
----- Wed May 16 13:41:38 2018::37836::Execing EMAgent process is taking longer than expected 120 secs -----
----- Wed May 16 13:41:38 2018::37836::Time elapsed between Launch of Watchdog process and execing EMAgent is 10158 secs -----
2018-05-16 13:41:39,120 [1:main] WARN - Missing filename for log handler 'wsm'
2018-05-16 13:41:39,139 [1:main] WARN - Missing filename for log handler 'opss'
2018-05-16 13:41:39,141 [1:main] WARN - Missing filename for log handler 'opsscfg'
EONSPROVIDER: oracle.eons.proxy.impl.ONSFactoryImpl
May 16, 2018 1:41:46 PM oracle.eons.proxy.impl.ConnectionManagerImpl readFormFactor
WARNING: unable to locate formfactor file - /u01/app/oracle/agent12c/core/12.1.0.5.0/eons/conf/.formfactor
Agent is going down due to an OutOfMemoryError
----- Wed May 16 13:50:04 2018::69265::Checking status of EMAgent : 37836 -----
----- Wed May 16 13:50:04 2018::69265::EMAgent exited at Wed May 16 13:50:04 2018 with return value 57. -----
----- Wed May 16 13:50:04 2018::69265::EMAgent will be restarted because of an Out of Memory Exception. -----
----- Wed May 16 13:50:04 2018::69265::EMAgent is Thrashing. Exiting loop. -----
----- Wed May 16 13:50:04 2018::69265::Commiting Process death. -----
----- Wed May 16 13:50:04 2018::69265::writeAbnormalExitTimestampToAgntStmp: exitCause=OOM : restartRequired=0 -----
----- Wed May 16 13:50:04 2018::69265::Exited due to Thrash. -----