Hello everyone,
since a while we experience crashes of the Oracle Forms runtime resp. the JVM it is running in. This happens once a month and is not connected with heavy load situations. Till now we could not find out the reason for it.
The crash means that the Forms applicatoin is not reachable anymore and only a restart of the Weblogic helps to fix that problem till the next crash.
Each crash let the JVM create a hs_err_pid*.log file which means that a fatal error occurred. Each log file looks similar. I attached some interesting information of those log files at the end of the post.
Can anyone derive a possible reason for the crash, maybe from the stacktrace? Or has anyone an idea how to debug or further investigate the problem? Any hint is appreciated.
Thank you in advance,
Arnd
hs_err_pidxxxxx.log:
# JRE version: Java(TM) SE Runtime Environment (8.0_73-b02) (build 1.8.0_73-b02)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.73-b02 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# C [libopmnperf.so+0x2085] perfCacheUpdate+0x1c5
Stack: [0x00007fec87dfe000,0x00007fec87eff000], sp=0x00007fec87efaf10, free space=1011k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C [libopmnperf.so+0x2085] perfCacheUpdate+0x1c5
C [libopmnperf.so+0x1e3c] getProcStats+0x338
C [libifopmnwrp.so.0+0x821] Java_oracle_forms_servlet_ProcessMetrics_getProcessMetrics+0x73
J 28040 oracle.forms.servlet.ProcessMetrics.getProcessMetrics(I)[J (0 bytes) @ 0x00007fed9a937ff0 [0x00007fed9a937f40+0xb0]
J 30283 C2 oracle.forms.servlet.ProcessMetrics.<init>(I)V (384 bytes) @ 0x00007fed9df700a0 [0x00007fed9df6f160+0xf40]
J 30314 C2 oracle.forms.servlet.RunformProcess.startProcess(Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;)V (390 bytes) @ 0x00007fed9dfa8b60 [0x00007fed9dfa6ec0+0x1ca0]
J 28233 C1 oracle.forms.servlet.RunformProcess.<init>(Ljava/lang/String;[Ljava/lang/String;Ljava/lang/String;Ljava/util/logging/Logger;Ljavax/servlet/ServletContext;ILjava/lang/String;Z)V (502 bytes) @ 0x00007fed9991a4c4 [0x00007fed99918520+0x1fa4]
J 29686 C2 oracle.forms.servlet.RunformSession.<init>(Ljava/lang/String;[Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/ServletContext;Ljava/util/logging/Logger;I)V (581 bytes) @ 0x00007fed9dcf5c44 [0x00007fed9dcf4f60+0xce4]
J 25304 C2 oracle.forms.servlet.ListenerServlet$Request.getRunformSession(Z)V (366 bytes) @ 0x00007fed9d57be70 [0x00007fed9d57b240+0xc30]
J 29567 C2 oracle.forms.servlet.ListenerServlet.doGet(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V (15 bytes) @ 0x00007fed9de68f20 [0x00007fed9de68300+0xc20]
J 25194 C2 javax.servlet.http.HttpServlet.service(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V (251 bytes) @ 0x00007fed9c59a494 [0x00007fed9c599e40+0x654]
J 25216 C2 javax.servlet.http.HttpServlet.service(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V (43 bytes) @ 0x00007fed9c131638 [0x00007fed9c1315e0+0x58]
J 29343 C2 weblogic.servlet.internal.ServletStubImpl.execute(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;Lweblogic/servlet/internal/FilterChainImpl;)V (679 bytes) @ 0x00007fed9b1392c0 [0x00007fed9b138f00+0x3c0]
J 25322 C2 weblogic.servlet.internal.TailFilter.doFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;Ljavax/servlet/FilterChain;)V (121 bytes) @ 0x00007fed9c4e6a7c [0x00007fed9c4e6a00+0x7c]
J 25321 C2 oracle.security.jps.ee.http.JpsAbsFilter$1.run()Ljava/lang/Object; (132 bytes) @ 0x00007fed9d79feec [0x00007fed9d79fd40+0x1ac]
v ~StubRoutines::call_stub
V [libjvm.so+0x68c656] JavaCalls::call_helper(JavaValue*, methodHandle*, JavaCallArguments*, Thread*)+0x1056
V [libjvm.so+0x72531c] JVM_DoPrivileged+0x27c
J 330 java.security.AccessController.doPrivileged(Ljava/security/PrivilegedExceptionAction;Ljava/security/AccessControlContext;)Ljava/lang/Object; (0 bytes) @ 0x00007fed98fa5915 [0x00007fed98fa5840+0xd5]
J 25336 C2 oracle.security.jps.ee.http.JpsAbsFilter.doFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;Ljavax/servlet/FilterChain;)V (655 bytes) @ 0x00007fed9d7be144 [0x00007fed9d7bd540+0xc04]
J 25282 C2 oracle.security.jps.ee.http.JpsFilter.doFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;Ljavax/servlet/FilterChain;)V (59 bytes) @ 0x00007fed9c42f99c [0x00007fed9c42f940+0x5c]
J 25104 C2 oracle.dms.servlet.DMSServletFilter.doFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;Ljavax/servlet/FilterChain;)V (452 bytes) @ 0x00007fed9d7201a0 [0x00007fed9d71ef60+0x1240]
J 25297 C2 weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.wrapRun(Lweblogic/servlet/internal/ServletStub;Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)Ljava/lang/Object; (571 bytes) @ 0x00007fed9d6b393c [0x00007fed9d6b2780+0x11bc]
J 25212 C2 weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run()Ljava/lang/Object; (17 bytes) @ 0x00007fed9c529b64 [0x00007fed9c529b20+0x44]
J 23011 C2 weblogic.security.acl.internal.AuthenticatedSubject.doAs(Lweblogic/security/subject/AbstractSubject;Ljava/security/PrivilegedAction;)Ljava/lang/Object; (106 bytes) @ 0x00007fed9cec7fa4 [0x00007fed9cec7ca0+0x304]
J 29344 C2 weblogic.servlet.internal.WebAppServletContext.doSecuredExecute(Lweblogic/servlet/internal/ServletInvocationContext;Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;ZZZ)V (661 bytes) @ 0x00007fed9cdc61d0 [0x00007fed9cdc5ba0+0x630]
J 25332 C2 weblogic.servlet.internal.WebAppServletContext.execute(Lweblogic/servlet/internal/ServletRequestImpl;Lweblogic/servlet/internal/ServletResponseImpl;)V (568 bytes) @ 0x00007fed9a04630c [0x00007fed9a045ec0+0x44c]
J 25324 C2 weblogic.servlet.internal.ServletRequestImpl.runInternal()V (550 bytes) @ 0x00007fed9c7b28a8 [0x00007fed9c7b2100+0x7a8]
J 25292 C2 weblogic.servlet.provider.ContainerSupportProviderImpl$WlsRequestExecutor.run()V (11 bytes) @ 0x00007fed9b441d70 [0x00007fed9b441bc0+0x1b0]
J 20034 C2 weblogic.work.ExecuteThread.execute(Lweblogic/work/WorkAdapter;)V (267 bytes) @ 0x00007fed9c5fce80 [0x00007fed9c5fcc40+0x240]
J 22906% C2 weblogic.work.ExecuteThread.run()V (192 bytes) @ 0x00007fed9ce35a74 [0x00007fed9ce35760+0x314]
v ~StubRoutines::call_stub
V [libjvm.so+0x68c656] JavaCalls::call_helper(JavaValue*, methodHandle*, JavaCallArguments*, Thread*)+0x1056
V [libjvm.so+0x68cb61] JavaCalls::call_virtual(JavaValue*, KlassHandle, Symbol*, Symbol*, JavaCallArguments*, Thread*)+0x321
V [libjvm.so+0x68d007] JavaCalls::call_virtual(JavaValue*, Handle, KlassHandle, Symbol*, Symbol*, Thread*)+0x47
V [libjvm.so+0x723dc0] thread_entry(JavaThread*, Thread*)+0xa0
V [libjvm.so+0xa69cff] JavaThread::thread_main_inner()+0xdf
V [libjvm.so+0xa69e2c] JavaThread::run()+0x11c
V [libjvm.so+0x91d918] java_start(Thread*)+0x108
C [libpthread.so.0+0x7aa1] start_thread+0xd1