Hi,
We are facing problem with one of the Critical application which is hosted on Weblogic10 and had a crash this month with the <java.lang.OutOfMemoryError: Java heap space> issue. I am not able to figure the root cause why this error was produced, but this application handle processing of very huge files and we recently migrated from WL9.1 to WL10, so not sure if that caused the problem to appear.
Below is the application config details
-------------------------------------------------------
Weblogic10
EJB Component Deployed on a cluster of 2 Nodes in Weblogic
WL Server Java VM Params
Server1 = -XX:MaxPermSize=1024m -Xms2048m -Xmx2048m
Server2 = -XX:MaxPermSize=1024m -Xms2048m -Xmx2048m
Pls refer below the errors reported on the 2 WL servers which host our application, Looking at the error it appears the server is running out of heap space on the first server, so I think the GC is spending too much time trying to free up memory or the server1 not allowing any more threads.
I am really not able to understand the Server2 logs, I am seeing some stuck error messages and not sure if it is also the result of Java Heap Space issue before the server was restarted.
Appreciate any suggestions or advice to get rid of the <java.lang.OutOfMemoryError: Java heap space> issue permanently.
Server1 Logs
-------------------
####<Jul 2, 2009 10:50:36 PM GMT> <Notice> <StdErr> <g1u0309c.austin.hp.com> <te1> <[STUCK] ExecuteThread: '19' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1246575036324> <BEA-000000> <***************************************************************************>
####<Jul 2, 2009 10:50:36 PM GMT> <Notice> <StdErr> <g1u0309c.austin.hp.com> <te1> <[STUCK] ExecuteThread: '19' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1246575036324> <BEA-000000> <java.lang.OutOfMemoryError: Java heap space>
####<Jul 2, 2009 10:50:36 PM GMT> <Notice> <StdErr> <g1u0309c.austin.hp.com> <te1> <[STUCK] ExecuteThread: '19' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1246575036324> <BEA-000000> <Reason: There is a panic condition in the server. The server is configured to exit on panic>
####<Jul 2, 2009 10:50:36 PM GMT> <Notice> <StdErr> <g1u0309c.austin.hp.com> <te1> <[STUCK] ExecuteThread: '19' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1246575036324> <BEA-000000> <***************************************************************************>
####<Jul 2, 2009 10:50:36 PM GMT> <Notice> <StdErr> <g1u0309c.austin.hp.com> <te1> <[STUCK] ExecuteThread: '19' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1246575036324> <BEA-000000> <The WebLogic Server encountered a critical failure>
####<Jul 2, 2009 10:50:36 PM GMT> <Warning> <RMI> <g1u0309c.austin.hp.com> <te1> <[STUCK] ExecuteThread: '9' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1246575036325> <BEA-080004> <An error was thrown by rmi server: weblogic.corba.cos.naming.RootNamingContextImpl.resolve_any([Lweblogic.corba.cos.naming.NamingContextAnyPackage.WNameComponent;)
java.lang.OutOfMemoryError: Java heap space.
java.lang.OutOfMemoryError: Java heap space
>
Server2 Logs
-------------------
####<Jul 2, 2009 10:35:13 PM GMT> <Error> <WebLogicServer> <g2u0754.austin.hp.com> <te2> <[ACTIVE] ExecuteThread: '13' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1246574113896> <BEA-000337> <[STUCK] ExecuteThread: '19' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "701" seconds working on the request "weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl@18b117f", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
weblogic.timers.internal.TimerManagerImpl.complete(TimerManagerImpl.java:652)
weblogic.timers.internal.TimerImpl.run(TimerImpl.java:285)
weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:464)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:200)
weblogic.work.ExecuteThread.run(ExecuteThread.java:172)
>
####<Jul 2, 2009 10:35:13 PM GMT> <Info> <WebLogicServer> <g2u0754.austin.hp.com> <te2> <[STUCK] ExecuteThread: '19' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1246574113911> <BEA-000339> <[STUCK] ExecuteThread: '19' for queue: 'weblogic.kernel.Default (self-tuning)' has become "unstuck".>
####<Jul 2, 2009 10:40:41 PM GMT> <Notice> <StdErr> <g2u0754.austin.hp.com> <te2> <[STUCK] ExecuteThread: '17' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1246574441307> <BEA-000000> <Failed to create lookup object, Exception =javax.naming.CommunicationException [Root exception is weblogic.rjvm.PeerGoneException: No message was received for: '240' seconds]>
####<Jul 2, 2009 10:40:41 PM GMT> <Notice> <StdErr> <g2u0754.austin.hp.com> <te2> <[STUCK] ExecuteThread: '17' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1246574441307> <BEA-000000> <Could not create home, e=java.lang.NullPointerException: narrowFrom'null' narrowTo:'interface sard.ua.db.dbAcc.interfaces.DBAccHome'It is invalid to call narrow with null parameters>
####<Jul 2, 2009 10:40:41 PM GMT> <Notice> <Stdout> <g2u0754.austin.hp.com> <te2> <[STUCK] ExecuteThread: '17' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1246574441307> <BEA-000000> <Failed: home.create(), e=java.lang.NullPointerException>
####<Jul 2, 2009 10:56:42 PM GMT> <Notice> <WebLogicServer> <g2u0754.austin.hp.com> <> <main> <> <> <> <1246575402360> <BEA-000395> <Following extensions directory contents added to the end of the classpath:
/opt/bea/wls10/weblogic10/platform/lib/p13n/p13n-schemas.jar:/opt/bea/wls10/weblogic10/platform/lib/p13n/p13n_common.jar:/opt/bea/wls10/weblogic10/platform/lib/p13n/p13n_system.jar:/opt/bea/wls10/weblogic10/platform/lib/wlp/netuix_common.jar:/opt/bea/wls10/weblogic10/platform/lib/wlp/netuix_schemas.jar:/opt/bea/wls10/weblogic10/platform/lib/wlp/netuix_system.jar:/opt/bea/wls10/weblogic10/platform/lib/wlp/wsrp-client.jar:/opt/bea/wls10/weblogic10/platform/lib/wlp/wsrp-common.jar>
####<Jul 2, 2009 10:56:42 PM GMT> <Info> <WebLogicServer> <g2u0754.austin.hp.com> <> <main> <> <> <> <1246575402821> <BEA-000377> <Starting WebLogic Server with Java HotSpot(TM) Server VM Version 1.5.0.10 jinteg:10.01.07-10:12 IA64 from Hewlett-Packard Company>
####<Jul 2, 2009 10:56:44 PM GMT> <Info> <Management> <g2u0754.austin.hp.com> <> <main> <> <> <> <1246575404064> <BEA-141107> <Version: WebLogic Server Temporary Patch for CR370091 Tue Jul 08 17:31:21 EDT 2008
WebLogic Server Temporary Patch for CR345742 Fri Apr 04 19:16:52 IST 2008
WebLogic Server 10.0 MP1 Thu Oct 18 20:17:44 EDT 2007 1005184 >
With Regards
M.MuthuRekha