Out Of memory in prod

mponamgiAug 31 2013

One of the prod clusters is running on below crashed both nodes with an out of memory exception which looks like below.

How do we get to know if the crash is due to a memory leak in the code or elsewhere?

Any inputs which helps to identify on the issue and submit data to application team supporting the need to update the systems is appreciated.

RHEL 4.6 on a 32 bit OS

BEA JRockit(R) R27.5.0

Below is the dump file generated when the JVM was crashed.

####################################################################################################

===== BEGIN DUMP =============================================================

JRockit dump produced after 21 days, 21:54:48 on Thu Aug 29 23:22:36 2013

**********************************************************

* If you see this dump, please go to *

* http://edocs.bea.com/jrockit/go2troubleshooting.html *

* for troubleshooting information. *

**********************************************************

Additional information is available in:

/wls_domains/prod_slimseu_wli_domain/jrockit.20717.dump

No snapshot file (core dump) will be created because core dumps have been

disabled. To enable core dumping, try "ulimit -c unlimited"

before starting JRockit again.

Error Message: Out of memory [68]

Signal info : si_signo=11, si_code=2 si_addr=(nil)

Fatal Error : Lock Iteration lockTokenIterInit src/jvm/code/runtime/refiter.c:233

Version : BEA JRockit(R) R27.5.0-110_o-99226-1.5.0_14-20080528-1505-linux-ia32

GC Strategy : Mode: throughput. Currently using strategy: genparpar

GC Status : OC is not running. Last finished OC was OC#6379.

: YC is not running. Last finished YC was YC#46347.

OC History : Strategy genparpar was used for OC#5680 to OC#5848.

: Strategy singleparpar was used for OC#5849.

: Strategy genparpar was used for OC#5850 to OC#6034.

: Strategy singleparpar was used for OC#6035.

: Strategy genparpar was used for OC#6036 to OC#6379.

YC History : Ran 7 YCs before OC#6375.

: Ran 6 YCs before OC#6376.

: Ran 4 YCs before OC#6377.

: Ran 7 YCs before OC#6378.

: Ran 5 YCs before OC#6379.

: Ran 0 YCs since last OC.

YC Promotion : Last YC successfully promoted all objects

Heap : 0x8100000 - 0x88100000 (Size: 2048 MB)

Compaction : 0x80100000 - 0x88100000 (Current compaction type: external)

NurseryList : 0x6d634138 - 0x798afad0

KeepArea : 0x79218a20 - 0x7c8122f0

NurseryMarker: [ 0x79218a20, 0x7c8122f0 ]

CompRefs : References are 32-bit.

CPU : Intel Core 2 SSE SSE2 SSE3 SSSE3 SSE4.1 EM64T

Number CPUs : 4

Tot Phys Mem : 21292417024 (20306 MB)

OS version : Red Hat Enterprise Linux ES release 4 (Nahant Update 6)

Linux version 2.6.9-67.ELsmp (brewbuilder@ls20-bc1-14.build.redhat.com) (gcc version 3.4.6 20060404 (Red Hat 3.4.6-8)) #1 SMP Wed Nov 7 13:58:04 EST 2007 (i686)

Thread System: NPTL

Java locking : Normal

State : JVM is running

Command Line : -Xms2048m -Xmx2048m -Xss=256k -da -Dplatform.home=/opt/bea922/aldsp301/weblogic92 -Dwls.home=/opt/bea922/aldsp301/weblogic92/server -Dwli.home=/opt/bea922/aldsp301/weblogic92/integration -Denvironment=prod -Dlog4j.configuration=file:/wls_domains/prod_slimseu_wli_domain/apacheLog4jCfg.xml -Dweblogic.management.discover=false -Dweblogic.management.server=http://meermin.european.idexxi.com:30050 -Dwlw.iterativeDev=false -Dwlw.testConsole=false -Dwlw.logErrorsToConsole=true -Dweblogic.ext.dirs=/opt/bea922/aldsp301/patch_weblogic922/profiles/default/sysext_manifest_classpath -Xverbose:gcreport,memdbg,gcpause -Xverboselog:/wls_logs/prod_slimseu_wli_domain/prod_slimseu_wli.gc -Duser.timezone=GMT -Dweblogic.Domain=prod_slimseu_wli_domain -Dweblogic.Name=prod_slimseu_wli_02 -Djava.security.policy=/opt/bea922/aldsp301/weblogic92/server/lib/weblogic.policy -Dsun.java.launcher=SUN_STANDARD weblogic.Server

java.home : /opt/bea922/aldsp301/jrockit-R27.5.0/jre

j.class.path : /opt/bea922/aldsp301/aldsp_3.0/lib/binxml.jar:/opt/bea922/aldsp301/patch_weblogic922/profiles/default/sys_manifest_classpath/weblogic_patch.jar:/opt/bea922/aldsp301/jrockit-R27.5.0/lib/tools.jar:/opt/bea922/aldsp301/weblogic92/server/lib/weblogic_sp.jar:/opt/bea922/aldsp301/weblogic92/server/lib/weblogic.jar:/opt/bea922/aldsp301/weblogic92/server/lib/webservices.jar::/opt/bea922/aldsp301/weblogic92/integration/lib/worklist-system-required.jar:/opt/bea922/aldsp301/weblogic92/server/lib/jpd.jar:/opt/bea922/aldsp301/weblogic92/integration/L10N:/opt/bea922/aldsp301/weblogic92/common/eval/pointbase/lib/pbclient51.jar:/opt/bea922/aldsp301/weblogic92/server/lib/xqrl.jar::

j.lib.path : /opt/bea922/aldsp301/jrockit-R27.5.0/jre/lib/i386/jrockit:/opt/bea922/aldsp301/jrockit-R27.5.0/jre/lib/i386:/opt/bea922/aldsp301/jrockit-R27.5.0/jre/../lib/i386:/opt/bea922/aldsp301/patch_weblogic922/profiles/default/native:/opt/bea922/aldsp301/weblogic92/server/native/linux/i686:/opt/bea922/aldsp301/weblogic92/server/native/linux/i686/oci920_8

JAVA_HOME : /opt/bea922/aldsp301/jrockit-R27.5.0

_JAVA_OPTIONS: <not set>

LD_LIBRARY_PATH: /opt/bea922/aldsp301/jrockit-R27.5.0/jre/lib/i386/jrockit:/opt/bea922/aldsp301/jrockit-R27.5.0/jre/lib/i386:/opt/bea922/aldsp301/jrockit-R27.5.0/jre/../lib/i386:/opt/bea922/aldsp301/patch_weblogic922/profiles/default/native:/opt/bea922/aldsp301/weblogic92/server/native/linux/i686:/opt/bea922/aldsp301/weblogic92/server/native/linux/i686/oci920_8

LD_ASSUME_KERNEL: <not set>

C Heap : 1 memory allocations have failed

: First failure was a mmMalloc of 20 bytes

: Last failure was a mmMalloc of 20 bytes

StackOverFlow: 0 StackOverFlowErrors have occured

OutOfMemory : 3 OutOfMemoryErrors have occured

Registers (from ThreadContext: 0xa96d4e94 / OS context: 0xa96d4f90):

eax = 00001267 ecx = a96d520c edx = 00000043 ebx = b7f5d74c

esp = a96d5280 ebp = a96d5298 esi = 00000044 edi = a96d52e8

es = 0000007b cs = 00000073 ss = 0000007b ds = 0000007b

fs = 00000000 gs = 00000033

eip = b7dabd55 eflags = 00000282

Stack:

(* marks the word pointed to by the stack pointer)

a96d5280: b7f72020* 00000200 b7f5d74c a96d52e8 a96d52e8 a96d52a4

a96d5298: a96d52b8 b7f258c4 00000044 b7f5d74c a96d52e8 a96d5310

a96d52b0: a96d5358 a96d5360 a96d52d8 b7f258ef 00000044 b7f5d74c

a96d52c8: a96d52e8 a96d5360 a96d5310 a96d5358 a96d52f8 b7ecee0a

Code:

(* marks the word pointed to by the instruction pointer)

b7dabd24: 4c892074 458b0824 2404c710 b7f72020 0c244489 000200b8

b7dabd3c: 24448900 92fae804 01b80017 a3000000 b7f72004 001267b8

b7dabd54: 0000a300* 04c70000 00003f24 9aeae800 768d0017 27bc8d00

b7dabd6c: 00000000 e589fc55 53c03157 b9e87d8d 00000004 00c0ec81

Loaded modules:

(* denotes the module causing the exception)

08048000-08056ff3 /opt/bea922/aldsp301/jrockit-R27.5.0/bin/java

0093f000-0094c85b /lib/tls/libpthread.so.0

00833000-00853c8f /lib/tls/libm.so.6

00858000-00859967 /lib/libdl.so.2

00705000-0082a578 /lib/tls/libc.so.6

006eb000-0070039b /lib/ld-linux.so.2

b7d16000-b7f6c6a7 */opt/bea922/aldsp301/jrockit-R27.5.0/jre/lib/i386/jrockit/libjvm.so

00db2000-00db9a76 /lib/tls/librt.so.1

b7cdb000-b7cf9103 /lib/libnss_centrifydc.so.2

0099a000-009a8fef /lib/libresolv.so.2

00d76000-00d8866f /lib/libnsl.so.1

b7cd0000-b7cd8a57 /lib/libnss_files.so.2

b7be5000-b7bef81b /opt/bea922/aldsp301/jrockit-R27.5.0/jre/lib/i386/libverify.so

b7bc2000-b7be2317 /opt/bea922/aldsp301/jrockit-R27.5.0/jre/lib/i386/libjava.so

b6b1b000-b6b20f13 /opt/bea922/aldsp301/jrockit-R27.5.0/jre/lib/i386/native_threads/libhpi.so

b65ae000-b65bc4c4 /opt/bea922/aldsp301/jrockit-R27.5.0/jre/lib/i386/libzip.so

b4e71000-b4e82f03 /opt/bea922/aldsp301/jrockit-R27.5.0/jre/lib/i386/libnet.so

b4e69000-b4e6f207 /opt/bea922/aldsp301/jrockit-R27.5.0/jre/lib/i386/libnio.so

b5f03000-b5f06313 /lib/libnss_dns.so.2

b3a6e000-b3a73246 /opt/bea922/aldsp301/jrockit-R27.5.0/jre/lib/i386/libmanagement.so

b39e1000-b39ea357 /opt/bea922/aldsp301/jrockit-R27.5.0/jre/lib/i386/libjmapi.so

b36d8000-b36d9de4 /opt/bea922/aldsp301/weblogic92/server/native/linux/i686/libwlfileio2.so

a59ed000-a5a4c026 /opt/bea922/aldsp301/jrockit-R27.5.0/jre/lib/i386/libawt.so

a5926000-a59eb21f /opt/bea922/aldsp301/jrockit-R27.5.0/jre/lib/i386/libmlib_image.so

a5920000-a5924f1c /opt/bea922/aldsp301/jrockit-R27.5.0/jre/lib/i386/headless/libmawt.so

a53f8000-a54710a7 /opt/bea922/aldsp301/jrockit-R27.5.0/jre/lib/i386/libfontmanager.so

"[ACTIVE] ExecuteThread: '25' fo" id=75 idx=0x128 tid=20894 lastJavaFrame=0xa96d54cc

Stack 0: start=0xa9698000, end=0xa96da000, guards=0xa969d000 (ok), forbidden=0xa969b000

Thread Stack Trace:

at dumpForceDump+117()@0xb7dabd55

at vmFatalErrorMsgV+84()@0xb7f258c4

at vmFatalErrorMsg+31()@0xb7f258ef

at fatalError+42()@0xb7ecee0a

at lockTokenIterInit+151()@0xb7ecf317

at trProcessLocksForThread+102()@0xb7edda26

at get_all_locks+106()@0xb7df61ca

at javalockConvertTokenForObject+74()@0xb7df83aa

-- Java stack --

at jrockit/vm/Locks.convertThinLockedToFatLocked(Ljava/lang/Object;)V(Native Method)

at jrockit/vm/Locks.createMonitorAndConvert(Locks.java:1521)[inlined]

at jrockit/vm/Locks.wait(Locks.java:2184)[inlined]

at weblogic/rjvm/ResponseImpl.waitForData(ResponseImpl.java:81)[optimized]

^-- Holding lock: weblogic/rjvm/ResponseImpl@0x6daebaa8[thin lock]

at weblogic/rjvm/ResponseImpl.getTxContext(ResponseImpl.java:108)[inlined]

at weblogic/rjvm/BasicOutboundRequest.sendReceive(BasicOutboundRequest.java:109)[optimized]

at weblogic/rmi/cluster/ClusterableRemoteRef.invoke(ClusterableRemoteRef.java:335)[inlined]

at weblogic/rmi/cluster/ClusterableRemoteRef.invoke(ClusterableRemoteRef.java:252)[optimized]

at com/bea/ld/Server_ydm4ie_EOImpl_1030_WLStub.executeOperationStreaming(Lcom/bea/dsp/ejb/QualName;[Ljava/lang/Object;Lcom/bea/ld/QueryAttributes;Lcom/bea/ld/filter/FilterXQuery;)Lcom/bea/dsp/ejb/StreamingQueryResult;(Unknown Source)[optimized]

at com/bea/dsp/das/ejb/EJBClient.invokeOperation(EJBClient.java:148)[optimized]

at com/bea/dsp/das/DataAccessServiceImpl.invokeOperation(DataAccessServiceImpl.java:171)[inlined]

at com/bea/dsp/das/DataAccessServiceImpl.invoke(DataAccessServiceImpl.java:122)[inlined]

at com/bea/dsp/control/core/delegates/impl/DSPControlInvocationDelegate.executeFunction(DSPControlInvocationDelegate.java:219)[inlined]

at com/bea/dsp/control/core/delegates/impl/DSPControlInvocationDelegate.invokeOnDataService(DSPControlInvocationDelegate.java:64)[optimized]

at com/bea/dsp/control/core/DSPControlImpl.invoke(DSPControlImpl.java:81)[optimized]

at com/idexx/lims/controls/dsp/ResultsReportDSPControlFileBean.getPathEventTimeStampForDepartment(ResultsReportDSPControlFileBean.java:267)[optimized]

at com/idexx/lims/orderresults/dao/OrderResultsDAOImpl.getPathEventTimeStampForDepartment(OrderResultsDAOImpl.java:109)[optimized]

at com/idexx/lims/controls/OrderResultsCustomControlImpl.releaseResultsAndSaveReport(OrderResultsCustomControlImpl.java:360)[optimized]

at com/idexx/lims/controls/OrderResultsCustomControlBean.releaseResultsAndSaveReport(OrderResultsCustomControlBean.java:440)[optimized]

at sun/reflect/GeneratedMethodAccessor831.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;(Unknown Source)[optimized]

at sun/reflect/DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)[optimized]

at java/lang/reflect/Method.invoke(Method.java:585)[optimized]

at com/bea/wli/knex/runtime/jcs/container/JcsProxy.invokeBeehiveControl(JcsProxy.java:666)[optimized]

at com/bea/wli/knex/runtime/jcs/container/JcsProxy.invoke(JcsProxy.java:433)[optimized]

at $Proxy163.releaseResultsAndSaveReport(LorderResults/logical/orderResultsWs/ReleaseResultsAndSaveReportDocumentXmlBean;)LorderResults/logical/orderResultsWs/ReleaseResultsAndSaveReportResponseDocumentXmlBean;(Unknown Source)

at com/idexx/lims/controls/process/OrderResultsProcess.orderResultsCustomControlReleaseResultsAndSaveReport(OrderResultsProcess.java:649)

at sun/reflect/GeneratedMethodAccessor830.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;(Unknown Source)

at sun/reflect/DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)[optimized]

at java/lang/reflect/Method.invoke(Method.java:585)[optimized]

at com/bea/wli/bpm/runtime/Perform.invoke(Perform.java:39)[inlined]

at com/bea/wli/bpm/runtime/Perform.execute(Perform.java:50)[optimized]

at com/bea/wli/bpm/runtime/SyncReceive.messageDelivery(SyncReceive.java:63)[optimized]

at com/bea/wli/bpm/runtime/ProcessState.processMessage(ProcessState.java:217)[optimized]

at com/bea/wli/bpm/runtime/ProcessState.dispatchRequest(ProcessState.java:241)[inlined]

at com/bea/wli/bpm/runtime/JpdContainer.dispatchProcessRequest(JpdContainer.java:1077)[optimized]

at com/bea/wli/bpm/runtime/JpdContainer.preInvoke(JpdContainer.java:1041)[optimized]

at com/bea/wli/knex/runtime/core/container/Invocable.invoke(Invocable.java:248)[optimized]

at com/bea/wli/bpm/runtime/JpdContainer.invoke(JpdContainer.java:814)[optimized]

at com/bea/wli/knex/runtime/core/bean/BaseContainerBean.invokeBase(BaseContainerBean.java:224)[optimized]

at com/bea/wli/knex/runtime/core/bean/SLSBContainerBean.invoke(SLSBContainerBean.java:136)[optimized]

at com/bea/wlwgen/StatelessContainer_2hozgx_ELOImpl.invoke(StatelessContainer_2hozgx_ELOImpl.java:137)[optimized]

at com/bea/wlwgen/SLSBContAdpt.invokeOnBean(SLSBContAdpt.java:29)[optimized]

at com/bea/wli/knex/runtime/core/bean/BaseDispatcherBean.runAsInvoke(BaseDispatcherBean.java:185)[optimized]

at com/bea/wli/knex/runtime/core/bean/BaseDispatcherBean.invoke(BaseDispatcherBean.java:54)[optimized]

at com/bea/wli/knex/runtime/core/bean/SyncDispatcherBean.invoke(SyncDispatcherBean.java:168)[optimized]

at com/bea/wli/knex/runtime/core/bean/SyncDispatcher_k1mrl8_EOImpl.invoke(SyncDispatcher_k1mrl8_EOImpl.java:133)[optimized]

at com/bea/wli/knex/runtime/core/dispatcher/Dispatcher.remoteDispatch(Dispatcher.java:165)[inlined]

at com/bea/wli/knex/runtime/core/dispatcher/ServiceHandleImpl.invoke(ServiceHandleImpl.java:460)[optimized]

at com/bea/wli/knex/runtime/core/call/JavaCall.invoke(JavaCall.java:56)[optimized]

at com/bea/wli/bpm/runtime/SubFlowCall.invoke(SubFlowCall.java:112)[optimized]

at com/bea/wli/knex/runtime/core/control/ServiceControlImpl.invoke(ServiceControlImpl.java:1307)[inlined]

at com/bea/control/ProcessControlImpl.invoke(ProcessControlImpl.java:825)[optimized]

at com/bea/wli/knex/runtime/core/control/ServiceControlImpl.invoke(ServiceControlImpl.java:1174)[optimized]

at com/bea/control/ProcessControlImpl.invoke(ProcessControlImpl.java:760)[optimized]

at com/idexx/lims/controls/process/OrderResultsProcessControlBean.releaseResultsClientRequestwithReturn(OrderResultsProcessControlBean.java:394)

at sun/reflect/GeneratedMethodAccessor828.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;(Unknown Source)

at sun/reflect/DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)[optimized]

at java/lang/reflect/Method.invoke(Method.java:585)[optimized]

at com/bea/wli/knex/runtime/jcs/container/JcsProxy.invokeBeehiveControl(JcsProxy.java:666)[optimized]

at com/bea/wli/knex/runtime/jcs/container/JcsProxy.invoke(JcsProxy.java:446)[optimized]

at $Proxy171.releaseResultsClientRequestwithReturn(LorderResults/logical/orderResultsWs/ReleaseResultsAndSaveReportDocumentXmlBean;)LorderResults/logical/orderResultsWs/ReleaseResultsAndSaveReportResponseDocumentXmlBean;(Unknown Source)

at com/idexx/lims/controls/process/OrderResultsProcess.orderResultsProcessControlReleaseResultsClientRequestwithReturn111(OrderResultsProcess.java:1686)[optimized]

at sun/reflect/GeneratedMethodAccessor827.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;(Unknown Source)