Application Development Software

Hi,

I am facing this issue everyday atlease once JVM crashed. We do not know where the problem occurs. TIBCO process calls inside Java Method, this method will use CM SDK API and try to publish / retrieve from Oracle. The tibco process stops running status with below dump. We have to force start manually. I am sure the exception thrown not from application related exception thrown. The same set of code working production environment without any issue. It occurs in development environment.

Do we required to install any patch set of OS / CM SDK / JRE / Increase Memory / Heap Size ?

Appreciated If you can help us !!

Dev Env Details
============
TIBCO process running in Widows 2003 Server R2 Service Pack 1
4GB RAM
3GHZ CPU speed
Tibco Heap size allocated 1GB
Java Heap Size allocated 256MB

CM SDK (ifs service) running / located in IBM AIX Server

CM SDK Jar Version: (Already copied from Metalink of below)
08/09/2005 02:14 AM 9,923,914 cmsdk_linux.jar
08/25/2005 10:41 AM 9,517,469 cmsdk_windows.jar
updated patch set CMSDK 9.0.4.2.2

1. DB version
Ans. SQL> show parameter db_name
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
db_name string ts00mir
SQL> select * from v$version;
BANNER
----------------------------------------------------------------
Oracle Database 10g Enterprise Edition Release 10.2.0.3.0 - 64bi
PL/SQL Release 10.2.0.3.0 - Production
CORE 10.2.0.3.0 Production
TNS for IBM/AIX RISC System/6000: Version 10.2.0.3.0 - Productio
NLSRTL Version 10.2.0.3.0 – Production

2. CMSDK version on the Linux box
Ans. Version 9.0.4.2.2 (Oracle Application Server 10g Release 2 (10.1.2.0.2))

CMSDK is not running on Windows / TIBCO deployed process place.

DUMP Info
========
#
# An unexpected error has been detected by HotSpot Virtual Machine:
#
# EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x00000000, pid=10056, tid=5324
#
# Java VM: Java HotSpot(TM) Server VM (1.5.0_11-b03 mixed mode)
# Problematic frame:
# C 0x00000000
#

--------------- T H R E A D ---------------

Current thread (0x00baf4c0): JavaThread "JobCourier2" [_thread_in_native, id=5324]

siginfo: ExceptionCode=0xc0000005, reading address 0x00000000

Registers:
EAX=0x00dc1684, EBX=0x00dc3458, ECX=0x50002da1, EDX=0x6051149d
ESP=0x1b09a8f0, EBP=0x1b09b15c, ESI=0x00dc1684, EDI=0x18c566b8
EIP=0x00000000, EFLAGS=0x00000283

Top of Stack: (sp=0x1b09a8f0)
0x1b09a8f0: 60510e31 00dc1684 00dc3458 00000000
0x1b09a900: 18a62220 00000058 00000000 00000000
0x1b09a910: 00000000 605af35c 18a61e18 1b09b1a0
0x1b09a920: 60510a6e 18ac752c 18a60044 18a62220
0x1b09a930: 00000058 000000cf 00000000 00000000
0x1b09a940: 00000000 00000049 19bd251c 00000000
0x1b09a950: 00000000 000000af 1b09aaa8 6054d3ed
0x1b09a960: 00000049 00000092 00000049 00000000

Instructions: (pc=0x00000000)
0xfffffff0:


Stack: [0x1afa0000,0x1b0a0000), sp=0x1b09a8f0, free space=1002k
Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
j oracle.jdbc.driver.T2CPreparedStatement.t2cDefineExecuteFetch(JIIIIZZ[BIBII[SI[B[CII[S[BIIZZ[Loracle/jdbc/driver/Accessor;[[[B[J[BI[CI[SI)I+0
j oracle.jdbc.driver.T2CPreparedStatement.doDefineExecuteFetch()V+182
j oracle.jdbc.driver.T2CPreparedStatement.execute_for_rows(Z)V+34
j oracle.jdbc.driver.OracleStatement.execute_maybe_describe()V+153
j oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout()V+146
v ~C2IAdapter
J oracle.jdbc.driver.OraclePreparedStatement.executeQuery()Ljava/sql/ResultSet;
v ~I2CAdapter
j oracle.ifs.server.S_LibraryServiceRdbmsEventExchanger.processEventsFromOtherServices()V+176
v ~C2IAdapter
J oracle.ifs.server.S_LibraryServiceRdbmsEventExchanger.tick()V
v ~I2CAdapter
j oracle.ifs.server.S_LibrarySession.DMSetCallbackObject(Loracle/ifs/common/SessionCallback;)V+27
j oracle.ifs.beans.LibrarySession.DMSetCallbackObject(Loracle/ifs/common/SessionCallback;)V+40
j oracle.ifs.beans.LibrarySession.<init>(Loracle/ifs/server/S_LibrarySessionInterface;Loracle/ifs/common/Credential;Loracle/ifs/common/ConnectOptions;Ljava/lang/String;)V+315
j oracle.ifs.beans.LibraryService.connect(Loracle/ifs/common/Credential;Loracle/ifs/common/ConnectOptions;)Loracle/ifs/beans/LibrarySession;+66
j com.mnm.mic.utils.IFSConnection.connect(Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;)Loracle/ifs/beans/LibrarySession;+50
j com.mnm.mic.utils.TIBCOUtils.tibIFSConnect()Loracle/ifs/beans/LibrarySession;+30
j com.mnm.mic.micFolder.tibReponseList(Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;ILjava/util/Date;Ljava/util/Date;)Lcom/mnm/mic/data/RetrieveListResponse;+3
j sun.reflect.GeneratedMethodAccessor30.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;+154
v ~C2IAdapter
J sun.reflect.DelegatingMethodAccessorImpl.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;
J java.lang.reflect.Method.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;
v ~I2CAdapter
j com.tibco.plugin.java.JavaMethodActivity.for(Lcom/tibco/pe/plugin/ProcessContext;Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;+45
j com.tibco.plugin.java.JavaMethodActivity.eval(Lcom/tibco/pe/plugin/ProcessContext;Lcom/tibco/xml/datamodel/XiNode;)Lcom/tibco/xml/datamodel/XiNode;+97
j com.tibco.pe.plugin.Activity.eval(Lcom/tibco/pe/plugin/ProcessContext;Lcom/tibco/xml/xdata/InputData;)Lcom/tibco/xml/datamodel/XiNode;+22
j com.tibco.pe.core.TaskImpl.eval(Lcom/tibco/pe/plugin/ProcessContext;)Ljava/lang/String;+1177
j com.tibco.pe.core.Job.a(J)J+1211
j com.tibco.pe.core.Job.k()J+23
j com.tibco.pe.core.JobDispatcher$JobCourier.a(Lcom/tibco/pe/core/Job;)Z+4
j com.tibco.pe.core.JobDispatcher$JobCourier.run()V+184
v ~StubRoutines::call_stub

--------------- P R O C E S S ---------------

Java Threads: ( => current thread )
0x1bef0da0 JavaThread "OracleTimeoutPollingThread" daemon [_thread_blocked, id=5628]
0x1beece70 JavaThread "Thread-27" daemon [_thread_blocked, id=9528]
0x00bede70 JavaThread "Thread-19" [_thread_blocked, id=7392]
0x00bec030 JavaThread "AMI Worker 1" [_thread_blocked, id=9256]
0x00bece70 JavaThread "HawkEngineQueueHandler" [_thread_in_native, id=9000]
0x00bdecf0 JavaThread "TIBCO EMS TCPLink Reader (3693977)" daemon [_thread_in_native, id=8212]
0x00bd2ba0 JavaThread "TIBCO EMS TCPLink Reader (3693968)" daemon [_thread_in_native, id=6264]
0x00bd0060 JavaThread "TIBCO EMS Connections Pinger" daemon [_thread_blocked, id=3492]
0x00bb4b80 JavaThread "WaitNotify Cleanup Sweep" daemon [_thread_blocked, id=7116]
0x00bb3880 JavaThread "JobCourier7" [_thread_blocked, id=3664]
0x00bb2480 JavaThread "JobCourier6" [_thread_blocked, id=8752]
0x00bb1080 JavaThread "JobCourier5" [_thread_blocked, id=3608]
0x00bb1c60 JavaThread "JobCourier4" [_thread_blocked, id=8352]
0x00bb0860 JavaThread "JobCourier3" [_thread_blocked, id=1672]
=>0x00baf4c0 JavaThread "JobCourier2" [_thread_in_native, id=5324]
0x00baf7e0 JavaThread "JobCourier1" [_thread_blocked, id=3576]
0x00bad900 JavaThread "JobCourier0" [_thread_blocked, id=4612]
0x00bae200 JavaThread "EngineReminderQueue" [_thread_blocked, id=8988]
0x00b069c0 JavaThread "Low Memory Detector" daemon [_thread_blocked, id=8960]
0x00b05d50 JavaThread "CompilerThread1" daemon [_thread_blocked, id=6644]
0x00b04c30 JavaThread "CompilerThread0" daemon [_thread_blocked, id=5936]
0x00b03c00 JavaThread "AdapterThread" daemon [_thread_blocked, id=6488]
0x00b00010 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=4280]
0x00afe090 JavaThread "Finalizer" daemon [_thread_blocked, id=8896]
0x00afe550 JavaThread "Reference Handler" daemon [_thread_blocked, id=7288]
0x00af34e0 JavaThread "main" [_thread_blocked, id=1332]

Other Threads:
0x00afd0d0 VMThread [id=976]
0x00b07570 WatcherThread [id=9268]

VM state:not at safepoint (normal execution)

VM Mutex/Monitor currently owned by a thread: None

Heap
def new generation total 7424K, used 2134K [0x03e50000, 0x04650000, 0x05ac0000)
eden space 6656K, 32% used [0x03e50000, 0x04065b48, 0x044d0000)
from space 768K, 0% used [0x04590000, 0x04590000, 0x04650000)
to space 768K, 0% used [0x044d0000, 0x044d0000, 0x04590000)
tenured generation total 65116K, used 19289K [0x05ac0000, 0x09a57000, 0x13e50000)
the space 65116K, 29% used [0x05ac0000, 0x06d96680, 0x06d96800, 0x09a57000)
compacting perm gen total 29440K, used 29413K [0x13e50000, 0x15b10000, 0x17e50000)
the space 29440K, 99% used [0x13e50000, 0x15b09690, 0x15b09800, 0x15b10000)
No shared spaces configured.

Dynamic libraries:
0x00400000 - 0x00427000 d:\TIBCO\bw\5.4\bin\bwengine.exe
0x7c800000 - 0x7c8c0000 C:\WINDOWS\system32\ntdll.dll
0x77e40000 - 0x77f42000 C:\WINDOWS\system32\kernel32.dll
0x77380000 - 0x77412000 C:\WINDOWS\system32\USER32.dll
0x77c00000 - 0x77c49000 C:\WINDOWS\system32\GDI32.dll
0x77f50000 - 0x77fec000 C:\WINDOWS\system32\ADVAPI32.dll
0x77c50000 - 0x77cf0000 C:\WINDOWS\system32\RPCRT4.dll
0x6d8e0000 - 0x6dc85000 d:\TIBCO\jre\1.5.0\bin\server\jvm.dll
0x76aa0000 - 0x76acd000 C:\WINDOWS\system32\WINMM.dll
0x77ba0000 - 0x77bfa000 C:\WINDOWS\system32\MSVCRT.dll
0x71bc0000 - 0x71bc8000 C:\WINDOWS\system32\rdpsnd.dll
0x771f0000 - 0x77201000 C:\WINDOWS\system32\WINSTA.dll
0x71c40000 - 0x71c98000 C:\WINDOWS\system32\NETAPI32.dll
0x76b70000 - 0x76b7b000 C:\WINDOWS\system32\PSAPI.DLL
0x6d300000 - 0x6d308000 d:\TIBCO\jre\1.5.0\bin\hpi.dll
0x6d710000 - 0x6d71c000 d:\TIBCO\jre\1.5.0\bin\verify.dll
0x6d380000 - 0x6d39d000 d:\TIBCO\jre\1.5.0\bin\java.dll
0x6d730000 - 0x6d73f000 d:\TIBCO\jre\1.5.0\bin\zip.dll
0x6d540000 - 0x6d553000 D:\TIBCO\jre\1.5.0\bin\net.dll
0x71c00000 - 0x71c17000 C:\WINDOWS\system32\WS2_32.dll
0x71bf0000 - 0x71bf8000 C:\WINDOWS\system32\WS2HELP.dll
0x71b20000 - 0x71b62000 C:\WINDOWS\System32\mswsock.dll
0x76ed0000 - 0x76eff000 C:\WINDOWS\system32\DNSAPI.dll
0x76f70000 - 0x76f77000 C:\WINDOWS\System32\winrnr.dll
0x76f10000 - 0x76f3e000 C:\WINDOWS\system32\WLDAP32.dll
0x76f80000 - 0x76f88000 C:\WINDOWS\system32\rasadhlp.dll
0x1a6c0000 - 0x1a6cc000 D:\TIBCO\tibrv\bin\tibrvjsd.dll
0x1a6d0000 - 0x1a6de000 d:\TIBCO\tibrv\bin\tibrvcmq.dll
0x1a6e0000 - 0x1a6fa000 d:\TIBCO\tibrv\bin\tibrvcm.dll
0x1a700000 - 0x1a75c000 d:\TIBCO\tibrv\bin\tibrv.dll
0x7c340000 - 0x7c396000 C:\WINDOWS\system32\MSVCR71.dll
0x1a760000 - 0x1a768000 d:\TIBCO\tibrv\bin\tibrvft.dll
0x1a770000 - 0x1a777000 d:\TIBCO\tibrv\bin\tibrvsd.dll
0x1a780000 - 0x1a7a6000 d:\TIBCO\tpcl\5.5\bin\SSLEAY32.dll
0x1a7b0000 - 0x1a883000 d:\TIBCO\tpcl\5.5\bin\LIBEAY32.dll
0x71bb0000 - 0x71bb9000 C:\WINDOWS\system32\WSOCK32.dll
0x5f270000 - 0x5f2c9000 C:\WINDOWS\system32\hnetcfg.dll
0x71ae0000 - 0x71ae8000 C:\WINDOWS\System32\wshtcpip.dll
0x68000000 - 0x6802f000 C:\WINDOWS\system32\rsaenh.dll
0x76920000 - 0x769e4000 C:\WINDOWS\system32\USERENV.dll
0x76f50000 - 0x76f63000 C:\WINDOWS\system32\Secur32.dll
0x62f00000 - 0x62f0f000 E:\Oracle\product\10.1.0\Client_1\BIN\ocijdbc10.dll
0x1bfa0000 - 0x1bfcf000 E:\Oracle\product\10.1.0\Client_1\bin\OCI.dll
0x60600000 - 0x607de000 E:\Oracle\product\10.1.0\Client_1\bin\OraClient10.Dll
0x610a0000 - 0x6114b000 E:\Oracle\product\10.1.0\Client_1\bin\oracore10.dll
0x612a0000 - 0x61332000 E:\Oracle\product\10.1.0\Client_1\bin\oranls10.dll
0x613a0000 - 0x613b3000 E:\Oracle\product\10.1.0\Client_1\bin\oraunls10.dll
0x60000000 - 0x60007000 E:\Oracle\product\10.1.0\Client_1\bin\orauts.dll
0x77670000 - 0x777a4000 C:\WINDOWS\system32\ole32.dll
0x60800000 - 0x60806000 E:\Oracle\product\10.1.0\Client_1\bin\oravsn10.dll
0x60500000 - 0x605b7000 E:\Oracle\product\10.1.0\Client_1\bin\oracommon10.dll
0x1bfd0000 - 0x1c3dc000 E:\Oracle\product\10.1.0\Client_1\bin\orageneric10.dll
0x61350000 - 0x61361000 E:\Oracle\product\10.1.0\Client_1\bin\orasnls10.dll
0x1c3e0000 - 0x1c4dd000 E:\Oracle\product\10.1.0\Client_1\bin\oraxml10.dll
0x60020000 - 0x60030000 C:\WINDOWS\system32\MSVCIRT.dll
0x61480000 - 0x6156a000 E:\Oracle\product\10.1.0\Client_1\bin\oran10.dll
0x61400000 - 0x6142c000 E:\Oracle\product\10.1.0\Client_1\bin\oranl10.dll
0x61960000 - 0x61972000 E:\Oracle\product\10.1.0\Client_1\bin\oranldap10.dll
0x615a0000 - 0x6168d000 E:\Oracle\product\10.1.0\Client_1\bin\orannzsbb10.dll
0x62000000 - 0x62044000 E:\Oracle\product\10.1.0\Client_1\bin\oraldapclnt10.dll
0x616b0000 - 0x616c7000 E:\Oracle\product\10.1.0\Client_1\bin\orancrypt10.dll
0x76cf0000 - 0x76d0a000 C:\WINDOWS\system32\iphlpapi.dll
0x77d00000 - 0x77d8c000 C:\WINDOWS\system32\OLEAUT32.dll
0x61730000 - 0x61767000 E:\Oracle\product\10.1.0\Client_1\bin\oranro10.dll
0x61910000 - 0x6191c000 E:\Oracle\product\10.1.0\Client_1\bin\orantcp10.dll
0x617c0000 - 0x617c6000 E:\Oracle\product\10.1.0\Client_1\bin\oranhost10.dll
0x616a0000 - 0x616a6000 E:\Oracle\product\10.1.0\Client_1\bin\orancds10.dll
0x61820000 - 0x61826000 E:\Oracle\product\10.1.0\Client_1\bin\orantns10.dll
0x60a00000 - 0x60cbc000 E:\Oracle\product\10.1.0\Client_1\bin\orapls10.dll
0x62500000 - 0x62507000 E:\Oracle\product\10.1.0\Client_1\bin\oraslax10.dll
0x60e00000 - 0x60f02000 E:\Oracle\product\10.1.0\Client_1\bin\oraplp10.dll
0x1c4e0000 - 0x1c531000 E:\Oracle\product\10.1.0\Client_1\bin\orahasgen10.dll
0x1c540000 - 0x1c573000 E:\Oracle\product\10.1.0\Client_1\bin\oraocr10.dll
0x1c580000 - 0x1c5a4000 E:\Oracle\product\10.1.0\Client_1\bin\oraocrb10.dll
0x62600000 - 0x62676000 E:\Oracle\product\10.1.0\Client_1\bin\orasql10.dll

VM Arguments:
jvm_args: -Djavax.xml.parsers.DocumentBuilderFactory=org.apache.xerces.jaxp.DocumentBuilderFactoryImpl -DpalettePath=d:/TIBCO/bw/5.4/lib/palettes;d:/TIBCO/bw/plugins/lib/palettes;d:/TIBCO/tra/5.5/lib/palettes -Dtibco_tra_version=5.5 -DTIBCO_DOMAIN_NAME=DEV06 -DjavaCode=d:/TIBCO/bw/5.4/lib/palettes -Dtibco_home=d:/TIBCO -DTIBCO_DOMAIN_HOME=d:/TIBCO/tra/5.5/../domain -DTIBCO_TRA_HOME=d:/TIBCO/tra/5.5 -Djavax.xml.parsers.SAXParserFactory=org.apache.xerces.jaxp.SAXParserFactoryImpl -Djava.library=d:/TIBCO/jre/1.5.0/bin/server/jvm.dll -Djava.library.path=d:/TIBCO/bw/5.4/hotfix/bin;d:/TIBCO/bw/5.4/bin;d:/TIBCO/tra/5.5/hotfix/bin;d:/TIBCO/tra/5.5/bin;d:/TIBCO/tra/5.5/hotfix/icjava/6.2/lib;d:/TIBCO/tra/5.5/icjava/6.2/lib;d:/TIBCO/tra/5.5/hotfix/icjava/6.2/bin;d:/TIBCO/tra/5.5/icjava/6.2/bin;d:/TIBCO/tpcl/5.5/bin;d:/TIBCO/tibrv/bin;d:/TIBCO/hawk/bin;E:\Oracle\product\10.1.0\Client_1\bin;E:\Oracle\product\10.1.0\Client_1\jre\1.4.2\bin\client;E:\Oracle\product\10.1.0\Client_1\jre\1.4.2\bin;C:\Perl\site\bin;C:\Perl\bin;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;D:\Tibco\tibrv\BIN;C:\Program Files\IBM\Director\bin;C:\Program Files\Common Files\IBM\ICC\cimom\binPSPORACLE_HOME%/bin -Xmx256M -Xss256K -Dapplication.processid=10056 -Dwrapper.tra.file=d:/TIBCO/tra/domain/DEV06/application/ES-MID-RECEIVINGSIDE/ES-MID-RECEIVINGSIDE-RetrieveListWS.tra
java_command: <unknown>
Launcher Type: generic

Environment Variables:
CLASSPATH=D:\Tibco\tibrv\LIB\tibrvj.jar
PATH=d:/TIBCO/bw/5.4/hotfix/bin;d:/TIBCO/bw/5.4/bin;d:/TIBCO/tra/5.5/hotfix/bin;d:/TIBCO/tra/5.5/bin;d:/TIBCO/tra/5.5/hotfix/icjava/6.2/lib;d:/TIBCO/tra/5.5/icjava/6.2/lib;d:/TIBCO/tra/5.5/hotfix/icjava/6.2/bin;d:/TIBCO/tra/5.5/icjava/6.2/bin;d:/TIBCO/tpcl/5.5/bin;d:/TIBCO/tibrv/bin;d:/TIBCO/hawk/bin;E:\Oracle\product\10.1.0\Client_1\bin;E:\Oracle\product\10.1.0\Client_1\jre\1.4.2\bin\client;E:\Oracle\product\10.1.0\Client_1\jre\1.4.2\bin;C:\Perl\site\bin;C:\Perl\bin;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;D:\Tibco\tibrv\BIN;C:\Program Files\IBM\Director\bin;C:\Program Files\Common Files\IBM\ICC\cimom\binPSPORACLE_HOME/bin
USERNAME=eif_admin_dev
LD_LIBRARY_PATH=d:/TIBCO/tpcl/5.5/lib/httpclient_3.0;d:/TIBCO/tra/5.5/hotfix/lib;d:/TIBCO/tra/5.5/lib;d:/TIBCO/tpcl/5.5/lib;d:/TIBCO/tibrv/lib;d:/TIBCO/jre/1.5.0/bin/server;d:/TIBCO/jre/1.5.0/bin;d:/TIBCO/tra/5.5/hotfix/icjava/6.2/lib;d:/TIBCO/tra/5.5/icjava/6.2/lib;;d:/TIBCO/tpcl/5.5/tomcat/5.5/compatible/lib
OS=Windows_NT
PROCESSOR_IDENTIFIER=x86 Family 15 Model 6 Stepping 8, GenuineIntel



--------------- S Y S T E M ---------------

OS: Windows Server 2003 family Build 3790 Service Pack 1

CPU:total 1 (cores per cpu 2, threads per core 2) family 15 model 6 stepping 8, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ht

Memory: 4k page, physical 2097151k(1849528k free), swap 4194303k(2947460k free)

vm_info: Java HotSpot(TM) Server VM (1.5.0_11-b03) for windows-x86, built on Dec 15 2006 00:54:53 by "java_re" with MS VC++ 6.0