Integration

The following dump us seen with WebLogic server ver WLS-10.3 on Linux (Linux st-lnx18a 2.6.18-53.1.13.9.1.el5xen #1 SMP Fri Feb 15 02:03:16 EST 2008 x86_64):

===== BEGIN DUMP =============================================================
JRockit dump produced after 0 days, 00:13:09 on Sun Aug 9 21:15:20 2009

**********************************************************
* If you see this dump, please go to *
* http://edocs.bea.com/jrockit/go2troubleshooting.html *
* for troubleshooting information. *
**********************************************************

Additional information is available in:
/home/c8nbhat/pt/850811R2rlnx/webserv/peoplesoft/jrockit.31335.dump
/home/c8nbhat/pt/850811R2rlnx/webserv/peoplesoft/core or core.31335

Error Message: Illegal memory access. [54]
Signal info : si_signo=11, si_code=2 si_addr=0xf3476000
Version : BEA JRockit(R) R27.6.0-50_o-100423-1.6.0_05-20080626-2104-linux-ia32
GC Strategy : Mode: throughput. Currently using strategy: genparpar
GC Status : OC is not running. Last finished OC was OC#2.
: YC is not running. Last finished YC was YC#3.
OC History : Strategy genparpar was used for OC#1 to OC#2.
YC History : Ran 1 YCs before OC#1.
: Ran 1 YCs before OC#2.
: Ran 1 YCs since last OC.
YC Promotion : Last YC successfully promoted all objects
Heap : 0x8100000 - 0x28100000 (Size: 512 MB)
Compaction : 0x8100000 - 0xa100000 (Current compaction type: internal)
NurseryList : 0x8473f78 - 0x19294318
KeepArea : 0x19a09df0 - 0x19294318
NurseryMarker: [ 0x10c2c0e8, 0x147b2698 ]
CompRefs : References are 32-bit.
CPU : Intel Core 2 SSE SSE2 SSE3 SSSE3 Core EM64T
Number CPUs : 2
Tot Phys Mem : 11534336000 (11000 MB)
OS version : Red Hat Enterprise Linux Enterprise Linux Server release 4.1 (Carthage)
Linux version 2.6.18-53.1.13.9.1.el5xen (mockbuild@ca-build17.us.oracle.com) (gcc version 4.1.2 20070626 (Red Hat 4.1.2-14)) #1 SMP Fri Feb 15 02:03:16 EST 2008 (x86_64)
Thread System: NPTL
Java locking : Lazy unlocking enabled (class banning) (transfer banning)
State : JVM is running
Command Line : -XnoOpt -XXnoJITInline -Xms512m -Xmx512m -Dtoplink.xml.platform=oracle.toplink.platform.xml.jaxp.JAXPPlatform -Dcom.sun.xml.namespace.QName.useCompatibleSerialVersionUID=1.0 -Dweblogic.Chunksize=65536 -Djava.util.logging.config.file=/home/c8nbhat/pt/850811R2rlnx/webserv/peoplesoft/applications/peoplesoft/logging.properties -Dorg.apache.commons.logging.Log=org.apache.commons.logging.impl.Jdk14Logger -Dweblogic.Name=PIA -Dps_vault=/home/c8nbhat/pt/850811R2rlnx/webserv/peoplesoft/psvault -Dweblogic.system.BootIdentityFile=/home/c8nbhat/pt/850811R2rlnx/webserv/peoplesoft/servers/WebLogicAdmin/security/boot.properties -Djavax.net.ssl.trustStore=/home/c8nbhat/pt/850811R2rlnx/webserv/peoplesoft/keystore/pskey -Dweblogic.management.server=HTTP://st-lnx18a:9999 -Dweblogic.ProductionModeEnabled=true -Djava.security.policy=/home/gpachaiy/WLS-10.3-Linux/wlserver_10.3/server/lib/weblogic.policy -Dssl.debug=false -Dps_home=/home/c8nbhat/pt/850811R2rlnx -Dsun.java.launcher=SUN_STANDARD weblogic.Server
java.home : /home/gpachaiy/WLS-10.3-Linux/jrockit_160_05/jre
j.class.path : .:/home/c8nbhat/pt/850811R2rlnx/webserv/peoplesoft/lib/ps_patch.jar:/home/c8nbhat/pt/850811R2rlnx/webserv/peoplesoft/applications/peoplesoft/PSIGW.war/WEB-INF/lib/mail.jar:/home/gpachaiy/WLS-10.3-Linux/patch_wlw1030/profiles/default/sys_manifest_classpath/weblogic_patch.jar:/home/gpachaiy/WLS-10.3-Linux/patch_wls1030/profiles/default/sys_manifest_classpath/weblogic_patch.jar:/home/gpachaiy/WLS-10.3-Linux/patch_cie660/profiles/default/sys_manifest_classpath/weblogic_patch.jar:/home/gpachaiy/WLS-10.3-Linux/jrockit_160_05/lib/tools.jar:/home/gpachaiy/WLS-10.3-Linux/wlserver_10.3/server/lib/weblogic_sp.jar:/home/gpachaiy/WLS-10.3-Linux/wlserver_10.3/server/lib/weblogic.jar:
j.lib.path : /home/gpachaiy/WLS-10.3-Linux/jrockit_160_05/jre/lib/i386/jrockit:/home/gpachaiy/WLS-10.3-Linux/jrockit_160_05/jre/lib/i386:/home/gpachaiy/WLS-10.3-Linux/jrockit_160_05/jre/../lib/i386::/usr/local/lib:/usr/lib/X11:/tuxedo/prod/10gR3-j10gR3-64bit/lib:/cobol/prod/svrexp-5.0-64bit/coblib:/lib:/pt/products/oel-5-x86_64/lib:/home/gpachaiy/WLS-10.3-Linux/wlserver_10.3/server/native/linux/i686:/home/gpachaiy/WLS-10.3-Linux/wlserver_10.3/server/native/linux/i686/oci920_8:/home/gpachaiy/WLS-10.3-Linux/wlserver_10.3/server/native/linux/i686:/home/gpachaiy/WLS-10.3-Linux/wlserver_10.3/server/native/linux/i686/oci920_8:/home/c8nbhat/pt/850811R2rlnx/verity/linux/_ilnx21/bin
JAVA_HOME : /home/gpachaiy/WLS-10.3-Linux/jrockit_160_05
JAVAOPTIONS: <not set>
LD_LIBRARY_PATH: /home/gpachaiy/WLS-10.3-Linux/jrockit_160_05/jre/lib/i386/jrockit:/home/gpachaiy/WLS-10.3-Linux/jrockit_160_05/jre/lib/i386:/home/gpachaiy/WLS-10.3-Linux/jrockit_160_05/jre/../lib/i386::/usr/local/lib:/usr/lib/X11:/tuxedo/prod/10gR3-j10gR3-64bit/lib:/cobol/prod/svrexp-5.0-64bit/coblib:/lib:/pt/products/oel-5-x86_64/lib:/home/gpachaiy/WLS-10.3-Linux/wlserver_10.3/server/native/linux/i686:/home/gpachaiy/WLS-10.3-Linux/wlserver_10.3/server/native/linux/i686/oci920_8:/home/gpachaiy/WLS-10.3-Linux/wlserver_10.3/server/native/linux/i686:/home/gpachaiy/WLS-10.3-Linux/wlserver_10.3/server/native/linux/i686/oci920_8:/home/c8nbhat/pt/850811R2rlnx/verity/linux/_ilnx21/bin
LD_ASSUME_KERNEL: <not set>
C Heap : Good; no memory allocations have failed
StackOverFlow: 1 StackOverFlowError has occured
OutOfMemory : 0 OutOfMemoryErrors have occured

Registers (from ThreadContext: 0xf3468580 / OS context: 0xf346867c):
eax = 00000000 ecx = 00006831 edx = 00000000 ebx = ee86d22c
esp = f3468970 ebp = f34900f0 esi = f282123c edi = f3476000
es = 0000002b cs = 00000023 ss = 3100002b ds = 0000002b
fs = 00000003 gs = 00000063
eip = 00c24077 eflags = 00000246

Stack:
(* marks the word pointed to by the stack pointer)
f3468970: f0ca9a14* ee757b46 f3468cb8 00000000 0002740c ee757b19
f3468988: 001b001a 00000000 0000000c 00000049 00000000 00000007
f34689a0: 0048004a 00000000 0000000c 08078924 00000000 00000101
f34689b8: f5fcddc4 00000000 00000101 004c004b 00000000 0000000c

Code:
(* marks the word pointed to by the instruction pointer)
00c24044: 4c8b0824 b60f1024 e30c2444 83d7892c 117403e2 49aa047a
00c2405c: 49aa1f74 f2831b74 aa027501 c1ca8949 e28302e9 01c06903
00c24074: f3010101* f3d189ab 24448baa 90c35f08 90909090 90909090
00c2408c: 90909090 0c24448b 10244439 4052820f 90660007 0c244c8b

Loaded modules:
(* denotes the module causing the exception)
08048000-08058253 /home/gpachaiy/WLS-10.3-Linux/jrockit_160_05/jre/bin/java
ffffe000-ffffe7bf /home/gpachaiy/WLS-10.3-Linux/jrockit_160_05/jre/bin/java
00cf8000-00d0a35f /lib/libpthread.so.0
00d17000-00d3b1c7 /lib/libm.so.6
00d11000-00d12ceb /lib/libdl.so.2
00bb6000-00cef10b */lib/libc.so.6
00b99000-00bb1e53 /lib/ld-linux.so.2
f7d2b000-f7f82367 /home/gpachaiy/WLS-10.3-Linux/jrockit_160_05/jre/lib/i386/jrockit/libjvm.so
003c6000-003cc46b /lib/librt.so.1
f71e3000-f71ed9bb /home/gpachaiy/WLS-10.3-Linux/jrockit_160_05/jre/lib/i386/libverify.so
f71be000-f71e0df7 /home/gpachaiy/WLS-10.3-Linux/jrockit_160_05/jre/lib/i386/libjava.so
f71a7000-f71b9bff /lib/libnsl.so.1
f6100000-f6105ef0 /home/gpachaiy/WLS-10.3-Linux/jrockit_160_05/jre/lib/i386/native_threads/libhpi.so
f5e67000-f5e753e4 /home/gpachaiy/WLS-10.3-Linux/jrockit_160_05/jre/lib/i386/libzip.so
f51a7000-f51b9b38 /home/gpachaiy/WLS-10.3-Linux/jrockit_160_05/jre/lib/i386/libnet.so
f4dc4000-f4dc9666 /home/gpachaiy/WLS-10.3-Linux/jrockit_160_05/jre/lib/i386/libmanagement.so
f495a000-f4960424 /home/gpachaiy/WLS-10.3-Linux/jrockit_160_05/jre/lib/i386/libnio.so
f351a000-f3523357 /home/gpachaiy/WLS-10.3-Linux/jrockit_160_05/jre/lib/i386/libjmapi.so
f40e7000-f40e8de4 /home/gpachaiy/WLS-10.3-Linux/wlserver_10.3/server/native/linux/i686/libwlfileio2.so
f610e000-f611264b /home/c8nbhat/pt/850811R2rlnx/webserv/peoplesoft/applications/peoplesoft/PSOL.war/WEB-INF/lib/linux/libpsjniverity.so
ee5ab000-ee844cbf /home/c8nbhat/pt/850811R2rlnx/verity/linux/_ilnx21/bin/libvdk30.so
ee482000-ee561897 /usr/lib/libstdc++.so.6
00d40000-00d4a91f /lib/libgcc_s.so.1


"[STANDBY] ExecuteThread: '1' fo" id=18 idx=0x48 tid=31368 lastJavaFrame=0xf3494dfc

Stack 0: start=0xf3474000, end=0xf3496000, guards=0xf3479000 (disabled), forbidden=0xf3477000
Thread Stack Trace:
at memset+55()@0xc24077
at CDB_readparse_aux+102()@0xee759ca5
at CDB_readparse+53()@0xee759c37
at CSetParse+121()@0xee7634e6
at VduCharSetNew+558()@0xee75ae5b
at VduLocaleNewEx+1064()@0xee78eeff
at VduLocaleNew+35()@0xee78eace
at LocSetSessionLocale+80()@0xee78f675
at UtlInitPhase1+1101()@0xee7cb423
at UTL_init+166()@0xee7cb6c6
at VdkGlobalNew+1171()@0xee61f133
at IVdkSessionNew+108()@0xee650a8c
at VdkSessionNew+375()@0xee650966
at ZN10vdkSessionC1EPcS0S0_+140()@0xf611020c
at ZN13SearchSessionC1EPcS0S0_PPKcS0_S0_+109()@0xf611093d
at Java_com_peoplesoft_pt_psol_psverity_psjniverity_vdkOpenSearch+843()@0xf6110efb
-- Java stack --
at com/peoplesoft/pt/psol/psverity/psjniverity.vdkOpenSearch(Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;[Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;)I(Native Method)
at com/peoplesoft/pt/psol/psverity/VeritySearch.search(VeritySearch.java:190)
at com/peoplesoft/pt/psol/FullTextSearch.doGet(FullTextSearch.java:338)
at com/peoplesoft/pt/psol/FullTextSearch.doPost(FullTextSearch.java:414)
at javax/servlet/http/HttpServlet.service(HttpServlet.java:727)
at com/peoplesoft/pt/psol/FullTextSearch.service(FullTextSearch.java:114)
at javax/servlet/http/HttpServlet.service(HttpServlet.java:820)
at weblogic/servlet/internal/StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:227)
at weblogic/servlet/internal/StubSecurityHelper.invokeServlet(StubSecurityHelper.java:125)
at weblogic/servlet/internal/ServletStubImpl.execute(ServletStubImpl.java:292)
at weblogic/servlet/internal/ServletStubImpl.execute(ServletStubImpl.java:175)
at weblogic/servlet/internal/WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3498)
at weblogic/security/acl/internal/AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
at weblogic/security/service/SecurityManager.runAs(Lweblogic/security/acl/internal/AuthenticatedSubject;Lweblogic/security/acl/internal/AuthenticatedSubject;Ljava/security/PrivilegedAction;)Ljava/lang/Object;(Unknown Source)
at weblogic/servlet/internal/WebAppServletContext.securedExecute(WebAppServletContext.java:2180)
at weblogic/servlet/internal/WebAppServletContext.execute(WebAppServletContext.java:2086)
at weblogic/servlet/internal/ServletRequestImpl.run(ServletRequestImpl.java:1406)
at weblogic/work/ExecuteThread.execute(ExecuteThread.java:201)
at weblogic/work/ExecuteThread.run(ExecuteThread.java:173)
at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
-- end of trace
Pending Exception:
java/lang/StackOverflowError:
--- End of stack trace

Extended, platform specific info:
libc release: 2.5-stable
Elf headers:
libc ehdrs: EI: 7f454c46010101000000000000000000 ET: 3 EM: 3 V: 1 ENTRY: 00bcbf30 PHOFF: 00000034 SHOFF: 001826c4 EF: 0x0 HS: 52 PS: 32 PHN; 10 SS: 40 SHN: 75 STIDX: 74
libpthread ehdrs: EI: 7f454c46010101000000000000000000 ET: 3 EM: 3 V: 1 ENTRY: 00cfc7d0 PHOFF: 00000034 SHOFF: 0001e4a4 EF: 0x0 HS: 52 PS: 32 PHN; 9 SS: 40 SHN: 40 STIDX: 39
libjvm ehdrs: EI: 7f454c46010101000000000000000000 ET: 3 EM: 3 V: 1 ENTRY: 0004bf70 PHOFF: 00000034 SHOFF: 012c54a4 EF: 0x0 HS: 52 PS: 32 PHN; 4 SS: 40 SHN: 29 STIDX: 26

**********************************************************
* If you see this dump, please go to *
* http://edocs.bea.com/jrockit/go2troubleshooting.html *
* for troubleshooting information. *
**********************************************************

===== END DUMP ===============================================================

If I use the *-Xss256k* option while booting the WL server, in
JAVA_OPTIONS="-jrockit -XnoOpt -XXnoJITInline -Xms512m -Xmx512m -Dtoplink.xml.platform=oracle.toplink.platform.xml.jaxp.JAXPPlatform -Dcom.sun.xml.namespace.QName.useCompatibleSerialVersionUID=1.0"
then, there is no crash seen.

i.e., the JAVA_OPTIONS value is modified to:
JAVA_OPTIONS="-jrockit –Xss256k -XnoOpt -XXnoJITInline -Xms512m -Xmx512m -Dtoplink.xml.platform=oracle.toplink.platform.xml.jaxp.JAXPPlatform -Dcom.sun.xml.namespace.QName.useCompatibleSerialVersionUID=1.0"

The java program (FullTextSearch.java) shown above is actually executing a native (C++) code through a JNI module and expecting a bunch of search results.

1) Is this some problem with WebLogic on Linux that it is not able to fetch results in return due to stack size?
2) How is the option "–Xss256k" helping WebLogic not to crash on this execution?
3) Whenther using the option -Xss256k is the recommended approach for this?

Thanks.

Edited by: user766627 on Aug 10, 2009 11:23 AM