Java HotSpot Virtual Machine

Hi all,

I'm turning to this forum to ask if anyone had any similar problems or has any idea how to approach solving them. I already filed a bug report some time ago: [http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=7030307] but since I haven't been able to create a test case to reproduce it, it was closed. I'll just paste the report here:

FULL PRODUCT VERSION :
java version "1.6.0_25-ea"
Java(TM) SE Runtime Environment (build 1.6.0_25-ea-b01)
Java HotSpot(TM) 64-Bit Server VM (build 20.0-b08, mixed mode)

the same observed also with latest Java 7 beta:

java version "1.7.0-ea"
Java(TM) SE Runtime Environment (build 1.7.0-ea-b131)
Java HotSpot(TM) 64-Bit Server VM (build 21.0-b02, mixed mode)


FULL OS VERSION :
SunOS nobiex1 5.10 Generic_141444-09 sun4v sparc SUNW,Sun-Blade-T6320

EXTRA RELEVANT SYSTEM CONFIGURATION :
32 GB of RAM

A DESCRIPTION OF THE PROBLEM :
We have an application that loads almost 20,000 classes, uses about 17 GB of heap (when stabilized) on each computer and is distributed over several computers. Lately we have observed that a slowness has creeped in the execution of the code. We haven't yet established the exact cause and time of when this started to happen, because when it was apparent that this is causing trouble, no action of reverting back application version or JVM helped to overcome the issue. We have ruled out GC pauses - we don't have GC problems. To illustrate the slowness: A simple HTTP request on an embedded Jetty engine that returns a static file served off a local filesystem takes about 2 seconds. What we did first was to manually insert instrumentation into the application code that measured real-time of the whole request processing and also CPU time (using java.lang.management.ThreadMXBean). We have found out that the thread that takes 2 seconds to process the simple local-file-serving request takes very litle CPU time. It seems that it must be suspended by something. We have not been able to tell what causes this suspension of the thread, since when we load Yourkit Java profiler agent into the JVM, the slowness dissapears. It's like Heisenberg effect - the measurement changes the world so that the problem dissapears. We have turned-on HotSpot compilation debug output and could not detect any suspicious behavior. We would like to submit a bug report that was reproducible, but I'm afraid we can't do that at the moment since we have been unable to isolate the problem yet. So if you have any idea about how to investigate this further, you can contact xxxxx@xxxxx . Just to "wet your curiosity" here are the JVM flags we are currently using:

-Xmx26624m
-XX:MaxPermSize=1024m
-XX:+DisableExplicitGC
-XX:+UseParallelGC
-XX:ParallelGCThreads=32
-XX:+UseParallelOldGC
-XX:LargePageSizeInBytes=256m
-XX:+UseLargePages
-XX:+UseCompressedOops
-XX:+MaxFDLimit
-XX:ReservedCodeCacheSize=512m
-agentpath:/app/tbill/thinkbill/system/yjp-9.5.4/bin/solaris-sparc-64/libyjpagent.so
com.marand.thinkbill.system.Boot


THE PROBLEM WAS REPRODUCIBLE WITH -Xint FLAG: Did not try

THE PROBLEM WAS REPRODUCIBLE WITH -server FLAG: Yes

STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
Don't have exact steps yet

REPRODUCIBILITY :
This bug can be reproduced always.

CUSTOMER SUBMITTED WORKAROUND :
Loading Yourkit Java profiler agent into the JVM.
Posted Date : 2011-03-23 16:45:40.0

Submitted On 17-MAY-2011
5er_levart
I have some further information for you. We are still experiencing this. Even in the latest JDK 7 ea-142 build. We found out that attaching to JVM with jvisualvm and starting "Sampling CPU" profiling session early in the startup sequence of our APP inhibits the slow-down behavior even if later the jvisualvm is detatched from the JVM. If we attach the jvisualvm for the first time at a later time after APP start-up and the slow-down behavior already starts to build-up then attaching/profiling does not help preventing it. In either case the profiler only shows what is going on in the Java code and there's nothing unusual there. I think the problem will have to be tracked down by profiling the JVM internals. Can you provide some pointers on how one can approach that?


Submitted On 18-MAY-2011
5er_levart
Further testing on our part with different platforms shows that this problem is not observed on the Linux/amd64 and not on Solaris/amd64 platforms. So it only happens on the Solaris/SPARC platform. We have observed the problem on different JVMs: (1.6.0_25, 7-ea-142) and different versions of OS: (Solaris 11 Express/SPARC, Solaris 10/SPARC).