We have been load testing our java web application and observe high cpu usage with 50 users (which doesn't seem practical). The CPU shoots up above 80%. While profiling it with java flight recording (JFR) we see that the context switch rate is 8400 per second (as seen in the Hot threads tab on java mission control). Analyzing the hot threads in jfr, it seems the cpu usage is distributed across the application threads with each thread using less than 3% cpu.
Increasing the user load to 100, 150 or 200 users we see the cpu shooting up above 90%, the throughput (transactions per second) remaining constant (as seen for 50 users load) while the response time crosses the acceptable threshold values (3 sec). Decreasing the user load to 20 users shows the cpu usage averages out to be above 55%. It certainly isn't true that the application threads are using up the cpu since our application is not a CPU bound application. The Hot Packages tab under Code tab group confirms this by showing that most of the time the application spends in is executing database queries.
We use glassfish 3.1.2.2 as our application server where the max thread pool is configured to be of 100. Oracle Linux Server release 6.4 is our operating system with linux kernel version as 2.6.39-400.214.4.el6uek.x86_64. I tried executing linux commands namely "watch -n0.5 pidstat -w -I -p " and "watch -n.5 grep ctxt /proc//status" to see the voluntary and involuntary thread context switching at OS level but they don't give any results.
Suspecting that high context switching could be causing the cpu to shoot up, do you have guidelines on what could be done to confirm that thread context switching is the cause of high cpu and what are there ways to tune the jvm or the application if that's the cause?
Thanks!