We're seeing huge performance problems on a web application and we're trying to find the bottleneck. I am not a sysadmin so there is some stuff I don't quite get. Some basic investigation shows the CPU to be idle, lots of memory to be available, no swapping, no I/O, but a high average load.
The software stack on this server looks like this:
The applications running on this server talk with an Oracle database on a different server.
This server SPARC M7-8 has 1TB of RAM and 1 CPMU.
last pid: 19624; load avg: 14.4, 15.1, 17.7; up 9+11:43:25 12:29:38
1468 processes: 1452 sleeping, 16 on cpu
CPU states: 97.0% idle, 2.9% user, 0.1% kernel, 0.0% iowait, 0.0% swap
Kernel: 6320 ctxsw, 3692 trap, 6592 intr, 31951 syscall, 9 fork, 1687 flt
Memory: 957G phys mem, 529G free mem, 32G total swap, 32G free swap
PID USERNAME NLWP PRI NICE SIZE RES STATE TIME CPU COMMAND
11860 oraprod 7 0 0 256G 256G cpu/** 195:36 0.22% oracle
20683 oraprod 7 50 0 256G 256G cpu/** 102:51 0.22% oracle
1077 oraprod 1 0 0 257G 257G cpu/** 324:57 0.20% oracle
1068 oraprod 1 0 0 258G 258G cpu/** 20.0H 0.20% oracle
28877 oraprod 7 0 0 257G 257G cpu/** 152:56 0.20% oracle
19017 oraprod 1 40 0 256G 256G cpu/34 5:24 0.19% oracle
12026 oraprod 7 0 0 256G 256G cpu/** 119:31 0.19% oracle
14849 oraprod 1 0 0 260G 260G cpu/** 71.2H 0.19% oracle
9847 oraprod 1 40 0 256G 256G cpu/77 58:09 0.19% oracle
5442 oraprod 7 0 0 256G 256G cpu/** 136:26 0.19% oracle
29017 oraprod 7 50 0 256G 256G cpu/** 80:57 0.19% oracle
3677 oraprod 7 0 0 256G 256G cpu/** 125:32 0.19% oracle
18658 oraprod 7 0 0 256G 256G cpu/** 0:19 0.18% oracle
15556 oraprod 1 0 0 256G 256G cpu/** 34:32 0.17% oracle
18881 oraprod 1 60 0 256G 256G sleep 0:10 0.12% oracle
19623 oraprod 1 0 0 256G 256G cpu/** 0:00 0.07% oracle
19613 root 1 48 0 13M 12M cpu/** 0:00 0.01% top
15402 oraprod 1 50 0 63M 53M sleep 28:08 0.00% tnslsnr
11987 oraprod 19 59 0 256G 256G sleep 18:39 0.00% oracle
9141 oraprod 1 60 0 256G 256G sleep 0:03 0.00% oracle
11985 oraprod 15 59 0 256G 256G sleep 30:23 0.00% oracle
703 root 19 59 0 43M 40M sleep 34:20 0.00% ldmd
15993 oraprod 1 60 0 256G 256G sleep 0:06 0.00% oracle
12097 oraprod 1 59 0 256G 256G sleep 1:40 0.00% oracle
1 root 1 59 0 3360K 2624K sleep 6:20 0.00% init
11995 oraprod 1 59 0 256G 256G sleep 37:29 0.00% oracle
9213 oraqa 1 59 0 64G 64G sleep 0:55 0.00% oracle
4827 oraqa 1 59 0 64G 64G sleep 23:18 0.00% oracle
13208 oraprod 1 59 0 256G 256G sleep 0:37 0.00% oracle
535 root 36 59 0 515M 513M sleep 2:28 0.00% nscd