Infrastructure Software

Hi,

I am currently troubleshooting a 32 bit application on a E25K that is experiencing serious performance issues. This application is written in C++ and is multithreaded. The following is the result of a sar -p:

0:00:00 atch/s pgin/s ppgin/s pflt/s vflt/s s lock/s
0:01:00 324.4 1.12 1.13 780.64 14301.06 13.53
0:02:00 318.74 0.22 0.42 739.64 12077.03 11.72
0:03:00 274.95 0.03 0.03 603.98 10283.32 17.18
0:04:01 416.39 0 0 852.36 11776.43 17.68
0:05:01 186.17 0 0 467.4 12513.4 17.67
0:06:01 406.54 0.57 0.78 793.53 13649.78 17.17
0:07:01 629.08 0 0 1344.46 21333.22 16.44
0:08:01 567.27 0 0 1253.16 18958.6 16.57
0:09:01 226.79 0 0 569.52 10687.88 16.46
0:10:01 411.8 0 0 946.51 21645.13 16.01
0:11:01 460.4 0.22 0.25 1006.71 25266.86 16.25
0:12:01 229.97 0 0 563.89 26130.14 16.41
0:13:01 241.63 0 0 589.36 28244.06 18.05
0:14:01 361.45 0 0 817.04 26158.89 16.96
0:15:01 586.36 0.08 0.1 1283.04 25869.95 15.47
0:16:02 502.78 0.48 0.58 1153.14 22702.86 16.61
0:17:02 390.1 0 0 914.26 19424.1 15.56
0:18:02 486.76 0 0 1013.14 15823.19 14.6
0:19:02 414.11 0.02 0.02 967.93 14824.62 14.41
0:20:02 246.38 0 0 572.46 17374.39 13.38
0:21:02 187.93 0.55 0.82 488.3 12465.13 12.27
0:22:02 254.93 0 0 615.96 22771.34 11.67
0:23:02 129.83 0 0 336.87 22121.44 11.84
0:24:02 174.77 0 0 433.79 18832.96 11.62
0:25:02 205.14 0.02 0.02 495.64 26721.29 11.75
0:26:02 154.02 0.58 0.82 398.39 20468.75 11.43
0:27:02 161.14 0 0 431.21 28539.48 10.94
0:28:03 264 0 0 601.43 24593.38 17.61
0:29:03 190.15 0 0 481.83 30039.83 17.77
0:30:03 123.65 0 0 314.96 24664.35 17.64
0:31:03 428.93 0 0 835.61 30031.93 17.48
0:32:03 290.78 0 0 1003.93 20481.83 17.17
0:33:03 190.62 0 0 463.38 26561.15 16.97
0:34:03 202.9 0 0 494.09 25631.2 16.88
0:35:03 104.78 0 0 305.29 27225.81 16.48
0:36:03 216.81 0 0 495.71 26844.12 17.01
0:37:03 271.98 0 0 954.25 41285.17 16.48
0:38:03 285.6 0.03 0.05 675.66 41537.12 16.51
0:39:03 240.07 0.77 1.76 511.78 38990.65 16.87
0:40:03 233.33 0.63 1.51 488.65 43014.7 17.05
0:41:04 167.98 0 0 433.41 39095.41 17.51
0:42:04 302.71 0 0 680.84 39159.67 16.91
0:43:04 307.59 0.42 1 750.19 34357.57 15.81
0:44:04 354.38 0.55 1.53 820.04 39713.88 14.28
0:45:04 245.51 0.4 1.2 568.82 39706.93 14.25
0:46:04 255.01 0.03 0.03 622.49 39335.37 13.08
0:47:04 219.22 0 0 544.72 46045.72 12.65
0:48:04 202.25 0 0 591.48 46753.37 12.85
0:49:04 113.17 0.38 1.2 250.84 38384.45 12.85
0:50:04 147.08 0.22 1.17 405.19 50810.44 12.95
0:51:04 248.73 0 0 678.67 47208.8 12.62
0:52:04 226.93 0 0 548.89 44120.68 12.12
0:53:04 145.84 0 0 408.57 47304.96 11.92
0:54:05 127.52 0 0 275.51 48781.32 11.75
0:55:05 131.11 0.18 0.8 375.6 48435.98 11.95
0:56:05 159.82 0 0 446.35 45547.25 11.88
0:57:05 151.71 0 0 485.22 48757.82 12.12
0:58:05 221.27 0 0 456.17 46219.85 11.92
0:59:05 199.9 0.02 0.02 537.82 38478.61 12.05
1:00:01 230.26 0 0 511.16 52717.18 9.94
1:01:01 215.46 0.03 0.08 573.41 56929.79 14.58
1:02:01 204.57 0.02 0.02 534.12 71757.77 10.55
1:03:01 182.61 0 0 571.87 55057.1 9.71
1:04:01 179.31 0 0 443.85 65293.11 9.72
1:05:01 130.71 0 0 349.35 69461.31 9.91
1:06:01 172.24 0 0 539.01 67447.19 9.51
1:07:01 129.49 0 0 319.64 55244.27 8.15
1:08:02 121.04 0.03 0.03 451.5 66876.73 7.72


Are the high number of slock/s and vflt/s an issue? If so how should i go about isolating where the bottleneck or source of the offending process is. I should also note that i have a run queue of >80 on a 24 processor system so suffice it to say something is definitely wrong. The system has a total of 49Gb of memory. Thanks in advance for your suggestions.

Michael Lavastida