Debugging high IDLE cpu usage
I have a basic x86 Solaris 11 Express File Server. It usually sits totally idle, however I've noticed that the % of time in kernel mode, even when totally idle is around 35-40%. So at "idle" the CPU is only really 60% idle.
I've tried to run a lockstat to see what's going on and got
an@Nas:~# lockstat -gkIW sleep 60
Profiling interrupt: 11648 events in 60.038 seconds (194 events/sec)
Count genr cuml rcnt nsec Hottest CPU+PIL Caller
-------------------------------------------------------------------------------
10534 90% ---- 0.00 165530 cpu[1] thread_start
9750 84% ---- 0.00 147359 cpu[1] idle
8128 70% ---- 0.00 94807 cpu[1] cpu_idle_mwait
7852 67% ---- 0.00 84202 cpu[1] i86_mwait
2541 22% ---- 0.00 407347 cpu[1]+11 cpupm_utilization_event
2541 22% ---- 0.00 407347 cpu[1]+11 cpupm_change_state
2540 22% ---- 0.00 407412 cpu[1]+11 cpupm_plat_change_state
2539 22% ---- 0.00 407466 cpu[1]+11 cpupm_state_change
2530 22% ---- 0.00 408167 cpu[1]+11 pg_ev_thread_swtch
2512 22% ---- 0.00 401944 cpu[1]+11 swtch
2511 22% ---- 0.00 407944 cpu[1]+11 cmt_ev_thread_swtch_pwr
2509 22% ---- 0.00 408047 cpu[1]+11 speedstep_power
1311 11% ---- 0.00 421383 cpu[0]+11 speedstep_pstate_transition
1307 11% ---- 0.00 422634 cpu[0]+11 write_ctrl
1307 11% ---- 0.00 422634 cpu[0]+11 cpu_acpi_write_port
1306 11% ---- 0.00 422939 cpu[0]+11 outw
1237 11% ---- 0.00 392091 cpu[1]+11 do_splx
1197 10% ---- 0.00 393779 cpu[1]+11 xc_call
1197 10% ---- 0.00 393779 cpu[1]+11 xc_common
I have no idea what the thread_start() calls are for, and why there are so many of them.
Any clues on where I can start to look? I'm a total Solaris admin n00b so you're going to have to be gentle.