by Marc Nesello
Introduction
In this article I want to give a short overview of the monitoring tool zonestat, which can be used to monitor the zones in Oracle Solaris 11. This article will discuss the monitoring of CPU resources and memory resources of native Solaris zones. Monitoring zones is important as this will help you to find out which zones need more resources and which zones barely utilize the resource assigned to them. That way you can ensure that the resources of the server are distributed well among the zones.
You can either use zonestat in the global zone to get an overview of all zones on the server or you can use zonestat from within the non-global zone to observe the current zone. The command zonestat offers many reports to choose from. For example, reports about processes, memory, CPUs, network and shared memory. This article discusses two reports that monitor the usage of the CPU and memory. These reports are called processor-set and memory.
When using the zonestat or libzonestat commands, ensure that the service system/zones-monitoring:default is running in the global zone, so that the zonestat or libzonestat commands are able to monitor the zones.
Monitoring Zones with Dedicated CPUs
Imagine you have a native zone called myzone in Oracle Solaris 11 which is configured with the dedicated-cpu resource with 1 to 2 CPUs. This means that Oracle Solaris 11 creates a resource pool specifically for the zone and assigns 1 to 2 CPUs to this pool.
You can get an overview of the CPU resource usage by using the processor-set report. Just call zonestat -r processor-set 60 1. This will display the processor-set report once after one minute (60 seconds). If you call zonestat from the global zone, the report will look like this:
-bash-4.1$ zonestat -r processor-set 60 1
Collecting data for first interval...
Interval: 1, Duration: 0:01:00
PROCESSOR_SET TYPE ONLINE/CPUS MIN/MAX
pset_default default-pset 2/2 1/-
ZONE USED %USED CAP %CAP SHRS %SHR %SHRU
[total] 0.12 6.30% - - - - -
[system] 0.00 0.38% - - - - -
global 0.11 5.92% - - - - -
PROCESSOR_SET TYPE ONLINE/CPUS MIN/MAX
myzone dedicated-cpu 2/2 1/2
ZONE USED %USED CAP %CAP SHRS %SHR %SHRU
[total] 1.00 50.1% - - - - -
[system] 0.00 0.00% - - - - -
myzone 1.00 50.1% - - - - -
The first block shows the statistics of the default processor set, which is only used by the global zone in this case. The second block shows the statistics of the processor-set assigned to the zone myzone.
In the second block you can see the minimum, maximum, and currently assigned CPUs to this processor set, which are 2. Then three lines starting with [total], [system], and myzone follow, and they display the CPU usage (in percentage and number of CPUs). The line starting with [system] usually describes the usage of this resource (CPU) by other zones or the global zone. But, because the dedicated-cpu resource option is used, the processor set is exclusively assigned to the zone. No other zones or the global zone can use this processor set, so [system] in this case is 0. After the line beginning with [system] there is a line for each zone using this resource. Again, this processor set is used only by this zone, so only the zone (myzone) is displayed here, showing that it uses 1.00 CPUs or almost 50.1% percent of the CPUs in the processor set. The [total] line is the sum of the [system] and myzone lines.
You also can call zonestat in the nonglobal zone. In this case the report only shows statistics of the processor set that is used by the zone.
Monitoring Zones with Capped CPU
If myzone has not dedicated CPUs, but a CPU cap, zonestat is also a way to observe the cap.
You can use the same command zonestat -r processor-set 60 1 for this. Running it from a global zone, the output will look like this:
-bash-4.1$ zonestat -r processor-set 60 1
Collecting data for first interval...
Interval: 1, Duration: 0:01:00
PROCESSOR_SET TYPE ONLINE/CPUS MIN/MAX
pset_default default-pset 4/4 1/-
ZONE USED %USED CAP %CAP SHRS %SHR %SHRU
[total] 1.20 30.1% - - - - -
[system] 0.01 0.33% - - - - -
myzone 0.99 24.9% 1.50 66.5% - - -
global 0.14 3.63% - - - - -
solmarc10 0.04 1.19% 1.50 3.19% - - -
From the global zone all zones are displayed, and which processor set they use. In this case the global zone and the zones myzone and solmarc10 use the default processor set. You can observe the caps and CPU usage in the above output. Both have a cap of 1.5 CPUs and are only using part of their cap.
When running this from within a non-global zone only the CPU cap of the current zone is displayed, and the CPU usage of all other zones is summarized in the line [system].
Monitoring Zones with Capped Memory
To observe memory resource usage the zonestat memory report can be used. In this case the zone myzone is configured with a physical memory cap of 2GB and the zone solmarc10 is configured with a physical memory cap of 1 GB. When calling zonestat -r memory 60 1 from the global zone, following output is shown:
-bash-4.1$ zonestat -r memory 60 1
Collecting data for first interval...
Interval: 1, Duration: 0:01:00
PHYSICAL-MEMORY SYSTEM MEMORY
mem_default 15.7G
ZONE USED %USED CAP %CAP
[total] 4059M 25.1% - -
[system] 2570M 15.9% - -
global 1149M 7.12% - -
myzone 193M 1.20% 2048M 9.45%
solmarc10 146M 0.90% 1024M 14.2%
VIRTUAL-MEMORY SYSTEM MEMORY
vm_default 23.7G
ZONE USED %USED CAP %CAP
[total] 5933M 24.3% - -
[system] 2911M 11.9% - -
global 2710M 11.1% - -
myzone 169M 0.69% 3072M 5.53%
solmarc10 141M 0.58% 1536M 9.23%
LOCKED-MEMORY SYSTEM MEMORY
mem_default 15.7G
ZONE USED %USED CAP %CAP
[total] 2727M 16.9% - -
[system] 2719M 16.8% - -
global 7408K 0.04% - -
myzone 0 0.00% - -
solmarc10 0 0.00% - -
For each memory type a block is displayed. The first block shows statistics about physical memory. The column SYSTEM MEMORY shows the physical memory installed in the system. The second block is virtual memory, which means physical memory + swap files. In this case you can observe that both zones also have a cap set. The third block is locked memory, for which no cap was set in the installed zones. Running this report from the non-global zone will only display memory usage of the current zone and sums the memory usage of other zones and the global zone in the line [system].
There is also an API available to use the zonestat command functionality within programs. This is a great way for ISVs to monitor resource usage of the zone within their applications. This API is documented in the man page for the command libzonestat: https://docs.oracle.com/cd/E88353_01/html/E37842/libzonestat-3lib.html
See Also
To learn more about virtualization monitoring in Oracle Solaris Zones, you can view the complete white paper on the Oracle Technology Network: http://www.oracle.com/technetwork/server-storage/solaris11/documentation/virt-monitoring-solaris-zones-2660307.pdf
About the Author
Marc Nesello is a Senior Software Engineer at ISV Engineering in Germany. He is part of the team working with SAP, where he works with SAP on topics related with virtualization, Oracle Solaris monitoring or the Oracle Solaris Studio compiler.