Introduction
On modern systems the chances are very high that they are running consolidated workloads. In addition applications are becoming more and more complex using multiple elements running in multiple virtual environments as one cohesive whole. However in such a situation boot order of these elements can also be important as you many want to wait with starting certain applications only if others are already running.
To help address this Oracle Solaris 11.4 now introduces a new technology called Delegated Zones Restarter (DZR), which allows the administrator to set dependencies between Oracle Solaris Zones and even have the booting of one zone depend on an application running with another zone. This is now in part possible because of the introduction of Goal Services and now Oracle Solaris Zones are part of the Oracle Solaris Service Management Facility (SMF). For more specific information see the man page sec.zones(8).
In short, with the goal services you can define when you deem a zone to be fully booted when the service milestone/goals is online. You can add dependencies to other services to the milestone/goals service, for example on your application, and once everything is running the SMF service representing the zone in the global zone is online too. Buy then making the SMF service of another zone dependent on the first zone, this second zone will only start booting once the application in the first zone is online.
To be clear this is only for the boot order at boot time of the global zone. An administrator can always boot zones by hand if they want to.
Note: This works for both Native Zones and Kernel Zones as long as they're running Oracle Solaris 11.4.
Overview
In this article we'll build this in steps. We'll be using the environment as set up - shown Figure 1 - in the OTN article on how to install Oracle Solaris Zones, so if you don't have an environment with zones installed yet you can use that as a guide how to build it. It has a zone called testzone and two webzones, and we can simulate that the application running in the testzone requires a web server to be running.
Figure 1: The general setup.
First we'll have the testzone simply dependent on an Apache Webserver service running in the global zone, then we'll show how to change that dependency to one of the webzones, with the presumption that the zone is running means the webserver is up. Next we'll show how include the dependency on the webserver in the webzone. And then finally we'll show how you can set the dependencies on the testzone such that it will boot if either webzone is running.
Setting a Zone Dependency on a Service
The core task of the Delegated Zone Restarter is to boot zones in the correct order at time of boot or reboot. This means that for this feature to take effect you will need to set autoboot to true in the zonecfg. So in this case we'll need to confirm that autoboot is set for the testzone so it looks at it's dependencies, and not set for the webzones because we want them to delay the boot until we boot them by hand.
Figure 2: Making the testzone dependent on a service in the global zone.
First we're going to show how you can make the booting of the zone dependent on one or more services in the global zone, as shown in Figure 2. First we need to check if the global zone has the Apache Webserver installed, it should buy default but always good to validate:
root@global:~# pkg info apache-24 Name: web/server/apache-24 Summary: Apache Web Server V2.4 Description: The Apache HTTP Server Version 2.4 Category: Web Services/Application and Web Servers State: Installed Publisher: solaris Version: 2.4.33 Branch: 11.4.0.0.1.9.0 Packaging Date: June 18, 2018 at 6:41:14 PM Size: 29.06 MB FMRI: pkg://solaris/web/server/apache-24@2.4.33-11.4.0.0.1.9.0:20180618T184114Z Project URL: https://httpd.apache.org/ Source URL: https://archive.apache.org/dist/httpd/httpd-2.4.33.tar.gz
Next we can set the dependency:
root@global:~# zonecfg -z testzonezonecfg:testzone> add smf-dependencyzonecfg:testzone:smf-dependency> set fmri=svc:/network/http:apache24zonecfg:testzone:smf-dependency> endzonecfg:testzone> exitroot@global:~# svcs -l testzonefmri svc:/system/zones/zone:testzone name Solaris Zone enabled true state online next_state none state_time Wed Jun 27 21:42:27 2018 logfile /var/log/zones/testzone.messages restarter svc:/system/zones:default manifest /lib/svc/manifest/system/zones.xml dependency require_all/none svc:/network/http:apache24 (disabled)
Note the dependency on http:apache24 that has now been added to the zone service.
Now is we reboot the global zone, the testzone will not boot until we enable the http:apache24 service:
root@global:~# rebootreboot: Halting 1 zone. ... root@global:~# zoneadm list -cv ID NAME STATUS PATH BRAND IP 0 global running / solaris shared - testzone installed /system/zones/testzone solaris excl - webzone-1 installed /system/zones/webzone-1 solaris excl - webzone-2 installed /system/zones/webzone-2 solaris excl root@global:~# svcs -l testzonefmri svc:/system/zones/zone:testzone name Solaris Zone enabled true state offline next_state none state_time July 10, 2018 at 1:23:22 AM PDT logfile /var/log/zones/testzone.messages restarter svc:/system/zones:default manifest /lib/svc/manifest/system/zones.xml dependency require_all/none svc:/network/http:apache24 (disabled) root@global:~# svcadm enable http:apache24root@global:~# svcs -l testzonefmri svc:/system/zones/zone:testzone name Solaris Zone enabled true state offline next_state online state_time July 10, 2018 at 1:23:22 AM PDT logfile /var/log/zones/testzone.messages restarter svc:/system/zones:default manifest /lib/svc/manifest/system/zones.xml dependency require_all/none svc:/network/http:apache24 (online) ... root@global:~# svcs -l testzonefmri svc:/system/zones/zone:testzone name Solaris Zone enabled true state online next_state none state_time July 10, 2018 at 1:24:54 AM PDT logfile /var/log/zones/testzone.messages restarter svc:/system/zones:default manifest /lib/svc/manifest/system/zones.xml dependency require_all/none svc:/network/http:apache24 (online)
As you can see, after the reboot initially the zone service is enabled but not online waiting for the http:apache24 service to come online. Once we enable this service the zone service next_state transitions to online and a little later the state is now online too.
Note that you can set multiple dependencies for the single zone, something we'll use at the end of this article.
Setting a Dependency on Another Zone
In this step we replace the previous dependency with one on webzone-1 as shown in Figure 3.
Figure 3: Making the testzone dependent on webzone-1.
You can do this by simply setting a new FMRI:
root@global:~# zonecfg -z testzonezonecfg:testzone> select smf-dependency 0zonecfg:testzone:smf-dependency> infosmf-dependency 0: fmri: svc:/network/http:apache24 zonecfg:testzone:smf-dependency> set name=webzonecfg:testzone:smf-dependency> set fmri=svc:/system/zones/zone:webzone-1zonecfg:testzone:smf-dependency> infosmf-dependency 0: name: web fmri: svc:/system/zones/zone:webzone-1 zonecfg:testzone> exit
Note we're also giving the dependency the name web, this is mostly for later when we want to group dependencies. In general it's good make sure you set a name on the dependency when you first create it.
Now we reboot the zone and then boot webzone-1 while checking for the status on testzone:
root@global:~# rebootreboot: Halting 1 zone. ... root@global:~# zoneadm list -cv ID NAME STATUS PATH BRAND IP 0 global running / solaris shared - testzone installed /system/zones/testzone solaris excl - webzone-1 installed /system/zones/webzone-1 solaris excl - webzone-2 installed /system/zones/webzone-2 solaris excl root@global:~# svcs -l testzonefmri svc:/system/zones/zone:testzone name Solaris Zone enabled true state offline next_state none state_time July 10, 2018 at 2:31:00 AM PDT logfile /var/log/zones/testzone.messages restarter svc:/system/zones:default manifest /lib/svc/manifest/system/zones.xml dependency require_all/none svc:/system/zones/zone:webzone-1 (disabled) root@global:~# zoneadm -z webzone-1 bootroot@global:~# svcs -l testzone webzone-1fmri svc:/system/zones/zone:testzone name Solaris Zone enabled true state offline next_state online state_time July 10, 2018 at 2:31:00 AM PDT logfile /var/log/zones/testzone.messages restarter svc:/system/zones:default manifest /lib/svc/manifest/system/zones.xml dependency require_all/none svc:/system/zones/zone:webzone-1 (online) fmri svc:/system/zones/zone:webzone-1 name Solaris Zone enabled true (temporary) state online next_state none state_time July 10, 2018 at 3:11:06 AM PDT logfile /var/log/zones/webzone-1.messages restarter svc:/system/zones:default manifest /lib/svc/manifest/system/zones.xml ... root@global:~# svcs -l testzone webzone-1fmri svc:/system/zones/zone:testzone name Solaris Zone enabled true state online next_state none state_time July 10, 2018 at 3:11:25 AM PDT logfile /var/log/zones/testzone.messages restarter svc:/system/zones:default manifest /lib/svc/manifest/system/zones.xml dependency require_all/none svc:/system/zones/zone:webzone-1 (online) fmri svc:/system/zones/zone:webzone-1 name Solaris Zone enabled true (temporary) state online next_state none state_time July 10, 2018 at 3:11:06 AM PDT logfile /var/log/zones/webzone-1.messages restarter svc:/system/zones:default manifest /lib/svc/manifest/system/zones.xml
So once the webzone-1 is online, the testzone starts booting.
Appending the Goal Service
As stated above the state of the service representing the zone in the global zone is linked to the goal service milestone/goals running inside the zone.
Figure 4: Setting the goal service up to be dependent on a service.
So, as shown in Figure 4, if you want make this dependent on specific services inside the zone you'll need to change the goals for this service:
root@global:~# zlogin webzone-1[Connected to zone 'webzone-1' pts/1] Last login: Wed Jun 27 23:26:13 2018 on pts/4 Oracle Corporation SunOS 5.11 Solaris_11/11.4/ON/production.build-11.4-26:2018-06-07 June 2018 root@webzone-1:~# svcs -l apache24fmri svc:/network/http:apache24 name Apache 2.4 HTTP server enabled false state disabled next_state none state_time July 10, 2018 at 3:16:23 AM PDT logfile /var/svc/log/network-http:apache24.log restarter svc:/system/svc/restarter:default manifest /lib/svc/manifest/network/http-apache24.xml dependency optional_all/error svc:/system/filesystem/autofs:default (online) dependency require_all/none svc:/system/filesystem/local:default (online) dependency require_all/error svc:/milestone/network:default (online) root@webzone-1:~# svcs -l goalsfmri svc:/milestone/goals:default name goals services for the system enabled true state online next_state none state_time July 10, 2018 at 3:16:32 AM PDT logfile /var/svc/log/milestone-goals:default.log restarter svc:/system/svc/restarter:default manifest /lib/svc/manifest/milestone/goals.xml dependency require_all/restart svc:/milestone/multi-user-server (online) root@webzone-1:~# svcadm goals svc:/network/http:apache24root@webzone-1:~# svcs -l goalsfmri svc:/milestone/goals:default name goals services for the system enabled true state online next_state none state_time July 10, 2018 at 3:16:32 AM PDT logfile /var/svc/log/milestone-goals:default.log restarter svc:/system/svc/restarter:default manifest /lib/svc/manifest/milestone/goals.xml dependency require_all/restart svc:/network/http:apache24 (disabled) root@webzone-1:~# exitlogout [Connection to zone 'webzone-1' pts/1 closed]
At this point the goal service milestone/goals is now set to only go online once the http:apache24 service is up and running.
Now if we reboot the global zone again we can see that booting the testzone waits until the http:apache24 service is up (which we didn't enable yet):
root@global:~# rebootreboot: Halting 2 zones. ... root@global:~# zoneadm list -cv ID NAME STATUS PATH BRAND IP 0 global running / solaris shared - testzone installed /system/zones/testzone solaris excl - webzone-1 installed /system/zones/webzone-1 solaris excl - webzone-2 installed /system/zones/webzone-2 solaris excl root@global:~# zoneadm -z webzone-1 bootroot@global:~# zlogin webzone-1[Connected to zone 'webzone-1' pts/1] Last login: Tue Jul 10 19:30:23 2018 on pts/1 NOTE: system has 1 active defect; run 'fmadm list' for details. Oracle Corporation SunOS 5.11 Solaris_11/11.4/ON/production.build-11.4-26:2018-06-07 June 2018 You have new mail. root@webzone-1:~# svcs -xv svc:/network/http:apache24 (Apache 2.4 HTTP server) State: disabled since July 10, 2018 at 7:33:31 PM PDT Reason: Disabled by an administrator. See: http://support.oracle.com/msg/SMF-8000-05 See: http://httpd.apache.org See: man -M /usr/apache2/2.4/man -s 8 httpd See: /var/svc/log/network-http:apache24.log Impact: 1 dependent service is not running: svc:/milestone/goals:default svc:/milestone/goals:default (goals services for the system) State: maintenance since July 10, 2018 at 7:33:49 PM PDT Reason: goal service has dependencies that cannot be satisfied without administrative intervention. Reason: Service svc:/network/http:apache24 is disabled. See: http://support.oracle.com/msg/SMF-8000-GE Path: svc:/milestone/goals:default svc:/network/http:apache24 See: http://support.oracle.com/msg/(null) See: man -M /usr/share/man -s 7 smf See: man -M /usr/share/man -s 8 svcadm See: /var/svc/log/milestone-goals:default.log Impact: This service is not running. root@webzone-1:~# svcadm enable http:apache24root@webzone-1:~# exitlogout [Connection to zone 'webzone-1' pts/1 closed] You have new mail in /var/mail/root root@global:~# svcs -l testzone webzone-1fmri svc:/system/zones/zone:testzone name Solaris Zone enabled true state online next_state none state_time July 10, 2018 at 7:36:03 PM PDT logfile /var/log/zones/testzone.messages restarter svc:/system/zones:default manifest /lib/svc/manifest/system/zones.xml dependency require_all/none svc:/system/zones/zone:webzone-1 (online) fmri svc:/system/zones/zone:webzone-1 name Solaris Zone enabled true (temporary) state online next_state none state_time July 10, 2018 at 7:35:44 PM PDT logfile /var/log/zones/webzone-1.messages restarter svc:/system/zones:default manifest /lib/svc/manifest/system/zones.xml
You can see that both zones were in the installed state, testzone because it's waiting for webzone-1's goal service and webzone-1 because it doesn't have autoboot enabled. Once we boot webzone-1 and log in we can see the system warns that the milestone/goals service dependencies are not being met because the http:apache24 service isn't enabled. Once we enable that, the milestone is met and the testzone boots.
Setting Multiple Dependencies
Now let's say, as shown in Figure 5, the testzone is dependent on any Apache Webserver it can see, and it doesn't matter which one is up first. In other words, it doesn't care which of the webzones is up as long as one of them is.
Figure 5: Making the testzone dependent in either webzone-1 or webzone-2.
To achieve this you only have to make some minor changes to the zone configuration:
root@global:~# zonecfg -z testzonezonecfg:testzone> info zonename: testzone brand: solaris autoboot: true bootargs: -m verbose anet 0: linkname: net0 configure-allowed-address: true smf-dependency 0: name: web fmri: svc:/system/zones/zone:webzone-1 zonecfg:testzone> select smf-dependency 0zonecfg:testzone:smf-dependency> set grouping=require_anyzonecfg:testzone:smf-dependency> infosmf-dependency 0: name: web fmri: svc:/system/zones/zone:webzone-1 grouping: require_any zonecfg:testzone:smf-dependency> endzonecfg:testzone> add smf-dependencyzonecfg:testzone:smf-dependency> set name=webzonecfg:testzone:smf-dependency> set fmri=svc:/system/zones/zone:webzone-2zonecfg:testzone:smf-dependency> set grouping=require_anyzonecfg:testzone:smf-dependency> infosmf-dependency 1: name: web fmri: svc:/system/zones/zone:webzone-2 grouping: require_any zonecfg:testzone:smf-dependency> endzonecfg:testzone> exit
By adding a second dependency, and also calling this dependency web, and setting the grouping to require_any for both dependencies they will be grouped together as one combined dependency, and the zone should now boot when either zone is booted.
Note that we're not changing the goal service milestone/goals inside webzone-2, we're assuming you can do this if you want to.
Now we reboot, and enable/boot one of the two webzones:
root@global:~# zoneadm list -cv ID NAME STATUS PATH BRAND IP 0 global running / solaris shared - testzone installed /system/zones/testzone solaris excl - webzone-1 installed /system/zones/webzone-1 solaris excl - webzone-2 installed /system/zones/webzone-2 solaris excl root@global:~# svcs -l testzone webzone-1 webzone-2fmri svc:/system/zones/zone:testzone name Solaris Zone enabled true state offline next_state none state_time July 11, 2018 at 1:08:59 AM PDT logfile /var/log/zones/testzone.messages restarter svc:/system/zones:default manifest /lib/svc/manifest/system/zones.xml dependency require_any/none svc:/system/zones/zone:webzone-2 (disabled) svc:/system/zones/zone:webzone-1 (disabled) fmri svc:/system/zones/zone:webzone-1 name Solaris Zone enabled false state disabled next_state none state_time July 11, 2018 at 1:08:59 AM PDT logfile /var/log/zones/webzone-1.messages restarter svc:/system/zones:default manifest /lib/svc/manifest/system/zones.xml fmri svc:/system/zones/zone:webzone-2 name Solaris Zone enabled false state disabled next_state none state_time July 11, 2018 at 1:08:59 AM PDT logfile /var/log/zones/webzone-2.messages restarter svc:/system/zones:default manifest /lib/svc/manifest/system/zones.xml root@global:~# zoneadm -z webzone-2 bootroot@global:~# svcs -l testzone webzone-1 webzone-2fmri svc:/system/zones/zone:testzone name Solaris Zone enabled true state offline next_state online state_time July 11, 2018 at 1:08:59 AM PDT logfile /var/log/zones/testzone.messages restarter svc:/system/zones:default manifest /lib/svc/manifest/system/zones.xml dependency require_any/none svc:/system/zones/zone:webzone-2 (online) svc:/system/zones/zone:webzone-1 (disabled) fmri svc:/system/zones/zone:webzone-1 name Solaris Zone enabled false state disabled next_state none state_time July 11, 2018 at 1:08:59 AM PDT logfile /var/log/zones/webzone-1.messages restarter svc:/system/zones:default manifest /lib/svc/manifest/system/zones.xml fmri svc:/system/zones/zone:webzone-2 name Solaris Zone enabled true (temporary) state online next_state none state_time July 11, 2018 at 1:36:52 AM PDT logfile /var/log/zones/webzone-2.messages restarter svc:/system/zones:default manifest /lib/svc/manifest/system/zones.xml...root@global:~# svcs -l testzone webzone-1 webzone-2fmri svc:/system/zones/zone:testzone name Solaris Zone enabled true state online next_state none state_time July 11, 2018 at 1:37:10 AM PDT logfile /var/log/zones/testzone.messages restarter svc:/system/zones:default manifest /lib/svc/manifest/system/zones.xml dependency require_any/none svc:/system/zones/zone:webzone-2 (online) svc:/system/zones/zone:webzone-1 (disabled) fmri svc:/system/zones/zone:webzone-1 name Solaris Zone enabled false state disabled next_state none state_time July 11, 2018 at 1:08:59 AM PDT logfile /var/log/zones/webzone-1.messages restarter svc:/system/zones:default manifest /lib/svc/manifest/system/zones.xml fmri svc:/system/zones/zone:webzone-2 name Solaris Zone enabled true (temporary) state online next_state none state_time July 11, 2018 at 1:36:52 AM PDT logfile /var/log/zones/webzone-2.messages restarter svc:/system/zones:default manifest /lib/svc/manifest/system/zones.xml root@global:~# zoneadm list -cv ID NAME STATUS PATH BRAND IP 0 global running / solaris shared 1 webzone-2 running /system/zones/webzone-2 solaris excl 2 testzone running /system/zones/testzone solaris excl - webzone-1 installed /system/zones/webzone-1 solaris excl
As you can see, after the reboot the testzone is waiting for either webzone. And once we boot webzone-2 it takes a little bit of time for it to complete it's milestones/goals goal service and once it does testzone boots.
Note that if we hadn't set the grouping to require_any SMF would have defaulted to require_all, so if that's what you want you can just omit that from the definition of the dependencies.
Conclusion
This concludes the examples in this article, here are a few additional thoughts. First, these tools allow you to easily and quickly create inter-zone dependencies to fit most of your needs. And this is especially handy in cases where a lot of smaller services are layered and dependent on each other, or if one service coming online before others is an issue.
Second, it's important to note that at this point this is single node only. So if you migrate a zone to another system and you reboot the system, if another zone is dependent on it this zone will not boot until the dependency is met. You can of course still boot the zone by hand.
Last, if you give the dependencies different names, let's say you give two dependencies the name web and two others the name firewall, and you give them all grouping=require_any, this will a create a logical AND of two ORs. That is to say, the zone will boot if either of the dependencies in with the web name are met, and either of the dependencies with the firewall name are met. It could be that you only need one webserver to be up and one firewall.