Skip to Main Content

Infrastructure Software

Announcement

For appeals, questions and feedback about Oracle Forums, please email oracle-forums-moderators_us@oracle.com. Technical questions should be asked in the appropriate category. Thank you!

Interested in getting your voice heard by members of the Developer Marketing team at Oracle? Check out this post for AppDev or this post for AI focus group information.

Create (Boot) Order Using The Delegated Zone Restarter

Joost PronkvH-OracleJul 12 2018 — edited Jul 13 2018

Introduction

On modern systems the chances are very high that they are running consolidated workloads. In addition applications are becoming more and more complex using multiple elements running in multiple virtual environments as one cohesive whole. However in such a situation boot order of these elements can also be important as you many want to wait with starting certain applications only if others are already running.

To help address this Oracle Solaris 11.4 now introduces a new technology called Delegated Zones Restarter (DZR), which allows the administrator to set dependencies between Oracle Solaris Zones and even have the booting of one zone depend on an application running with another zone. This is now in part possible because of the introduction of Goal Services and now Oracle Solaris Zones are part of the Oracle Solaris Service Management Facility (SMF). For more specific information see the man page sec.zones(8).

In short, with the goal services you can define when you deem a zone to be fully booted when the service milestone/goals is online. You can add dependencies to other services to the milestone/goals service, for example on your application, and once everything is running the SMF service representing the zone in the global zone is online too. Buy then making the SMF service of another zone dependent on the first zone, this second zone will only start booting once the application in the first zone is online.

To be clear this is only for the boot order at boot time of the global zone. An administrator can always boot zones by hand if they want to.

Note: This works for both Native Zones and Kernel Zones as long as they're running Oracle Solaris 11.4.

Overview

In this article we'll build this in steps. We'll be using the environment as set up - shown Figure 1 - in the OTN article on how to install Oracle Solaris Zones, so if you don't have an environment with zones installed yet you can use that as a guide how to build it. It has a zone called testzone and two webzones, and we can simulate that the application running in the testzone requires a web server to be running.

OTN_Pictures_DZR-1.png

Figure 1: The general setup.

First we'll have the testzone simply dependent on an Apache Webserver service running in the global zone, then we'll show how to change that dependency to one of the webzones, with the presumption that the zone is running means the webserver is up. Next we'll show how include the dependency on the webserver in the webzone. And then finally we'll show how you can set the dependencies on the testzone such that it will boot if either webzone is running.

Setting a Zone Dependency on a Service

The core task of the Delegated Zone Restarter is to boot zones in the correct order at time of boot or reboot. This means that for this feature to take effect you will need to set autoboot to true in the zonecfg. So in this case we'll need to confirm that autoboot is set for the testzone so it looks at it's dependencies, and not set for the webzones because we want them to delay the boot until we boot them by hand.

OTN_Pictures_DZR-2.png

Figure 2: Making the testzone dependent on a service in the global zone.

First we're going to show how you can make the booting of the zone dependent on one or more services in the global zone, as shown in Figure 2. First we need to check if the global zone has the Apache Webserver installed, it should buy default but always good to validate:

root@global:~# pkg info apache-24          Name: web/server/apache-24        Summary: Apache Web Server V2.4    Description: The Apache HTTP Server Version 2.4       Category: Web Services/Application and Web Servers          State: Installed      Publisher: solaris        Version: 2.4.33         Branch: 11.4.0.0.1.9.0 Packaging Date: June 18, 2018 at  6:41:14 PM           Size: 29.06 MB           FMRI: pkg://solaris/web/server/apache-24@2.4.33-11.4.0.0.1.9.0:20180618T184114Z    Project URL: https://httpd.apache.org/     Source URL: https://archive.apache.org/dist/httpd/httpd-2.4.33.tar.gz

Next we can set the dependency:

root@global:~# zonecfg -z testzonezonecfg:testzone> add smf-dependencyzonecfg:testzone:smf-dependency> set fmri=svc:/network/http:apache24zonecfg:testzone:smf-dependency> endzonecfg:testzone> exitroot@global:~# svcs -l testzonefmri         svc:/system/zones/zone:testzone name         Solaris Zone enabled      true state        online next_state   none state_time   Wed Jun 27 21:42:27 2018 logfile      /var/log/zones/testzone.messages restarter    svc:/system/zones:default manifest     /lib/svc/manifest/system/zones.xml dependency   require_all/none svc:/network/http:apache24 (disabled)

Note the dependency on http:apache24 that has now been added to the zone service.

Now is we reboot the global zone, the testzone will not boot until we enable the http:apache24 service:

root@global:~# rebootreboot: Halting 1 zone.  ...  root@global:~# zoneadm list -cv  ID NAME             STATUS      PATH                         BRAND      IP        0 global           running     /                            solaris    shared    - testzone         installed   /system/zones/testzone       solaris    excl      - webzone-1        installed   /system/zones/webzone-1      solaris    excl      - webzone-2        installed   /system/zones/webzone-2      solaris    excl   root@global:~# svcs -l testzonefmri         svc:/system/zones/zone:testzone name         Solaris Zone enabled      true state        offline next_state   none state_time   July 10, 2018 at  1:23:22 AM PDT logfile      /var/log/zones/testzone.messages restarter    svc:/system/zones:default manifest     /lib/svc/manifest/system/zones.xml dependency   require_all/none svc:/network/http:apache24 (disabled) root@global:~# svcadm enable http:apache24root@global:~# svcs -l testzonefmri         svc:/system/zones/zone:testzone name         Solaris Zone enabled      true state        offline next_state   online state_time   July 10, 2018 at  1:23:22 AM PDT logfile      /var/log/zones/testzone.messages restarter    svc:/system/zones:default manifest     /lib/svc/manifest/system/zones.xml dependency   require_all/none svc:/network/http:apache24 (online)  ...  root@global:~# svcs -l testzonefmri         svc:/system/zones/zone:testzone name         Solaris Zone enabled      true state        online next_state   none state_time   July 10, 2018 at  1:24:54 AM PDT logfile      /var/log/zones/testzone.messages restarter    svc:/system/zones:default manifest     /lib/svc/manifest/system/zones.xml dependency   require_all/none svc:/network/http:apache24 (online)

As you can see, after the reboot initially the zone service is enabled but not online waiting for the http:apache24 service to come online. Once we enable this service the zone service next_state transitions to online and a little later the state is now online too.

Note that you can set multiple dependencies for the single zone, something we'll use at the end of this article.

Setting a Dependency on Another Zone

In this step we replace the previous dependency with one on webzone-1 as shown in Figure 3.

OTN_Pictures_DZR-3.png

Figure 3: Making the testzone dependent on webzone-1.

You can do this by simply setting a new FMRI:

root@global:~# zonecfg -z testzonezonecfg:testzone> select smf-dependency 0zonecfg:testzone:smf-dependency> infosmf-dependency 0:         fmri: svc:/network/http:apache24 zonecfg:testzone:smf-dependency> set name=webzonecfg:testzone:smf-dependency> set fmri=svc:/system/zones/zone:webzone-1zonecfg:testzone:smf-dependency> infosmf-dependency 0:         name: web         fmri: svc:/system/zones/zone:webzone-1 zonecfg:testzone> exit

Note we're also giving the dependency the name web, this is mostly for later when we want to group dependencies. In general it's good make sure you set a name on the dependency when you first create it.

Now we reboot the zone and then boot webzone-1 while checking for the status on testzone:

root@global:~# rebootreboot: Halting 1 zone.  ...  root@global:~# zoneadm list -cv  ID NAME             STATUS      PATH                         BRAND      IP        0 global           running     /                            solaris    shared    - testzone         installed   /system/zones/testzone       solaris    excl      - webzone-1        installed   /system/zones/webzone-1      solaris    excl      - webzone-2        installed   /system/zones/webzone-2      solaris    excl   root@global:~# svcs -l testzonefmri         svc:/system/zones/zone:testzone name         Solaris Zone enabled      true state        offline next_state   none state_time   July 10, 2018 at  2:31:00 AM PDT logfile      /var/log/zones/testzone.messages restarter    svc:/system/zones:default manifest     /lib/svc/manifest/system/zones.xml dependency   require_all/none svc:/system/zones/zone:webzone-1 (disabled) root@global:~# zoneadm -z webzone-1 bootroot@global:~# svcs -l testzone webzone-1fmri         svc:/system/zones/zone:testzone name         Solaris Zone enabled      true state        offline next_state   online state_time   July 10, 2018 at  2:31:00 AM PDT logfile      /var/log/zones/testzone.messages restarter    svc:/system/zones:default manifest     /lib/svc/manifest/system/zones.xml dependency   require_all/none svc:/system/zones/zone:webzone-1 (online)  fmri         svc:/system/zones/zone:webzone-1 name         Solaris Zone enabled      true (temporary) state        online next_state   none state_time   July 10, 2018 at  3:11:06 AM PDT logfile      /var/log/zones/webzone-1.messages restarter    svc:/system/zones:default manifest     /lib/svc/manifest/system/zones.xml  ...  root@global:~# svcs -l testzone webzone-1fmri         svc:/system/zones/zone:testzone name         Solaris Zone enabled      true state        online next_state   none state_time   July 10, 2018 at  3:11:25 AM PDT logfile      /var/log/zones/testzone.messages restarter    svc:/system/zones:default manifest     /lib/svc/manifest/system/zones.xml dependency   require_all/none svc:/system/zones/zone:webzone-1 (online)  fmri         svc:/system/zones/zone:webzone-1 name         Solaris Zone enabled      true (temporary) state        online next_state   none state_time   July 10, 2018 at  3:11:06 AM PDT logfile      /var/log/zones/webzone-1.messages restarter    svc:/system/zones:default manifest     /lib/svc/manifest/system/zones.xml

So once the webzone-1 is online, the testzone starts booting.

Appending the Goal Service

As stated above the state of the service representing the zone in the global zone is linked to the goal service milestone/goals running inside the zone.

OTN_Pictures_DZR-4.png

Figure 4: Setting the goal service up to be dependent on a service.

So, as shown in Figure 4, if you want make this dependent on specific services inside the zone you'll need to change the goals for this service:

root@global:~# zlogin webzone-1[Connected to zone 'webzone-1' pts/1] Last login: Wed Jun 27 23:26:13 2018 on pts/4 Oracle Corporation      SunOS 5.11      Solaris_11/11.4/ON/production.build-11.4-26:2018-06-07  June 2018 root@webzone-1:~# svcs -l apache24fmri         svc:/network/http:apache24 name         Apache 2.4 HTTP server enabled      false state        disabled next_state   none state_time   July 10, 2018 at  3:16:23 AM PDT logfile      /var/svc/log/network-http:apache24.log restarter    svc:/system/svc/restarter:default manifest     /lib/svc/manifest/network/http-apache24.xml dependency   optional_all/error svc:/system/filesystem/autofs:default (online) dependency   require_all/none svc:/system/filesystem/local:default (online) dependency   require_all/error svc:/milestone/network:default (online) root@webzone-1:~# svcs -l goalsfmri         svc:/milestone/goals:default name         goals services for the system enabled      true state        online next_state   none state_time   July 10, 2018 at  3:16:32 AM PDT logfile      /var/svc/log/milestone-goals:default.log restarter    svc:/system/svc/restarter:default manifest     /lib/svc/manifest/milestone/goals.xml dependency   require_all/restart svc:/milestone/multi-user-server (online) root@webzone-1:~# svcadm goals svc:/network/http:apache24root@webzone-1:~# svcs -l goalsfmri         svc:/milestone/goals:default name         goals services for the system enabled      true state        online next_state   none state_time   July 10, 2018 at  3:16:32 AM PDT logfile      /var/svc/log/milestone-goals:default.log restarter    svc:/system/svc/restarter:default manifest     /lib/svc/manifest/milestone/goals.xml dependency   require_all/restart svc:/network/http:apache24 (disabled) root@webzone-1:~# exitlogout  [Connection to zone 'webzone-1' pts/1 closed]

At this point the goal service milestone/goals is now set to only go online once the http:apache24 service is up and running.

Now if we reboot the global zone again we can see that booting the testzone waits until the http:apache24 service is up (which we didn't enable yet):

root@global:~# rebootreboot: Halting 2 zones.  ...  root@global:~# zoneadm list -cv  ID NAME             STATUS      PATH                         BRAND      IP        0 global           running     /                            solaris    shared    - testzone         installed   /system/zones/testzone       solaris    excl      - webzone-1        installed   /system/zones/webzone-1      solaris    excl      - webzone-2        installed   /system/zones/webzone-2      solaris    excl   root@global:~# zoneadm -z webzone-1 bootroot@global:~# zlogin webzone-1[Connected to zone 'webzone-1' pts/1] Last login: Tue Jul 10 19:30:23 2018 on pts/1 NOTE: system has 1 active defect; run 'fmadm list' for details. Oracle Corporation      SunOS 5.11      Solaris_11/11.4/ON/production.build-11.4-26:2018-06-07  June 2018 You have new mail. root@webzone-1:~# svcs -xv  svc:/network/http:apache24 (Apache 2.4 HTTP server)  State: disabled since July 10, 2018 at  7:33:31 PM PDT Reason: Disabled by an administrator.    See: http://support.oracle.com/msg/SMF-8000-05    See: http://httpd.apache.org    See: man -M /usr/apache2/2.4/man -s 8 httpd    See: /var/svc/log/network-http:apache24.log Impact: 1 dependent service is not running:         svc:/milestone/goals:default  svc:/milestone/goals:default (goals services for the system)  State: maintenance since July 10, 2018 at  7:33:49 PM PDT Reason: goal service has dependencies that cannot be satisfied without administrative intervention. Reason: Service svc:/network/http:apache24 is disabled.    See: http://support.oracle.com/msg/SMF-8000-GE   Path: svc:/milestone/goals:default           svc:/network/http:apache24    See: http://support.oracle.com/msg/(null)    See: man -M /usr/share/man -s 7 smf    See: man -M /usr/share/man -s 8 svcadm    See: /var/svc/log/milestone-goals:default.log Impact: This service is not running. root@webzone-1:~# svcadm enable http:apache24root@webzone-1:~# exitlogout  [Connection to zone 'webzone-1' pts/1 closed] You have new mail in /var/mail/root root@global:~# svcs -l testzone webzone-1fmri         svc:/system/zones/zone:testzone name         Solaris Zone enabled      true state        online next_state   none state_time   July 10, 2018 at  7:36:03 PM PDT logfile      /var/log/zones/testzone.messages restarter    svc:/system/zones:default manifest     /lib/svc/manifest/system/zones.xml dependency   require_all/none svc:/system/zones/zone:webzone-1 (online)  fmri         svc:/system/zones/zone:webzone-1 name         Solaris Zone enabled      true (temporary) state        online next_state   none state_time   July 10, 2018 at  7:35:44 PM PDT logfile      /var/log/zones/webzone-1.messages restarter    svc:/system/zones:default manifest     /lib/svc/manifest/system/zones.xml

You can see that both zones were in the installed state, testzone because it's waiting for webzone-1's goal service and webzone-1 because it doesn't have autoboot enabled. Once we boot webzone-1 and log in we can see the system warns that the milestone/goals service dependencies are not being met because the http:apache24 service isn't enabled. Once we enable that, the milestone is met and the testzone boots.

Setting Multiple Dependencies

Now let's say, as shown in Figure 5, the testzone is dependent on any Apache Webserver it can see, and it doesn't matter which one is up first. In other words, it doesn't care which of the webzones is up as long as one of them is.

OTN_Pictures_DZR-5.png

Figure 5: Making the testzone dependent in either webzone-1 or webzone-2.

To achieve this you only have to make some minor changes to the zone configuration:

root@global:~# zonecfg -z testzonezonecfg:testzone> info zonename: testzone brand: solaris autoboot: true bootargs: -m verbose anet 0:         linkname: net0         configure-allowed-address: true smf-dependency 0:         name: web         fmri: svc:/system/zones/zone:webzone-1 zonecfg:testzone> select smf-dependency 0zonecfg:testzone:smf-dependency> set grouping=require_anyzonecfg:testzone:smf-dependency> infosmf-dependency 0:         name: web         fmri: svc:/system/zones/zone:webzone-1         grouping: require_any zonecfg:testzone:smf-dependency> endzonecfg:testzone> add smf-dependencyzonecfg:testzone:smf-dependency> set name=webzonecfg:testzone:smf-dependency> set fmri=svc:/system/zones/zone:webzone-2zonecfg:testzone:smf-dependency> set grouping=require_anyzonecfg:testzone:smf-dependency> infosmf-dependency 1:         name: web         fmri: svc:/system/zones/zone:webzone-2         grouping: require_any zonecfg:testzone:smf-dependency> endzonecfg:testzone> exit

By adding a second dependency, and also calling this dependency web, and setting the grouping to require_any for both dependencies they will be grouped together as one combined dependency, and the zone should now boot when either zone is booted.

Note that we're not changing the goal service milestone/goals inside webzone-2, we're assuming you can do this if you want to.

Now we reboot, and enable/boot one of the two webzones:

root@global:~# zoneadm list -cv  ID NAME             STATUS      PATH                         BRAND      IP        0 global           running     /                            solaris    shared    - testzone         installed   /system/zones/testzone       solaris    excl      - webzone-1        installed   /system/zones/webzone-1      solaris    excl      - webzone-2        installed   /system/zones/webzone-2      solaris    excl   root@global:~# svcs -l testzone webzone-1 webzone-2fmri         svc:/system/zones/zone:testzone name         Solaris Zone enabled      true state        offline next_state   none state_time   July 11, 2018 at  1:08:59 AM PDT logfile      /var/log/zones/testzone.messages restarter    svc:/system/zones:default manifest     /lib/svc/manifest/system/zones.xml dependency   require_any/none svc:/system/zones/zone:webzone-2 (disabled) svc:/system/zones/zone:webzone-1 (disabled)  fmri         svc:/system/zones/zone:webzone-1 name         Solaris Zone enabled      false state        disabled next_state   none state_time   July 11, 2018 at  1:08:59 AM PDT logfile      /var/log/zones/webzone-1.messages restarter    svc:/system/zones:default manifest     /lib/svc/manifest/system/zones.xml  fmri         svc:/system/zones/zone:webzone-2 name         Solaris Zone enabled      false state        disabled next_state   none state_time   July 11, 2018 at  1:08:59 AM PDT logfile      /var/log/zones/webzone-2.messages restarter    svc:/system/zones:default manifest     /lib/svc/manifest/system/zones.xml root@global:~# zoneadm -z webzone-2 bootroot@global:~# svcs -l testzone webzone-1 webzone-2fmri         svc:/system/zones/zone:testzone name         Solaris Zone enabled      true state        offline next_state   online state_time   July 11, 2018 at  1:08:59 AM PDT logfile      /var/log/zones/testzone.messages restarter    svc:/system/zones:default manifest     /lib/svc/manifest/system/zones.xml dependency   require_any/none svc:/system/zones/zone:webzone-2 (online) svc:/system/zones/zone:webzone-1 (disabled)  fmri         svc:/system/zones/zone:webzone-1 name         Solaris Zone enabled      false state        disabled next_state   none state_time   July 11, 2018 at  1:08:59 AM PDT logfile      /var/log/zones/webzone-1.messages restarter    svc:/system/zones:default manifest     /lib/svc/manifest/system/zones.xml  fmri         svc:/system/zones/zone:webzone-2 name         Solaris Zone enabled      true (temporary) state        online next_state   none state_time   July 11, 2018 at  1:36:52 AM PDT logfile      /var/log/zones/webzone-2.messages restarter    svc:/system/zones:default manifest     /lib/svc/manifest/system/zones.xml...root@global:~# svcs -l testzone webzone-1 webzone-2fmri         svc:/system/zones/zone:testzone name         Solaris Zone enabled      true state        online next_state   none state_time   July 11, 2018 at  1:37:10 AM PDT logfile      /var/log/zones/testzone.messages restarter    svc:/system/zones:default manifest     /lib/svc/manifest/system/zones.xml dependency   require_any/none svc:/system/zones/zone:webzone-2 (online) svc:/system/zones/zone:webzone-1 (disabled)  fmri         svc:/system/zones/zone:webzone-1 name         Solaris Zone enabled      false state        disabled next_state   none state_time   July 11, 2018 at  1:08:59 AM PDT logfile      /var/log/zones/webzone-1.messages restarter    svc:/system/zones:default manifest     /lib/svc/manifest/system/zones.xml  fmri         svc:/system/zones/zone:webzone-2 name         Solaris Zone enabled      true (temporary) state        online next_state   none state_time   July 11, 2018 at  1:36:52 AM PDT logfile      /var/log/zones/webzone-2.messages restarter    svc:/system/zones:default manifest     /lib/svc/manifest/system/zones.xml root@global:~# zoneadm list -cv  ID NAME             STATUS      PATH                         BRAND      IP        0 global           running     /                            solaris    shared    1 webzone-2        running     /system/zones/webzone-2      solaris    excl      2 testzone         running     /system/zones/testzone       solaris    excl      - webzone-1        installed   /system/zones/webzone-1      solaris    excl  

As you can see, after the reboot the testzone is waiting for either webzone. And once we boot webzone-2 it takes a little bit of time for it to complete it's milestones/goals goal service and once it does testzone boots.

Note that if we hadn't set the grouping to require_any SMF would have defaulted to require_all, so if that's what you want you can just omit that from the definition of the dependencies.

Conclusion

This concludes the examples in this article, here are a few additional thoughts. First, these tools allow you to easily and quickly create inter-zone dependencies to fit most of your needs. And this is especially handy in cases where a lot of smaller services are layered and dependent on each other, or if one service coming online before others is an issue.

Second, it's important to note that at this point this is single node only. So if you migrate a zone to another system and you reboot the system, if another zone is dependent on it this zone will not boot until the dependency is met. You can of course still boot the zone by hand.

Last, if you give the dependencies different names, let's say you give two dependencies the name web and two others the name firewall, and you give them all grouping=require_any, this will a create a logical AND of two ORs. That is to say, the zone will boot if either of the dependencies in with the web name are met, and either of the dependencies with the firewall name are met. It could be that you only need one webserver to be up and one firewall.

Comments
Post Details
Added on Jul 12 2018
2 comments
1,215 views