Skip to Main Content

Use Vagrant and VirtualBox to setup Oracle Container Services for use with Kubernetes

Honglin Su-OracleMar 25 2018 — edited Mar 25 2018

Introduction

Recently we announced the Oracle Vagrant boxes GitHub repository. Now you have a more streamlined way to create virtual machines with Oracle software fully configured and ready to go inside of them. With Oracle VM VirtualBox, Oracle Linux and Vagrant, you can easily build a consistent workflow to create disposable Linux environment whose configuration and dependencies are isolated from your development machine on every platform (Windows, MacOS or Linux).

The example below is to show how simple to use Vagrant to set up Oracle Container Services for use with Kubernetes.

Install Required Software

Before you get started, install the required software:

Clone Oracle Vagrant GitHub Repository

If you haven't cloned or downloaded the GitHub repository, perform git clone to prepare for your own Vagrant boxes.

$ git clone https://github.com/oracle/vagrant-boxes

Cloning into 'vagrant-boxes'...

remote: Counting objects: 342, done.

remote: Compressing objects: 100% (58/58), done.

remote: Total 342 (delta 42), reused 71 (delta 31), pack-reused 249

Receiving objects: 100% (342/342), 69.52 KiB | 827.00 KiB/s, done.

Resolving deltas: 100% (170/170), done.

Setup Master Node

Go the specific sub-folder.

$ cd vagrant-boxes/Kubernetes/

The Vagrantfile will provision a Kubernetes cluster with one master and n worker nodes. This example shows 2 worker nodes.

First we'll set up the master node.

$ vagrant up master

Bringing machine 'master' up with 'virtualbox' provider...

==> master: Importing base box 'ol7-latest'...

==> master: Matching MAC address for NAT networking...

==> master: Setting the name of the VM: Kubernetes_master_1521995861580_23054

==> master: Clearing any previously set network interfaces...

==> master: Preparing network interfaces based on configuration...

    master: Adapter 1: nat

    master: Adapter 2: hostonly

==> master: Forwarding ports...

    master: 8001 (guest) => 8001 (host) (adapter 1)

    master: 22 (guest) => 2222 (host) (adapter 1)

==> master: Running 'pre-boot' VM customizations...

==> master: Booting VM...

...
...

==> master: Running provisioner: shell...

    master: Running: C:/Software/Cygwin64/tmp/vagrant-shell20180325-8000-ch4j4k.sh

    master: Installing and configuring Docker Engine

    master: Package btrfs-progs-4.9.1-1.0.2.el7.x86_64 already installed and latest version

    master: Resolving Dependencies

    master: --> Running transaction check

    master: ---> Package docker-engine.x86_64 0:17.12.0.ol-1.0.1.el7 will be installed

    ...

    ...

        master: Dependencies Resolved

    master:

    master: ================================================================================

    master:  Package                  Arch     Version                  Repository     Size

    master: ================================================================================

    master: Installing:

    master:  docker-engine            x86_64   17.12.0.ol-1.0.1.el7     ol7_preview    30 M

    master: Installing for dependencies:

    master:  audit-libs-python        x86_64   2.7.6-3.el7              ol7_latest     73 k

    master:  checkpolicy              x86_64   2.5-4.el7                ol7_latest    290 k

    master:  container-selinux        noarch   2:2.21-1.el7             ol7_addons     28 k

    master:  libcgroup                x86_64   0.41-13.el7              ol7_latest     64 k

    master:  libsemanage-python       x86_64   2.5-8.el7                ol7_latest    104 k

    master:  libtool-ltdl             x86_64   2.4.2-22.el7_3           ol7_latest     48 k

    master:  policycoreutils-python   x86_64   2.5-17.1.0.1.el7         ol7_latest    445 k

    master:  python-IPy               noarch   0.75-6.el7               ol7_latest     32 k

    master:  setools-libs             x86_64   3.3.8-1.1.el7            ol7_latest    611 k

    master:

    master: Transaction Summary

    master: ================================================================================

    master: Install  1 Package (+9 Dependent packages)

    master: Total download size: 32 M

    master: Installed size: 128 M

    master: Downloading packages:

    master: --------------------------------------------------------------------------------

    master: Total                                              7.3 MB/s |  32 MB  00:04

    ...

    master: Installed:

    master:   docker-engine.x86_64 0:17.12.0.ol-1.0.1.el7

    master:

    master: Dependency Installed:

    ...

    master: Complete!

    master: Creating 'btrfs' file system on: /dev/sdb

    master: Created symlink from /etc/systemd/system/multi-user.target.wants/docker.service to /usr/lib/systemd/system/docker.service.

    master: Installing and configuring Kubernetes packages

    master: Resolving Dependencies

    ...

    master: Dependencies Resolved

    master:

    master: ================================================================================

    master:  Package                    Arch       Version             Repository      Size

    master: ================================================================================

    master: Installing:

    master:  kubeadm                    x86_64     1.9.1-2.0.2.el7     ol7_addons      17 M

    master: Installing for dependencies:

    master:  kubectl                    x86_64     1.9.1-2.0.2.el7     ol7_addons     8.9 M

    master:  kubelet                    x86_64     1.9.1-2.0.2.el7     ol7_addons      17 M

    master:  kubernetes-cni             x86_64     0.6.0-2.0.1.el7     ol7_addons     797 k

    master:  kubernetes-cni-plugins     x86_64     0.6.0-2.0.1.el7     ol7_addons     8.5 M

    master:  socat                      x86_64     1.7.3.2-2.el7       ol7_latest     289 k

    master:

    master: Transaction Summary

    master: ================================================================================

    master: Install  1 Package (+5 Dependent packages)

    master:

    master: Total download size: 52 M

    master: Installed size: 279 M

    master: Downloading packages:

    master: --------------------------------------------------------------------------------

    master: Total                                              5.0 MB/s |  52 MB  00:10

...

    master: Installed:

    master:   kubeadm.x86_64 0:1.9.1-2.0.2.el7

    master:

     ...

    master: Complete!

    master: net.bridge.bridge-nf-call-ip6tables = 1

    master: net.bridge.bridge-nf-call-iptables = 1

    master: Your Kubernetes VM is ready to use!

==> master: Configuring proxy for Docker...

==> master: Running provisioner: shell...

    master: Running: inline script

==> master: Configuring proxy for Docker...

==> master: Running provisioner: shell...

    master: Running: inline script

==> master: Configuring proxy for Docker...

Login to the master node virtual machine and run as root to configure the master node. In this step, you will be asked to sign into the Oracle Container Registry.

$ vagrant ssh master

Welcome to Oracle Linux Server release 7.4 (GNU/Linux 4.1.12-112.16.4.el7uek.x86_64)

The Oracle Linux End-User License Agreement can be viewed here:

    * /usr/share/eula/eula.en_US

For additional packages, updates, documentation and community help, see:

    * http://yum.oracle.com/

[vagrant@master ~]$ su root

[root@master vagrant]# /vagrant/scripts/kubeadm-setup-master.sh

/vagrant/scripts/kubeadm-setup-master.sh: Login to container registry

Username: honglin.su@oracle.com

Password:

Login Succeeded

/vagrant/scripts/kubeadm-setup-master.sh: Setup Master node

Starting to initialize master node ...

Checking if env is ready ...

Checking whether docker can pull busybox image ...

Checking access to container-registry.oracle.com/kubernetes ...

v1.9.1: Pulling from kubernetes/kube-proxy-amd64

Digest: sha256:852fbdc6be8b357356c047bd9649e1c62f572c0e61a0526cd048c0d0dc675e4d

Status: Image is up to date for container-registry.oracle.com/kubernetes/kube-proxy-amd64:v1.9.1

Checking whether docker can run container ...

Checking iptables default rule ...

Checking br_netfilter module ...

Checking sysctl variables ...

Enabling kubelet ...

Created symlink from /etc/systemd/system/multi-user.target.wants/kubelet.service to /etc/systemd/system/kubelet.service.

Check successful, ready to run 'up' command ...

Waiting for kubeadm to setup master cluster...

Please wait ...

\ - 75% completed

Waiting for the control plane to become ready ...

...............

100% completed

clusterrole "flannel" created

clusterrolebinding "flannel" created

serviceaccount "flannel" created

configmap "kube-flannel-cfg" created

daemonset "kube-flannel-ds" created

Installing kubernetes-dashboard ...

...

/vagrant/scripts/kubeadm-setup-master.sh: Copying admin.conf for vagrant user

/vagrant/scripts/kubeadm-setup-master.sh: Copying admin.conf into host directory

/vagrant/scripts/kubeadm-setup-master.sh: Saving token for worker nodes

/vagrant/scripts/kubeadm-setup-master.sh: Master node ready, run

        /vagrant/scripts/kubeadm-setup-worker.sh

on the worker nodes

[root@master vagrant]# exit

Setup Worker Nodes

Go back to your development environment and setup the first worker node (worker1).

$ vagrant up worker1

Bringing machine 'worker1' up with 'virtualbox' provider...

==> worker1: Importing base box 'ol7-latest'...

==> worker1: Matching MAC address for NAT networking...

==> worker1: Setting the name of the VM: Kubernetes_worker1_1521998480587_67371

==> worker1: Fixed port collision for 22 => 2222. Now on port 2200.

==> worker1: Clearing any previously set network interfaces...

==> worker1: Preparing network interfaces based on configuration...

    worker1: Adapter 1: nat

    worker1: Adapter 2: hostonly

==> worker1: Forwarding ports...

    worker1: 22 (guest) => 2200 (host) (adapter 1)

==> worker1: Running 'pre-boot' VM customizations...

==> worker1: Booting VM...

        ...

    worker1: Package btrfs-progs-4.9.1-1.0.2.el7.x86_64 already installed and latest version

    worker1: Resolving Dependencies

    worker1: --> Running transaction check

    worker1: ---> Package docker-engine.x86_64 0:17.12.0.ol-1.0.1.el7 will be installed

    worker1: Dependencies Resolved

    ...

    worker1:

    worker1: ================================================================================

    worker1:  Package                  Arch     Version                  Repository     Size

    worker1: ================================================================================

    worker1: Installing:

    worker1:  docker-engine            x86_64   17.12.0.ol-1.0.1.el7     ol7_preview    30 M

    worker1: Installing for dependencies:

    worker1:  audit-libs-python        x86_64   2.7.6-3.el7              ol7_latest     73 k

    worker1:  checkpolicy              x86_64   2.5-4.el7                ol7_latest    290 k

    worker1:  container-selinux        noarch   2:2.21-1.el7             ol7_addons     28 k

    worker1:  libcgroup                x86_64   0.41-13.el7              ol7_latest     64 k

    worker1:  libsemanage-python       x86_64   2.5-8.el7                ol7_latest    104 k

    worker1:  libtool-ltdl             x86_64   2.4.2-22.el7_3           ol7_latest     48 k

    worker1:  policycoreutils-python   x86_64   2.5-17.1.0.1.el7         ol7_latest    445 k

    worker1:  python-IPy               noarch   0.75-6.el7               ol7_latest     32 k

    worker1:  setools-libs             x86_64   3.3.8-1.1.el7            ol7_latest    611 k

    worker1:

    worker1: Transaction Summary

    worker1: ================================================================================

    worker1: Install  1 Package (+9 Dependent packages)

    worker1: Total download size: 32 M

    worker1: Installed size: 128 M

    worker1: Downloading packages:

    worker1: --------------------------------------------------------------------------------

    worker1: Total                                              8.6 MB/s |  32 MB  00:03

    ...

    worker1: Installed:

    worker1:   docker-engine.x86_64 0:17.12.0.ol-1.0.1.el7

    worker1:

    worker1: Dependency Installed:

       ...

    worker1: Complete!

    worker1: Creating 'btrfs' file system on: /dev/sdb

    worker1: Created symlink from /etc/systemd/system/multi-user.target.wants/docker.service to /usr/lib/systemd/system/docker.service.

    worker1: Installing and configuring Kubernetes packages

    worker1: Resolving Dependencies

    worker1: --> Running transaction check

    worker1: ---> Package kubeadm.x86_64 0:1.9.1-2.0.2.el7 will be installed

    worker1: Dependencies Resolved

    ...

    worker1:

    worker1: ================================================================================

    worker1:  Package                    Arch       Version             Repository      Size

    worker1: ================================================================================

    worker1: Installing:

    worker1:  kubeadm                    x86_64     1.9.1-2.0.2.el7     ol7_addons      17 M

    worker1: Installing for dependencies:

    worker1:  kubectl                    x86_64     1.9.1-2.0.2.el7     ol7_addons     8.9 M

    worker1:  kubelet                    x86_64     1.9.1-2.0.2.el7     ol7_addons      17 M

    worker1:  kubernetes-cni             x86_64     0.6.0-2.0.1.el7     ol7_addons     797 k

    worker1:  kubernetes-cni-plugins     x86_64     0.6.0-2.0.1.el7     ol7_addons     8.5 M

    worker1:  socat                      x86_64     1.7.3.2-2.el7       ol7_latest     289 k

    worker1:

    worker1: Transaction Summary

    worker1: ================================================================================

    worker1: Install  1 Package (+5 Dependent packages)

    worker1: Total download size: 52 M

    worker1: Installed size: 279 M

    worker1: Downloading packages:

    worker1: --------------------------------------------------------------------------------

    worker1: Total                                               17 MB/s |  52 MB  00:03

    ...

    worker1: Installed:

    worker1:   kubeadm.x86_64 0:1.9.1-2.0.2.el7

    worker1:

    worker1: Dependency Installed:

    worker1:   kubectl.x86_64 0:1.9.1-2.0.2.el7

    worker1:   kubelet.x86_64 0:1.9.1-2.0.2.el7

    worker1:   kubernetes-cni.x86_64 0:0.6.0-2.0.1.el7

    worker1:   kubernetes-cni-plugins.x86_64 0:0.6.0-2.0.1.el7

    worker1:   socat.x86_64 0:1.7.3.2-2.el7

    worker1: Complete!

    worker1: net.bridge.bridge-nf-call-ip6tables = 1

    worker1: net.bridge.bridge-nf-call-iptables = 1

    worker1: Your Kubernetes VM is ready to use!

==> worker1: Configuring proxy for Docker...

Log into the first worker node and configure the worker node. Similarly, you will be asked to sign into the Oracle Container Registry.

$ vagrant ssh worker1

Welcome to Oracle Linux Server release 7.4 (GNU/Linux 4.1.12-112.16.4.el7uek.x86_64)

The Oracle Linux End-User License Agreement can be viewed here:

    * /usr/share/eula/eula.en_US

For additional packages, updates, documentation and community help, see:

    * http://yum.oracle.com/

[vagrant@worker1 ~]$ su root

[root@worker1 vagrant]# /vagrant/scripts/kubeadm-setup-worker.sh

/vagrant/scripts/kubeadm-setup-worker.sh: Login to container registry

Username: honglin.su@oracle.com

Password:

Login Succeeded

/vagrant/scripts/kubeadm-setup-worker.sh: Setup Worker node

Starting to initialize worker node ...

Checking if env is ready ...

Checking whether docker can pull busybox image ...

Checking access to container-registry.oracle.com/kubernetes ...

v1.9.1: Pulling from kubernetes/kube-proxy-amd64

Digest: sha256:852fbdc6be8b357356c047bd9649e1c62f572c0e61a0526cd048c0d0dc675e4d

Status: Image is up to date for container-registry.oracle.com/kubernetes/kube-proxy-amd64:v1.9.1

Checking whether docker can run container ...

Checking iptables default rule ...

Checking br_netfilter module ...

Checking sysctl variables ...

Enabling kubelet ...

Created symlink from /etc/systemd/system/multi-user.target.wants/kubelet.service to /etc/systemd/system/kubelet.service.

Check successful, ready to run 'join' command ...

[preflight] Running pre-flight checks.

[validation] WARNING: kubeadm doesn't fully support multiple API Servers yet

[discovery] Trying to connect to API Server "192.168.99.100:6443"

[discovery] Trying to connect to API Server "192.168.99.100:6443"

[discovery] Created cluster-info discovery client, requesting info from "https://192.168.99.100:6443"

[discovery] Created cluster-info discovery client, requesting info from "https://192.168.99.100:6443"

[discovery] Requesting info from "https://192.168.99.100:6443" again to validate TLS against the pinned public key

[discovery] Cluster info signature and contents are valid and TLS certificate validates against pinned roots, will use API Server "192.168.99.100:6443"

[discovery] Successfully established connection with API Server "192.168.99.100:6443"

[discovery] Requesting info from "https://192.168.99.100:6443" again to validate TLS against the pinned public key

[discovery] Cluster info signature and contents are valid and TLS certificate validates against pinned roots, will use API Server "192.168.99.100:6443"

[discovery] Successfully established connection with API Server "192.168.99.100:6443"

This node has joined the cluster:

* Certificate signing request was sent to master and a response

  was received.

* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the master to see this node join the cluster.

/vagrant/scripts/kubeadm-setup-worker.sh: Worker node ready

Repeat the previous steps to set up the second worker node (worker2).

$ vagrant up worker2

Bringing machine 'worker2' up with 'virtualbox' provider...

...

$ vagrant ssh worker2

...

[vagrant@worker2 ~]$ su root

[root@worker2 vagrant]# /vagrant/scripts/kubeadm-setup-worker.sh

...

Validate the Kubernetes Cluster Configuration

Now the cluster is ready. You log into the master node to verify your cluster setup.

$ vagrant ssh master

[vagrant@master ~]$ kubectl cluster-info

Kubernetes master is running at https://192.168.99.100:6443

KubeDNS is running at https://192.168.99.100:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

[vagrant@master ~]$ kubectl get nodes

NAME                 STATUS    ROLES     AGE       VERSION

master.vagrant.vm    Ready     master    49m       v1.9.1+2.0.2.el7

worker1.vagrant.vm   Ready     <none>    24m       v1.9.1+2.0.2.el7

worker2.vagrant.vm   Ready     <none>    6m        v1.9.1+2.0.2.el7

[vagrant@master ~]$ kubectl get pods --namespace=kube-system

NAME                                        READY     STATUS             RESTARTS   AGE

etcd-master.vagrant.vm                      1/1       Running            0          54m

kube-apiserver-master.vagrant.vm            1/1       Running            0          54m

kube-controller-manager-master.vagrant.vm   1/1       Running            0          55m

kube-dns-855949bbf-fdtmq                    3/3       Running            0          55m

kube-flannel-ds-5t29z                       1/1       Running            0          13m

kube-flannel-ds-78x8g                       1/1       Running            0          55m

kube-flannel-ds-qnc9g                       1/1       Running            0          30m

kube-proxy-2rxzw                            1/1       Running            0          30m

kube-proxy-lpjxc                            1/1       Running            0          55m

kube-proxy-zdvj6                            1/1       Running            0          13m

kube-scheduler-master.vagrant.vm            1/1       Running            0          55m

kubernetes-dashboard-7c966ddf6d-sjks4       0/1       ImagePullBackOff   0          55m

and you can see the virtual machine running status from Oracle VM VirtualBox Manager user interface.

VirtualBox-Vagrant-Box-K8S.png

Customize Vagrantfile

The Vagrantfile from the GitHub can be used as-is; there are a couple of parameters you can set to tailor the installation to your needs.

  • NB_WORKERS (default: 2): the number of worker nodes to provision.
  • USE_PREVIEW (default: true): when true, Vagrant provisioning script will use the Oracle Linux 7 Preview and Add-ons channels for both Docker Engine and Kubernetes (latest version is select by yum). Otherwhise it will only use the Add-ons channel.
  • MANAGE_FROM_HOST (default: false): when true, Vagrant will bind port 6443 from the master node to the host. This allows you to manage the cluster from the host itself using the generated admin.conf file (assuming kubectl is installed on the host).
  • BIND_PROXY (default: true): when true, Vagrant will bind the Kubernetes Proxy port from the master node to the host. Useful to access the Dashboard or any other application from outside the cluster. It is an easier alternative to ssh tunnel.
  • MEMORY (default: 2048): all VMs are provisioned with 2GB memory. This can be slightly reduced if memory is a concern.

Additional Information

For additional information about Oracle Linux, visit

Comments
Post Details
Added on Mar 25 2018
7 comments
3,421 views