Skip to Main Content

Infrastructure Software

Announcement

For appeals, questions and feedback about Oracle Forums, please email oracle-forums-moderators_us@oracle.com. Please ask technical questions in the appropriate category. Thank you!

Use Vagrant and VirtualBox to setup Oracle Container Services for use with Kubernetes

Honglin Su-OracleMar 25 2018 — edited Mar 25 2018

Introduction

Recently we announced the Oracle Vagrant boxes GitHub repository. Now you have a more streamlined way to create virtual machines with Oracle software fully configured and ready to go inside of them. With Oracle VM VirtualBox, Oracle Linux and Vagrant, you can easily build a consistent workflow to create disposable Linux environment whose configuration and dependencies are isolated from your development machine on every platform (Windows, MacOS or Linux).

The example below is to show how simple to use Vagrant to set up Oracle Container Services for use with Kubernetes.

Install Required Software

Before you get started, install the required software:

Clone Oracle Vagrant GitHub Repository

If you haven't cloned or downloaded the GitHub repository, perform git clone to prepare for your own Vagrant boxes.

$ git clone https://github.com/oracle/vagrant-boxes

Cloning into 'vagrant-boxes'...

remote: Counting objects: 342, done.

remote: Compressing objects: 100% (58/58), done.

remote: Total 342 (delta 42), reused 71 (delta 31), pack-reused 249

Receiving objects: 100% (342/342), 69.52 KiB | 827.00 KiB/s, done.

Resolving deltas: 100% (170/170), done.

Setup Master Node

Go the specific sub-folder.

$ cd vagrant-boxes/Kubernetes/

The Vagrantfile will provision a Kubernetes cluster with one master and n worker nodes. This example shows 2 worker nodes.

First we'll set up the master node.

$ vagrant up master

Bringing machine 'master' up with 'virtualbox' provider...

==> master: Importing base box 'ol7-latest'...

==> master: Matching MAC address for NAT networking...

==> master: Setting the name of the VM: Kubernetes_master_1521995861580_23054

==> master: Clearing any previously set network interfaces...

==> master: Preparing network interfaces based on configuration...

master: Adapter 1: nat

master: Adapter 2: hostonly

==> master: Forwarding ports...

master: 8001 (guest) => 8001 (host) (adapter 1)

master: 22 (guest) => 2222 (host) (adapter 1)

==> master: Running 'pre-boot' VM customizations...

==> master: Booting VM...

...
...

==> master: Running provisioner: shell...

master: Running: C:/Software/Cygwin64/tmp/vagrant-shell20180325-8000-ch4j4k.sh

master: Installing and configuring Docker Engine

master: Package btrfs-progs-4.9.1-1.0.2.el7.x86\_64 already installed and latest version

master: Resolving Dependencies

master: --> Running transaction check

master: ---> Package docker-engine.x86\_64 0:17.12.0.ol-1.0.1.el7 will be installed

...

...

    master: Dependencies Resolved

master:

master: ================================================================================

master:  Package                  Arch     Version                  Repository     Size

master: ================================================================================

master: Installing:

master:  docker-engine            x86\_64   17.12.0.ol-1.0.1.el7     ol7\_preview    30 M

master: Installing for dependencies:

master:  audit-libs-python        x86\_64   2.7.6-3.el7              ol7\_latest     73 k

master:  checkpolicy              x86\_64   2.5-4.el7                ol7\_latest    290 k

master:  container-selinux        noarch   2:2.21-1.el7             ol7\_addons     28 k

master:  libcgroup                x86\_64   0.41-13.el7              ol7\_latest     64 k

master:  libsemanage-python       x86\_64   2.5-8.el7                ol7\_latest    104 k

master:  libtool-ltdl             x86\_64   2.4.2-22.el7\_3           ol7\_latest     48 k

master:  policycoreutils-python   x86\_64   2.5-17.1.0.1.el7         ol7\_latest    445 k

master:  python-IPy               noarch   0.75-6.el7               ol7\_latest     32 k

master:  setools-libs             x86\_64   3.3.8-1.1.el7            ol7\_latest    611 k

master:

master: Transaction Summary

master: ================================================================================

master: Install  1 Package (+9 Dependent packages)

master: Total download size: 32 M

master: Installed size: 128 M

master: Downloading packages:

master: --------------------------------------------------------------------------------

master: Total                                              7.3 MB/s |  32 MB  00:04

...

master: Installed:

master:   docker-engine.x86\_64 0:17.12.0.ol-1.0.1.el7

master:

master: Dependency Installed:

...

master: Complete!

master: Creating 'btrfs' file system on: /dev/sdb

master: Created symlink from /etc/systemd/system/multi-user.target.wants/docker.service to /usr/lib/systemd/system/docker.service.

master: Installing and configuring Kubernetes packages

master: Resolving Dependencies

...

master: Dependencies Resolved

master:

master: ================================================================================

master:  Package                    Arch       Version             Repository      Size

master: ================================================================================

master: Installing:

master:  kubeadm                    x86\_64     1.9.1-2.0.2.el7     ol7\_addons      17 M

master: Installing for dependencies:

master:  kubectl                    x86\_64     1.9.1-2.0.2.el7     ol7\_addons     8.9 M

master:  kubelet                    x86\_64     1.9.1-2.0.2.el7     ol7\_addons      17 M

master:  kubernetes-cni             x86\_64     0.6.0-2.0.1.el7     ol7\_addons     797 k

master:  kubernetes-cni-plugins     x86\_64     0.6.0-2.0.1.el7     ol7\_addons     8.5 M

master:  socat                      x86\_64     1.7.3.2-2.el7       ol7\_latest     289 k

master:

master: Transaction Summary

master: ================================================================================

master: Install  1 Package (+5 Dependent packages)

master:

master: Total download size: 52 M

master: Installed size: 279 M

master: Downloading packages:

master: --------------------------------------------------------------------------------

master: Total                                              5.0 MB/s |  52 MB  00:10

...

master: Installed:

master:   kubeadm.x86\_64 0:1.9.1-2.0.2.el7

master:

  ...  

master: Complete!

master: net.bridge.bridge-nf-call-ip6tables = 1

master: net.bridge.bridge-nf-call-iptables = 1

master: Your Kubernetes VM is ready to use!

==> master: Configuring proxy for Docker...

==> master: Running provisioner: shell...

master: Running: inline script

==> master: Configuring proxy for Docker...

==> master: Running provisioner: shell...

master: Running: inline script

==> master: Configuring proxy for Docker...

Login to the master node virtual machine and run as root to configure the master node. In this step, you will be asked to sign into the Oracle Container Registry.

$ vagrant ssh master

Welcome to Oracle Linux Server release 7.4 (GNU/Linux 4.1.12-112.16.4.el7uek.x86_64)

The Oracle Linux End-User License Agreement can be viewed here:

\* /usr/share/eula/eula.en\_US

For additional packages, updates, documentation and community help, see:

\* [http://yum.oracle.com/](http://yum.oracle.com/)

[vagrant@master ~]$ su root

[root@master vagrant]# /vagrant/scripts/kubeadm-setup-master.sh

/vagrant/scripts/kubeadm-setup-master.sh: Login to container registry

Username: honglin.su@oracle.com

Password:

Login Succeeded

/vagrant/scripts/kubeadm-setup-master.sh: Setup Master node

Starting to initialize master node ...

Checking if env is ready ...

Checking whether docker can pull busybox image ...

Checking access to container-registry.oracle.com/kubernetes ...

v1.9.1: Pulling from kubernetes/kube-proxy-amd64

Digest: sha256:852fbdc6be8b357356c047bd9649e1c62f572c0e61a0526cd048c0d0dc675e4d

Status: Image is up to date for container-registry.oracle.com/kubernetes/kube-proxy-amd64:v1.9.1

Checking whether docker can run container ...

Checking iptables default rule ...

Checking br_netfilter module ...

Checking sysctl variables ...

Enabling kubelet ...

Created symlink from /etc/systemd/system/multi-user.target.wants/kubelet.service to /etc/systemd/system/kubelet.service.

Check successful, ready to run 'up' command ...

Waiting for kubeadm to setup master cluster...

Please wait ...

\ - 75% completed

Waiting for the control plane to become ready ...

...............

100% completed

clusterrole "flannel" created

clusterrolebinding "flannel" created

serviceaccount "flannel" created

configmap "kube-flannel-cfg" created

daemonset "kube-flannel-ds" created

Installing kubernetes-dashboard ...

...

/vagrant/scripts/kubeadm-setup-master.sh: Copying admin.conf for vagrant user

/vagrant/scripts/kubeadm-setup-master.sh: Copying admin.conf into host directory

/vagrant/scripts/kubeadm-setup-master.sh: Saving token for worker nodes

/vagrant/scripts/kubeadm-setup-master.sh: Master node ready, run

    /vagrant/scripts/kubeadm-setup-worker.sh

on the worker nodes

[root@master vagrant]# exit

Setup Worker Nodes

Go back to your development environment and setup the first worker node (worker1).

$ vagrant up worker1

Bringing machine 'worker1' up with 'virtualbox' provider...

==> worker1: Importing base box 'ol7-latest'...

==> worker1: Matching MAC address for NAT networking...

==> worker1: Setting the name of the VM: Kubernetes_worker1_1521998480587_67371

==> worker1: Fixed port collision for 22 => 2222. Now on port 2200.

==> worker1: Clearing any previously set network interfaces...

==> worker1: Preparing network interfaces based on configuration...

worker1: Adapter 1: nat

worker1: Adapter 2: hostonly

==> worker1: Forwarding ports...

worker1: 22 (guest) => 2200 (host) (adapter 1)

==> worker1: Running 'pre-boot' VM customizations...

==> worker1: Booting VM...

    ...  

worker1: Package btrfs-progs-4.9.1-1.0.2.el7.x86\_64 already installed and latest version

worker1: Resolving Dependencies

worker1: --> Running transaction check

worker1: ---> Package docker-engine.x86\_64 0:17.12.0.ol-1.0.1.el7 will be installed

worker1: Dependencies Resolved

...

worker1:

worker1: ================================================================================

worker1:  Package                  Arch     Version                  Repository     Size

worker1: ================================================================================

worker1: Installing:

worker1:  docker-engine            x86\_64   17.12.0.ol-1.0.1.el7     ol7\_preview    30 M

worker1: Installing for dependencies:

worker1:  audit-libs-python        x86\_64   2.7.6-3.el7              ol7\_latest     73 k

worker1:  checkpolicy              x86\_64   2.5-4.el7                ol7\_latest    290 k

worker1:  container-selinux        noarch   2:2.21-1.el7             ol7\_addons     28 k

worker1:  libcgroup                x86\_64   0.41-13.el7              ol7\_latest     64 k

worker1:  libsemanage-python       x86\_64   2.5-8.el7                ol7\_latest    104 k

worker1:  libtool-ltdl             x86\_64   2.4.2-22.el7\_3           ol7\_latest     48 k

worker1:  policycoreutils-python   x86\_64   2.5-17.1.0.1.el7         ol7\_latest    445 k

worker1:  python-IPy               noarch   0.75-6.el7               ol7\_latest     32 k

worker1:  setools-libs             x86\_64   3.3.8-1.1.el7            ol7\_latest    611 k

worker1:

worker1: Transaction Summary

worker1: ================================================================================

worker1: Install  1 Package (+9 Dependent packages)

worker1: Total download size: 32 M

worker1: Installed size: 128 M

worker1: Downloading packages:

worker1: --------------------------------------------------------------------------------

worker1: Total                                              8.6 MB/s |  32 MB  00:03

...

worker1: Installed:

worker1:   docker-engine.x86\_64 0:17.12.0.ol-1.0.1.el7

worker1:

worker1: Dependency Installed:

   ...  

worker1: Complete!

worker1: Creating 'btrfs' file system on: /dev/sdb

worker1: Created symlink from /etc/systemd/system/multi-user.target.wants/docker.service to /usr/lib/systemd/system/docker.service.

worker1: Installing and configuring Kubernetes packages

worker1: Resolving Dependencies

worker1: --> Running transaction check

worker1: ---> Package kubeadm.x86\_64 0:1.9.1-2.0.2.el7 will be installed

worker1: Dependencies Resolved

...

worker1:

worker1: ================================================================================

worker1:  Package                    Arch       Version             Repository      Size

worker1: ================================================================================

worker1: Installing:

worker1:  kubeadm                    x86\_64     1.9.1-2.0.2.el7     ol7\_addons      17 M

worker1: Installing for dependencies:

worker1:  kubectl                    x86\_64     1.9.1-2.0.2.el7     ol7\_addons     8.9 M

worker1:  kubelet                    x86\_64     1.9.1-2.0.2.el7     ol7\_addons      17 M

worker1:  kubernetes-cni             x86\_64     0.6.0-2.0.1.el7     ol7\_addons     797 k

worker1:  kubernetes-cni-plugins     x86\_64     0.6.0-2.0.1.el7     ol7\_addons     8.5 M

worker1:  socat                      x86\_64     1.7.3.2-2.el7       ol7\_latest     289 k

worker1:

worker1: Transaction Summary

worker1: ================================================================================

worker1: Install  1 Package (+5 Dependent packages)

worker1: Total download size: 52 M

worker1: Installed size: 279 M

worker1: Downloading packages:

worker1: --------------------------------------------------------------------------------

worker1: Total                                               17 MB/s |  52 MB  00:03

...

worker1: Installed:

worker1:   kubeadm.x86\_64 0:1.9.1-2.0.2.el7

worker1:

worker1: Dependency Installed:

worker1:   kubectl.x86\_64 0:1.9.1-2.0.2.el7

worker1:   kubelet.x86\_64 0:1.9.1-2.0.2.el7

worker1:   kubernetes-cni.x86\_64 0:0.6.0-2.0.1.el7

worker1:   kubernetes-cni-plugins.x86\_64 0:0.6.0-2.0.1.el7

worker1:   socat.x86\_64 0:1.7.3.2-2.el7

worker1: Complete!

worker1: net.bridge.bridge-nf-call-ip6tables = 1

worker1: net.bridge.bridge-nf-call-iptables = 1

worker1: Your Kubernetes VM is ready to use!

==> worker1: Configuring proxy for Docker...

Log into the first worker node and configure the worker node. Similarly, you will be asked to sign into the Oracle Container Registry.

$ vagrant ssh worker1

Welcome to Oracle Linux Server release 7.4 (GNU/Linux 4.1.12-112.16.4.el7uek.x86_64)

The Oracle Linux End-User License Agreement can be viewed here:

\* /usr/share/eula/eula.en\_US

For additional packages, updates, documentation and community help, see:

\* [http://yum.oracle.com/](http://yum.oracle.com/)

[vagrant@worker1 ~]$ su root

[root@worker1 vagrant]# /vagrant/scripts/kubeadm-setup-worker.sh

/vagrant/scripts/kubeadm-setup-worker.sh: Login to container registry

Username: honglin.su@oracle.com

Password:

Login Succeeded

/vagrant/scripts/kubeadm-setup-worker.sh: Setup Worker node

Starting to initialize worker node ...

Checking if env is ready ...

Checking whether docker can pull busybox image ...

Checking access to container-registry.oracle.com/kubernetes ...

v1.9.1: Pulling from kubernetes/kube-proxy-amd64

Digest: sha256:852fbdc6be8b357356c047bd9649e1c62f572c0e61a0526cd048c0d0dc675e4d

Status: Image is up to date for container-registry.oracle.com/kubernetes/kube-proxy-amd64:v1.9.1

Checking whether docker can run container ...

Checking iptables default rule ...

Checking br_netfilter module ...

Checking sysctl variables ...

Enabling kubelet ...

Created symlink from /etc/systemd/system/multi-user.target.wants/kubelet.service to /etc/systemd/system/kubelet.service.

Check successful, ready to run 'join' command ...

[preflight] Running pre-flight checks.

[validation] WARNING: kubeadm doesn't fully support multiple API Servers yet

[discovery] Trying to connect to API Server "192.168.99.100:6443"

[discovery] Trying to connect to API Server "192.168.99.100:6443"

[discovery] Created cluster-info discovery client, requesting info from "https://192.168.99.100:6443"

[discovery] Created cluster-info discovery client, requesting info from "https://192.168.99.100:6443"

[discovery] Requesting info from "https://192.168.99.100:6443" again to validate TLS against the pinned public key

[discovery] Cluster info signature and contents are valid and TLS certificate validates against pinned roots, will use API Server "192.168.99.100:6443"

[discovery] Successfully established connection with API Server "192.168.99.100:6443"

[discovery] Requesting info from "https://192.168.99.100:6443" again to validate TLS against the pinned public key

[discovery] Cluster info signature and contents are valid and TLS certificate validates against pinned roots, will use API Server "192.168.99.100:6443"

[discovery] Successfully established connection with API Server "192.168.99.100:6443"

This node has joined the cluster:

* Certificate signing request was sent to master and a response

was received.

* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the master to see this node join the cluster.

/vagrant/scripts/kubeadm-setup-worker.sh: Worker node ready

Repeat the previous steps to set up the second worker node (worker2).

$ vagrant up worker2

Bringing machine 'worker2' up with 'virtualbox' provider...

...

$ vagrant ssh worker2

...

[vagrant@worker2 ~]$ su root

[root@worker2 vagrant]# /vagrant/scripts/kubeadm-setup-worker.sh

...

Validate the Kubernetes Cluster Configuration

Now the cluster is ready. You log into the master node to verify your cluster setup.

$ vagrant ssh master

[vagrant@master ~]$ kubectl cluster-info

Kubernetes master is running at https://192.168.99.100:6443

KubeDNS is running at https://192.168.99.100:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

[vagrant@master ~]$ kubectl get nodes

NAME STATUS ROLES AGE VERSION

master.vagrant.vm Ready master 49m v1.9.1+2.0.2.el7

worker1.vagrant.vm Ready <none> 24m v1.9.1+2.0.2.el7

worker2.vagrant.vm Ready <none> 6m v1.9.1+2.0.2.el7

[vagrant@master ~]$ kubectl get pods --namespace=kube-system

NAME READY STATUS RESTARTS AGE

etcd-master.vagrant.vm 1/1 Running 0 54m

kube-apiserver-master.vagrant.vm 1/1 Running 0 54m

kube-controller-manager-master.vagrant.vm 1/1 Running 0 55m

kube-dns-855949bbf-fdtmq 3/3 Running 0 55m

kube-flannel-ds-5t29z 1/1 Running 0 13m

kube-flannel-ds-78x8g 1/1 Running 0 55m

kube-flannel-ds-qnc9g 1/1 Running 0 30m

kube-proxy-2rxzw 1/1 Running 0 30m

kube-proxy-lpjxc 1/1 Running 0 55m

kube-proxy-zdvj6 1/1 Running 0 13m

kube-scheduler-master.vagrant.vm 1/1 Running 0 55m

kubernetes-dashboard-7c966ddf6d-sjks4 0/1 ImagePullBackOff 0 55m

and you can see the virtual machine running status from Oracle VM VirtualBox Manager user interface.

VirtualBox-Vagrant-Box-K8S.png

Customize Vagrantfile

The Vagrantfile from the GitHub can be used as-is; there are a couple of parameters you can set to tailor the installation to your needs.

  • NB_WORKERS (default: 2): the number of worker nodes to provision.
  • USE_PREVIEW (default: true): when true, Vagrant provisioning script will use the Oracle Linux 7 Preview and Add-ons channels for both Docker Engine and Kubernetes (latest version is select by yum). Otherwhise it will only use the Add-ons channel.
  • MANAGE_FROM_HOST (default: false): when true, Vagrant will bind port 6443 from the master node to the host. This allows you to manage the cluster from the host itself using the generated admin.conf file (assuming kubectl is installed on the host).
  • BIND_PROXY (default: true): when true, Vagrant will bind the Kubernetes Proxy port from the master node to the host. Useful to access the Dashboard or any other application from outside the cluster. It is an easier alternative to ssh tunnel.
  • MEMORY (default: 2048): all VMs are provisioned with 2GB memory. This can be slightly reduced if memory is a concern.

Additional Information

For additional information about Oracle Linux, visit

Comments
Post Details
Added on Mar 25 2018
7 comments
4,019 views