Introduction
Recently we announced the Oracle Vagrant boxes GitHub repository. Now you have a more streamlined way to create virtual machines with Oracle software fully configured and ready to go inside of them. With Oracle VM VirtualBox, Oracle Linux and Vagrant, you can easily build a consistent workflow to create disposable Linux environment whose configuration and dependencies are isolated from your development machine on every platform (Windows, MacOS or Linux).
The example below is to show how simple to use Vagrant to set up Oracle Container Services for use with Kubernetes.
Install Required Software
Before you get started, install the required software:
Clone Oracle Vagrant GitHub Repository
If you haven't cloned or downloaded the GitHub repository, perform git clone to prepare for your own Vagrant boxes.
$ git clone https://github.com/oracle/vagrant-boxes
Cloning into 'vagrant-boxes'...
remote: Counting objects: 342, done.
remote: Compressing objects: 100% (58/58), done.
remote: Total 342 (delta 42), reused 71 (delta 31), pack-reused 249
Receiving objects: 100% (342/342), 69.52 KiB | 827.00 KiB/s, done.
Resolving deltas: 100% (170/170), done.
Setup Master Node
Go the specific sub-folder.
$ cd vagrant-boxes/Kubernetes/
The Vagrantfile will provision a Kubernetes cluster with one master and n worker nodes. This example shows 2 worker nodes.
First we'll set up the master node.
$ vagrant up master
Bringing machine 'master' up with 'virtualbox' provider...
==> master: Importing base box 'ol7-latest'...
==> master: Matching MAC address for NAT networking...
==> master: Setting the name of the VM: Kubernetes_master_1521995861580_23054
==> master: Clearing any previously set network interfaces...
==> master: Preparing network interfaces based on configuration...
master: Adapter 1: nat
master: Adapter 2: hostonly
==> master: Forwarding ports...
master: 8001 (guest) => 8001 (host) (adapter 1)
master: 22 (guest) => 2222 (host) (adapter 1)
==> master: Running 'pre-boot' VM customizations...
==> master: Booting VM...
...
...==> master: Running provisioner: shell...
master: Running: C:/Software/Cygwin64/tmp/vagrant-shell20180325-8000-ch4j4k.sh
master: Installing and configuring Docker Engine
master: Package btrfs-progs-4.9.1-1.0.2.el7.x86_64 already installed and latest version
master: Resolving Dependencies
master: --> Running transaction check
master: ---> Package docker-engine.x86_64 0:17.12.0.ol-1.0.1.el7 will be installed
...
...
master: Dependencies Resolved
master:
master: ================================================================================
master: Package Arch Version Repository Size
master: ================================================================================
master: Installing:
master: docker-engine x86_64 17.12.0.ol-1.0.1.el7 ol7_preview 30 M
master: Installing for dependencies:
master: audit-libs-python x86_64 2.7.6-3.el7 ol7_latest 73 k
master: checkpolicy x86_64 2.5-4.el7 ol7_latest 290 k
master: container-selinux noarch 2:2.21-1.el7 ol7_addons 28 k
master: libcgroup x86_64 0.41-13.el7 ol7_latest 64 k
master: libsemanage-python x86_64 2.5-8.el7 ol7_latest 104 k
master: libtool-ltdl x86_64 2.4.2-22.el7_3 ol7_latest 48 k
master: policycoreutils-python x86_64 2.5-17.1.0.1.el7 ol7_latest 445 k
master: python-IPy noarch 0.75-6.el7 ol7_latest 32 k
master: setools-libs x86_64 3.3.8-1.1.el7 ol7_latest 611 k
master:
master: Transaction Summary
master: ================================================================================
master: Install 1 Package (+9 Dependent packages)
master: Total download size: 32 M
master: Installed size: 128 M
master: Downloading packages:
master: --------------------------------------------------------------------------------
master: Total 7.3 MB/s | 32 MB 00:04
...
master: Installed:
master: docker-engine.x86_64 0:17.12.0.ol-1.0.1.el7
master:
master: Dependency Installed:
... master: Complete!
master: Creating 'btrfs' file system on: /dev/sdb
master: Created symlink from /etc/systemd/system/multi-user.target.wants/docker.service to /usr/lib/systemd/system/docker.service.
master: Installing and configuring Kubernetes packages
master: Resolving Dependencies
...
master: Dependencies Resolved
master:
master: ================================================================================
master: Package Arch Version Repository Size
master: ================================================================================
master: Installing:
master: kubeadm x86_64 1.9.1-2.0.2.el7 ol7_addons 17 M
master: Installing for dependencies:
master: kubectl x86_64 1.9.1-2.0.2.el7 ol7_addons 8.9 M
master: kubelet x86_64 1.9.1-2.0.2.el7 ol7_addons 17 M
master: kubernetes-cni x86_64 0.6.0-2.0.1.el7 ol7_addons 797 k
master: kubernetes-cni-plugins x86_64 0.6.0-2.0.1.el7 ol7_addons 8.5 M
master: socat x86_64 1.7.3.2-2.el7 ol7_latest 289 k
master:
master: Transaction Summary
master: ================================================================================
master: Install 1 Package (+5 Dependent packages)
master:
master: Total download size: 52 M
master: Installed size: 279 M
master: Downloading packages:
master: --------------------------------------------------------------------------------
master: Total 5.0 MB/s | 52 MB 00:10
... master: Installed:
master: kubeadm.x86_64 0:1.9.1-2.0.2.el7
master:
...
master: Complete!
master: net.bridge.bridge-nf-call-ip6tables = 1
master: net.bridge.bridge-nf-call-iptables = 1
master: Your Kubernetes VM is ready to use!
==> master: Configuring proxy for Docker...
==> master: Running provisioner: shell...
master: Running: inline script
==> master: Configuring proxy for Docker...
==> master: Running provisioner: shell...
master: Running: inline script
==> master: Configuring proxy for Docker...
Login to the master node virtual machine and run as root to configure the master node. In this step, you will be asked to sign into the Oracle Container Registry.
$ vagrant ssh master
Welcome to Oracle Linux Server release 7.4 (GNU/Linux 4.1.12-112.16.4.el7uek.x86_64)
The Oracle Linux End-User License Agreement can be viewed here:
* /usr/share/eula/eula.en_US
For additional packages, updates, documentation and community help, see:
* http://yum.oracle.com/
[vagrant@master ~]$ su root
[root@master vagrant]# /vagrant/scripts/kubeadm-setup-master.sh
/vagrant/scripts/kubeadm-setup-master.sh: Login to container registry
Username: honglin.su@oracle.com
Password:
Login Succeeded
/vagrant/scripts/kubeadm-setup-master.sh: Setup Master node
Starting to initialize master node ...
Checking if env is ready ...
Checking whether docker can pull busybox image ...
Checking access to container-registry.oracle.com/kubernetes ...
v1.9.1: Pulling from kubernetes/kube-proxy-amd64
Digest: sha256:852fbdc6be8b357356c047bd9649e1c62f572c0e61a0526cd048c0d0dc675e4d
Status: Image is up to date for container-registry.oracle.com/kubernetes/kube-proxy-amd64:v1.9.1
Checking whether docker can run container ...
Checking iptables default rule ...
Checking br_netfilter module ...
Checking sysctl variables ...
Enabling kubelet ...
Created symlink from /etc/systemd/system/multi-user.target.wants/kubelet.service to /etc/systemd/system/kubelet.service.
Check successful, ready to run 'up' command ...
Waiting for kubeadm to setup master cluster...
Please wait ...
\ - 75% completed
Waiting for the control plane to become ready ...
...............
100% completed
clusterrole "flannel" created
clusterrolebinding "flannel" created
serviceaccount "flannel" created
configmap "kube-flannel-cfg" created
daemonset "kube-flannel-ds" created
Installing kubernetes-dashboard ...
.../vagrant/scripts/kubeadm-setup-master.sh: Copying admin.conf for vagrant user
/vagrant/scripts/kubeadm-setup-master.sh: Copying admin.conf into host directory
/vagrant/scripts/kubeadm-setup-master.sh: Saving token for worker nodes
/vagrant/scripts/kubeadm-setup-master.sh: Master node ready, run
/vagrant/scripts/kubeadm-setup-worker.sh
on the worker nodes
[root@master vagrant]# exit
Setup Worker Nodes
Go back to your development environment and setup the first worker node (worker1).
$ vagrant up worker1
Bringing machine 'worker1' up with 'virtualbox' provider...
==> worker1: Importing base box 'ol7-latest'...
==> worker1: Matching MAC address for NAT networking...
==> worker1: Setting the name of the VM: Kubernetes_worker1_1521998480587_67371
==> worker1: Fixed port collision for 22 => 2222. Now on port 2200.
==> worker1: Clearing any previously set network interfaces...
==> worker1: Preparing network interfaces based on configuration...
worker1: Adapter 1: nat
worker1: Adapter 2: hostonly
==> worker1: Forwarding ports...
worker1: 22 (guest) => 2200 (host) (adapter 1)
==> worker1: Running 'pre-boot' VM customizations...
==> worker1: Booting VM...
...
worker1: Package btrfs-progs-4.9.1-1.0.2.el7.x86_64 already installed and latest version
worker1: Resolving Dependencies
worker1: --> Running transaction check
worker1: ---> Package docker-engine.x86_64 0:17.12.0.ol-1.0.1.el7 will be installed
worker1: Dependencies Resolved
...
worker1:
worker1: ================================================================================
worker1: Package Arch Version Repository Size
worker1: ================================================================================
worker1: Installing:
worker1: docker-engine x86_64 17.12.0.ol-1.0.1.el7 ol7_preview 30 M
worker1: Installing for dependencies:
worker1: audit-libs-python x86_64 2.7.6-3.el7 ol7_latest 73 k
worker1: checkpolicy x86_64 2.5-4.el7 ol7_latest 290 k
worker1: container-selinux noarch 2:2.21-1.el7 ol7_addons 28 k
worker1: libcgroup x86_64 0.41-13.el7 ol7_latest 64 k
worker1: libsemanage-python x86_64 2.5-8.el7 ol7_latest 104 k
worker1: libtool-ltdl x86_64 2.4.2-22.el7_3 ol7_latest 48 k
worker1: policycoreutils-python x86_64 2.5-17.1.0.1.el7 ol7_latest 445 k
worker1: python-IPy noarch 0.75-6.el7 ol7_latest 32 k
worker1: setools-libs x86_64 3.3.8-1.1.el7 ol7_latest 611 k
worker1:
worker1: Transaction Summary
worker1: ================================================================================
worker1: Install 1 Package (+9 Dependent packages)
worker1: Total download size: 32 M
worker1: Installed size: 128 M
worker1: Downloading packages:
worker1: --------------------------------------------------------------------------------
worker1: Total 8.6 MB/s | 32 MB 00:03
...
worker1: Installed:
worker1: docker-engine.x86_64 0:17.12.0.ol-1.0.1.el7
worker1:
worker1: Dependency Installed:
...
worker1: Complete!
worker1: Creating 'btrfs' file system on: /dev/sdb
worker1: Created symlink from /etc/systemd/system/multi-user.target.wants/docker.service to /usr/lib/systemd/system/docker.service.
worker1: Installing and configuring Kubernetes packages
worker1: Resolving Dependencies
worker1: --> Running transaction check
worker1: ---> Package kubeadm.x86_64 0:1.9.1-2.0.2.el7 will be installed
worker1: Dependencies Resolved
...
worker1:
worker1: ================================================================================
worker1: Package Arch Version Repository Size
worker1: ================================================================================
worker1: Installing:
worker1: kubeadm x86_64 1.9.1-2.0.2.el7 ol7_addons 17 M
worker1: Installing for dependencies:
worker1: kubectl x86_64 1.9.1-2.0.2.el7 ol7_addons 8.9 M
worker1: kubelet x86_64 1.9.1-2.0.2.el7 ol7_addons 17 M
worker1: kubernetes-cni x86_64 0.6.0-2.0.1.el7 ol7_addons 797 k
worker1: kubernetes-cni-plugins x86_64 0.6.0-2.0.1.el7 ol7_addons 8.5 M
worker1: socat x86_64 1.7.3.2-2.el7 ol7_latest 289 k
worker1:
worker1: Transaction Summary
worker1: ================================================================================
worker1: Install 1 Package (+5 Dependent packages)
worker1: Total download size: 52 M
worker1: Installed size: 279 M
worker1: Downloading packages:
worker1: --------------------------------------------------------------------------------
worker1: Total 17 MB/s | 52 MB 00:03
...
worker1: Installed:
worker1: kubeadm.x86_64 0:1.9.1-2.0.2.el7
worker1:
worker1: Dependency Installed:
worker1: kubectl.x86_64 0:1.9.1-2.0.2.el7
worker1: kubelet.x86_64 0:1.9.1-2.0.2.el7
worker1: kubernetes-cni.x86_64 0:0.6.0-2.0.1.el7
worker1: kubernetes-cni-plugins.x86_64 0:0.6.0-2.0.1.el7
worker1: socat.x86_64 0:1.7.3.2-2.el7
worker1: Complete!
worker1: net.bridge.bridge-nf-call-ip6tables = 1
worker1: net.bridge.bridge-nf-call-iptables = 1
worker1: Your Kubernetes VM is ready to use!
==> worker1: Configuring proxy for Docker...
Log into the first worker node and configure the worker node. Similarly, you will be asked to sign into the Oracle Container Registry.
$ vagrant ssh worker1
Welcome to Oracle Linux Server release 7.4 (GNU/Linux 4.1.12-112.16.4.el7uek.x86_64)
The Oracle Linux End-User License Agreement can be viewed here:
* /usr/share/eula/eula.en_US
For additional packages, updates, documentation and community help, see:
* http://yum.oracle.com/
[vagrant@worker1 ~]$ su root
[root@worker1 vagrant]# /vagrant/scripts/kubeadm-setup-worker.sh
/vagrant/scripts/kubeadm-setup-worker.sh: Login to container registry
Username: honglin.su@oracle.com
Password:
Login Succeeded
/vagrant/scripts/kubeadm-setup-worker.sh: Setup Worker node
Starting to initialize worker node ...
Checking if env is ready ...
Checking whether docker can pull busybox image ...
Checking access to container-registry.oracle.com/kubernetes ...
v1.9.1: Pulling from kubernetes/kube-proxy-amd64
Digest: sha256:852fbdc6be8b357356c047bd9649e1c62f572c0e61a0526cd048c0d0dc675e4d
Status: Image is up to date for container-registry.oracle.com/kubernetes/kube-proxy-amd64:v1.9.1
Checking whether docker can run container ...
Checking iptables default rule ...
Checking br_netfilter module ...
Checking sysctl variables ...
Enabling kubelet ...
Created symlink from /etc/systemd/system/multi-user.target.wants/kubelet.service to /etc/systemd/system/kubelet.service.
Check successful, ready to run 'join' command ...
[preflight] Running pre-flight checks.
[validation] WARNING: kubeadm doesn't fully support multiple API Servers yet
[discovery] Trying to connect to API Server "192.168.99.100:6443"
[discovery] Trying to connect to API Server "192.168.99.100:6443"
[discovery] Created cluster-info discovery client, requesting info from "https://192.168.99.100:6443"
[discovery] Created cluster-info discovery client, requesting info from "https://192.168.99.100:6443"
[discovery] Requesting info from "https://192.168.99.100:6443" again to validate TLS against the pinned public key
[discovery] Cluster info signature and contents are valid and TLS certificate validates against pinned roots, will use API Server "192.168.99.100:6443"
[discovery] Successfully established connection with API Server "192.168.99.100:6443"
[discovery] Requesting info from "https://192.168.99.100:6443" again to validate TLS against the pinned public key
[discovery] Cluster info signature and contents are valid and TLS certificate validates against pinned roots, will use API Server "192.168.99.100:6443"
[discovery] Successfully established connection with API Server "192.168.99.100:6443"
This node has joined the cluster:
* Certificate signing request was sent to master and a response
was received.
* The Kubelet was informed of the new secure connection details.
Run 'kubectl get nodes' on the master to see this node join the cluster.
/vagrant/scripts/kubeadm-setup-worker.sh: Worker node ready
Repeat the previous steps to set up the second worker node (worker2).
$ vagrant up worker2
Bringing machine 'worker2' up with 'virtualbox' provider...
...
$ vagrant ssh worker2
...
[vagrant@worker2 ~]$ su root
[root@worker2 vagrant]# /vagrant/scripts/kubeadm-setup-worker.sh
...
Validate the Kubernetes Cluster Configuration
Now the cluster is ready. You log into the master node to verify your cluster setup.
$ vagrant ssh master
[vagrant@master ~]$ kubectl cluster-info
Kubernetes master is running at https://192.168.99.100:6443
KubeDNS is running at https://192.168.99.100:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
[vagrant@master ~]$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
master.vagrant.vm Ready master 49m v1.9.1+2.0.2.el7
worker1.vagrant.vm Ready <none> 24m v1.9.1+2.0.2.el7
worker2.vagrant.vm Ready <none> 6m v1.9.1+2.0.2.el7
[vagrant@master ~]$ kubectl get pods --namespace=kube-system
NAME READY STATUS RESTARTS AGE
etcd-master.vagrant.vm 1/1 Running 0 54m
kube-apiserver-master.vagrant.vm 1/1 Running 0 54m
kube-controller-manager-master.vagrant.vm 1/1 Running 0 55m
kube-dns-855949bbf-fdtmq 3/3 Running 0 55m
kube-flannel-ds-5t29z 1/1 Running 0 13m
kube-flannel-ds-78x8g 1/1 Running 0 55m
kube-flannel-ds-qnc9g 1/1 Running 0 30m
kube-proxy-2rxzw 1/1 Running 0 30m
kube-proxy-lpjxc 1/1 Running 0 55m
kube-proxy-zdvj6 1/1 Running 0 13m
kube-scheduler-master.vagrant.vm 1/1 Running 0 55m
kubernetes-dashboard-7c966ddf6d-sjks4 0/1 ImagePullBackOff 0 55m
and you can see the virtual machine running status from Oracle VM VirtualBox Manager user interface.
Customize Vagrantfile
The Vagrantfile from the GitHub can be used as-is; there are a couple of parameters you can set to tailor the installation to your needs.
NB_WORKERS (default: 2): the number of worker nodes to provision.USE_PREVIEW (default: true): when true, Vagrant provisioning script will use the Oracle Linux 7 Preview and Add-ons channels for both Docker Engine and Kubernetes (latest version is select by yum). Otherwhise it will only use the Add-ons channel.MANAGE_FROM_HOST (default: false): when true, Vagrant will bind port 6443 from the master node to the host. This allows you to manage the cluster from the host itself using the generated admin.conf file (assuming kubectl is installed on the host).BIND_PROXY (default: true): when true, Vagrant will bind the Kubernetes Proxy port from the master node to the host. Useful to access the Dashboard or any other application from outside the cluster. It is an easier alternative to ssh tunnel.MEMORY (default: 2048): all VMs are provisioned with 2GB memory. This can be slightly reduced if memory is a concern.
Additional Information