Installing a kubernetes cluster on CentOS

15th March 2019 0 By Jonny

Nothing too exciting here – there are plenty of guides and instructions on how to install kubernetes on CentOS already. For my particular purposes though, I’m going to veer off the usual pathway taken and run my cluster as follows:

  • 8 VMs running CentOS 7.6 split across 2 (oldish) Intel i5 NUCs
  • 2 master nodes and 6 worker nodes, evenly split across the host servers – running Fedora 29 and using KVM
  • cri-o used as the container runtime engine on all nodes
  • kubeadm used to create the cluster
  • weave used to provide the networking component

The NUCs each have 16GB of RAM and an internal SSD of 128GB. Each VM will have 4GB of RAM and 2vCPUs allocated. The root disk of each node will be a 20GB raw disk image. The nodes are going to be imaginatively named centos-k8s-X with X being a number between 1 and 8. Nodes 1 and 2 will be the master nodes with the remainder being the worker nodes. Any storage used by the pods will be provided from an NFS server.

… skipping through the boring part of installing CentOS. Remember to disable the swap partition though!

Master node configuration

Installing kubeadm

Starting with the master nodes, centos-k8s-1 and centos-k8s-2, the first task is add the kubernetes repository, install kubeadm, kubelet, and kubectl. This will follow the instructions at kubernetes.io very closely. On the two master nodes the following commands should be run:

cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
exclude=kube*
EOF
setenforce 0
sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config

It’s a bit disappointing that SELinux has to be disabled, but apparently that is the case ‘until SELinux support is improved in kubelet’. With the kubernetes repo configured (and SELinux disabled) the kubeadm, kubelet, and kubectl packages can be installed:

yum install -y kubelet kubeadm kubectl --disableexcludes=kubernetes
systemctl enable kubelet

Installing cri-o

The usual container run time engine used on kubernetes clusters is docker, or more accurately, containerd. However, cri-o is a Red Hat sponsored project that is also OCI compliant and is an incubating CNCF project. It intends to be a ‘kubernetes native’ engine, and given it is OCI compliant, it should just be a drop in replacement for containerd. The Red Hat product, OpenShift, also uses cri-o as it’s default runtime engine in it’s next release.

Fortunately, building cri-o from scratch is not required and there are existing packages available for use. The installation instructions are in detail on the kubernetes web site.

The following commands are run on the master nodes:

modprobe overlay
modprobe br_netfilter

Hmmm, if we’re going to need modules loaded, then perhaps they should be automatically loaded when the system boots. I have created the following file to load these modules on system start:

#cat /etc/modules-load.d/k8s.conf
Load modules for k8s
overlay
br_netfilter
cat > /etc/sysctl.d/99-kubernetes-cri.conf <<EOF
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-ip6tables = 1
EOF

sysctl --system

It’s very important that IP forwarding is enabled on the kubernetes servers, and it is also important that bridged networks pass IP traffic through iptables. Next the cri-o repository is set up and cri-o installed.

yum-config-manager --add-repo=https://cbs.centos.org/repos/paas7-crio-115-candidate/x86_64/os/

yum install --nogpgcheck cri-o
systemctl enable crio

The initial command requires that yum-utils is installed. This will install the version 1.13 build of cri-o which is in-step with the version of kubernetes being deployed. When kubernetes is upgraded it should be checked if there are updated builds of cri-o available to match.

I have not changed the default kubernetes port. The local firewall should also be configured to allow the kubernetes traffic on TCP ports 6443 and 10250:

firewall-cmd --add-port 6443/tcp --perm
firewall-cmd --add-port 10250/tcp --perm
systemctl reload firewalld

By default, crio will try to use the systemd cgroup interface, whereas kubelet is expecting to use the cgroupfs interface. I’ve chosen to reconfigure crio to use the cgroupfs option by editting the /etc/crio/crio.conf file and making the following change:

cat /etc/crio/crio.conf | grep cgroup
#cgroup_manager is the cgroup management implementation to be used
#cgroup_manager = "systemd"
cgroup_manager = "cgroupfs"

A further change is also needed to the /etc/crio/crio./conf file. When we install the weave CNI plugin the weave plugins will be installed in the /opt/cni/bin directory, however crio is configured by default to look in the /usr/libexec/cni directory. The configuration of /etc/crio/crio.conf is modified as follows:

# grep cni /etc/crio/crio.conf
network_dir = "/etc/cni/net.d/"
plugin_dir = "/usr/libexec/cni"
plugin_dir = "/opt/cni/bin"

The cgroup_manager = “systemd” has been commented out and replaced with cgroup_manager = “cgroupfs”. I have also restarted the crio daemon after making these changes.

First Master Node creation

Creating the kubernetes cluster with kubeadm

Given this will be a multi-master kubernetes cluster, it’s not as straightforward as launching kubeadm init to bootstrap the kubernetes cluster. In order to make use of having multiple masters, I have decided to put an nginx reverse proxy in front of the master nodes. Therefore my later access to the cluster via kubectl will be using the load balanced name (kube.domain.name) which resolves to the nginx host. I’ll come back to the nginx configuration a bit later. In order to bootstrap the cluster so that it uses the nginx load balancer a kubeadm-config.yaml needs to be created with the load balancer details. My file looks as follows:

# cat kubeadm-config.yaml
apiVersion: kubeadm.k8s.io/v1beta1
kind: ClusterConfiguration
kubernetesVersion: stable
apiServer:
certSANs:
- "kube.ervine.dev"
controlPlaneEndpoint: "kube.ervine.dev:6443"

The kubernetes cluster can now be bootstrapped (created) using the following command:

kubeadm init --config=kubeadm-config.yaml --ignore-preflight-errors=all --cri-socket=/var/run/crio/crio.sock

As can be seen, all preflight errors are to be ignored – primarily because the kubeadm init process will check for docker, and I’ve chosen not to use docker. I have also had to direct kubeadm where it can find the container run time interface, again because I’m not using docker.

Hopefully, if everything has worked as it should do, the kubeadm process will finish and output a join command for subsequent nodes. Don’t use this kubeadm command, but do keep a copy of it though. I still have to enable the weave networking, add the second master node, and then look to join the worker nodes.

[addons] Applied essential addon: kube-proxy
Your Kubernetes master has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
You can now join any number of machines by running the following on each node as root:
kubeadm join kube.ervine.dev:6443 --token 6vtrzf.hab5bjevftagnzea --discovery-token-ca-cert-hash sha256:121fd2bd1b53638e3e5c2f5037bf61fc228ccaf095577bf3e4c4ba20c60f6208

Excitingly, the master node is now installed and up and running.

# export KUBECONFIG=/etc/kubernetes/admin.conf
# kubectl get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
centos-k8s-1.ervine.dev NotReady master 2m32s v1.13.4 192.168.11.21 CentOS Linux 7 (Core) 3.10.0-957.5.1.el7.x86_64 cri-o://1.13.1-1.rhaos4.1.git2ac1ec7.el7

The node is listed as ‘NotReady’ because there is no networking added yet.

Install weave networking to the cluster

There are many choices of networking plugin to choose for kubernetes. There is no right or wrong choice here, and I have chosen to use weave, mainly because in the past I’ve used flannel and I wanted to try a change. Installing the weave CNI plugin is easy and straightforward:

# kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d '\n')"

# kubectl -n kube-system get pods -o wide

NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
coredns-86c58d9df4-cxxrx 0/1 Pending 0 7m4s
coredns-86c58d9df4-dg8m9 0/1 Pending 0 7m4s
etcd-centos-k8s-1.ervine.dev 1/1 Running 1 6m44s 192.168.11.211 centos-k8s-1.ervine.dev
kube-apiserver-centos-k8s-1.ervine.dev 1/1 Running 1 5m57s 192.168.11.21 centos-k8s-1.ervine.dev
kube-controller-manager-centos-k8s-1.ervine.dev 1/1 Running 1 6m47s 192.168.11.21 centos-k8s-1.ervine.dev
kube-proxy-4jnrr 1/1 Running 0 7m4s 192.168.11.21 centos-k8s-1.ervine.dev
kube-scheduler-centos-k8s-1.ervine.dev 1/1 Running 1 6m20s 192.168.11.21 centos-k8s-1.ervine.dev
weave-net-6gdw2 0/2 ContainerCreating 0 6s 192.168.11.21 centos-k8s-1.ervine.dev

The weave CNI plugin is creating pods on the node to enable the kubernetes networking. When subsequent nodes are added the weave pods will be added to them as well.

Subsequent Master Node configuration

Installing the first master node is where the heavy lifting is performed. Adding subsequent master nodes, whilst requiring some manual steps, is relatively straightforward. All the steps listed for the master node configuration should have been performed (the software installation, firewall configuration, and crio configuration changes made).

Copy the PKI material across

From the first master node the following PKI cryptographic files should be copied to the next master node to be joined to the cluster

/etc/kubernetes/pki/ca.crt
/etc/kubernetes/pki/ca.key
/etc/kubernetes/pki/sa.key
/etc/kubernetes/pki/sa.pub
/etc/kubernetes/pki/front-proxy-ca.crt
/etc/kubernetes/pki/front-proxy-ca.key
/etc/kubernetes/pki/etcd/ca.crt
/etc/kubernetes/pki/etcd/ca.key
/etc/kubernetes/admin.conf

The /etc/kubernetes/pki and /etc/kubernetes/pki/etcd directories will probably need to be manually created on the second (and subsequent) master nodes. For large numbers of master nodes it would make sense to script this copy process, however for the small number of master nodes I’m using, a one time manual copy is used. Once these files are copied into place, the new master node can be joined to the cluster with some modifications made to the join command that was printed earlier:

kubeadm join kube.ervine.dev:6443 --token 6vtrzf.hab5bjevftagnzea --discovery-token-ca-cert-hash sha256:121fd2bd1b53638e3e5c2f5037bf61fc228ccaf095577bf3e4c4ba20c60f6 --ignore-preflight-errors=all --crio-socket=/var/run/crio/crio.sock --experimental-control-plane

The addition of the ‘–experimental-control-place’ option tells kubeadm that this is a master node being added to an existing cluster. The same options as before to ignore preflight errors and directing kubeadm to the correct crio.sock file are used as well. All being well, the node will join the cluster and kubectl will then report two master nodes.

kubectl get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
centos-k8s-1.ervine.dev Ready master 3d4h v1.13.4 192.168.11.21 CentOS Linux 7 (Core) 3.10.0-957.5.1.el7.x86_64 cri-o://1.13.1-1.rhaos4.1.git2ac1ec7.el7
centos-k8s-2.ervine.dev Ready master 3d4h v1.13.4 192.168.11.22 CentOS Linux 7 (Core) 3.10.0-957.5.1.el7.x86_64 cri-o://1.13.1-1.rhaos4.1.git2ac1ec7.el7

Adding worker nodes to the cluster

Adding worker nodes to the cluster is very similar to adding the master nodes. The same preparatory steps need to be performed on the worker nodes:

  • Install kubeadm, kubectl, kubelet
  • Install and configure cri-o
  • Module and sysctl configuration
  • Firewall configuration

The only item worth noting is that port 6443 is not required on the worker nodes, as this is used by the kube-apiserver which is a master node function.

Performing the worker node join

The worker node is joined using the kubeadm join command, the syntax is the same as that to add the second master node except that –experimental-control-plane is not added as a join option (this node is not going to be a control plane after all):

kubeadm join kube.ervine.dev:6443 --token 6vtrzf.hab5bjevftagnzea --discovery-token-ca-cert-hash sha256:121fd2bd1b53638e3e5c2f5037bf61fc228ccaf095577bf3e4c4ba20c60f6 --ignore-preflight-errors=all --crio-socket=/var/run/crio/crio.sock

This command should complete relatively quickly and finish with output resembling the following:

This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.
Run 'kubectl get nodes' on the master to see this node join the cluster.

This command was performed on all of the worker nodes which meant that the cluster nodes were as follows:

# kubectl get nodes -o wide
NAME                        STATUS   ROLES    AGE    VERSION   INTERNAL-IP     EXTERNAL-IP   OS-IMAGE                KERNEL-VERSION              CONTAINER-RUNTIME

centos-k8s-1.ervine.dev Ready master 3d4h v1.13.4 192.168.11.21 CentOS Linux 7 (Core) 3.10.0-957.5.1.el7.x86_64 cri-o://1.13.1-1.rhaos4.1.git2ac1ec7.el7

centos-k8s-2.ervine.dev Ready master 3d4h v1.13.4 192.168.11.22 CentOS Linux 7 (Core) 3.10.0-957.5.1.el7.x86_64 cri-o://1.13.1-1.rhaos4.1.git2ac1ec7.el7

centos-k8s-3.ervine.dev Ready 3d3h v1.13.4 192.168.11.23 CentOS Linux 7 (Core) 3.10.0-957.5.1.el7.x86_64 cri-o://1.13.1-1.rhaos4.1.git2ac1ec7.el7

centos-k8s-4.ervine.dev Ready 3d1h v1.13.4 192.168.11.24 CentOS Linux 7 (Core) 3.10.0-957.5.1.el7.x86_64 cri-o://1.13.1-1.rhaos4.1.git2ac1ec7.el7

centos-k8s-5.ervine.dev Ready 3d1h v1.13.4 192.168.11.25 CentOS Linux 7 (Core) 3.10.0-957.5.1.el7.x86_64 cri-o://1.13.1-1.rhaos4.1.git2ac1ec7.el7

centos-k8s-6.ervine.dev Ready 3d1h v1.13.4 192.168.11.26 CentOS Linux 7 (Core) 3.10.0-957.5.1.el7.x86_64 cri-o://1.13.1-1.rhaos4.1.git2ac1ec7.el7

centos-k8s-7.ervine.dev Ready 3d1h v1.13.4 192.168.11.27 CentOS Linux 7 (Core) 3.10.0-957.5.1.el7.x86_64 cri-o://1.13.1-1.rhaos4.1.git2ac1ec7.el7

centos-k8s-8.ervine.dev Ready 3d1h v1.13.4 192.168.11.28 CentOS Linux 7 (Core) 3.10.0-957.5.1.el7.x86_64 cri-o://1.13.1-1.rhaos4.1.git2ac1ec7.el7

Not quite done though …

It looks like I’m all finished here, however when checking through various pods and trying to deploy simple pods and check their network connectivity, I found that there were some networking problems. I (think I) eventually traced this to not having the weave port open on the nodes. Looking at:

# kubectl -n kube-system logs weave-net-6gdw2 -c weave


INFO: 2019/03/14 07:46:34.983049 Sniffing traffic on datapath (via ODP)
INFO: 2019/03/14 07:46:34.985372 ->[192.168.11.211:6783] attempting connection
INFO: 2019/03/14 07:46:34.985534 ->[192.168.11.211:40247] connection accepted
INFO: 2019/03/14 07:46:34.985750 ->[192.168.11.211:40247|66:48:9c:db:05:37(centos-k8s-1.ervine.dev)]: connection shutting down due to error: cannot connect to ourself
INFO: 2019/03/14 07:46:34.985807 ->[192.168.11.211:6783|66:48:9c:db:05:37(centos-k8s-1.ervine.dev)]: connection shutting down due to error: cannot connect to ourself
INFO: 2019/03/14 07:46:34.988785 Listening for HTTP control messages on 127.0.0.1:6784
INFO: 2019/03/14 07:46:34.988845 Listening for metrics requests on 0.0.0.0:6782

It would appear that weave is expecting port 6783 to be open, and possible port 6782 for metrics. On each node I ran

# firewall-cmd --add-port 6783/tcp --perm
# firewall-cmd --add-port 6782/tcp --perm
# firewall-cmd --add-port 6782/udp --perm
# firewall-cmd --add-port 30000-32767/tcp --perm
# systemctl reload firewalld

The ports 30000-32767 are added as these are used by the NodePort services and I’ll want to make sure that my hosted services are available. Once the above firewall commands had been added on all nodes, the weave pods reported connectivity with one another. At this point it looks like the cluster is good to go!