Bootstrapping Kubernetes clusters with kubeadm

The kubeadm tool offers the possibility of creating a minimum viable cluster. Some advantages are its simplicity, availability for almost any system and integration for automated provision and cluster life cycle management. All this without giving up the ability to provide consistent and interoperable clusters within the Certified Kubernetes Conformance Program.

In general, the creation and operation of a Kubernetes cluster is nothing more than that of its components and requires us to deal with a series of considerations and orderly instructions that it is often necessary to gather from somewhat scattered documentation.

In particular, I am collecting the necessary instructions to bootstrap a Kubernetes cluster based on the following components and versions:

Ubuntu 22.04.3 LTS
Docker Engine 24.0.6
cri-dockerd 0.3.4
Kubernetes 1.27.5
Calico 3.26.1

This post assumes you are familiar with the components of a Kubernetes cluster and is intended as a guide only.

Before starting

Before starting, it is necessary to make sure that each node that is going to be part of the cluster has certain characteristics configured to ensure the correct behavior of the components to be installed.

Forwarding IPv4 and letting iptables see bridged traffic

Pods need to communicate across the cluster transparently and independently of the node where they are deployed, so it is essential that each node has traffic forwarding enabled. Also it is necessary to ensure that the filesystem overlay required for the composition of the container layers is available.

So create a file containing the names of kernel modules that should be loaded at boot time:

        
      
cat << EOF | sudo tee /etc/modules-load.d/k8s.conf
overlay
br_netfilter
EOF

Then load the kernel modules:

sudo modprobe overlay
sudo modprobe br_netfilter

And verify that modules are loaded:

lsmod | grep br_netfilter
lsmod | grep overlay

Now create a file containing the system variables configuration files that should be loaded at boot time:

        
      
cat << EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables  = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward                 = 1
EOF

Let’s apply the kernel parameters at runtime so that the above changes take effect without rebooting:

sudo sysctl --system

Finally verify that system variables are indeed set to 1 in your sysctl config:

sysctl net.bridge.bridge-nf-call-iptables net.bridge.bridge-nf-call-ip6tables net.ipv4.ip_forward

Verifying MAC and product_uuid uniqueness

Some components use the network adapter ID (MAC address) and/or the motherboard ID (product_uuid) to uniquely identify the nodes.

Verify the MAC address is unique for every node:

ip link

Verify the product_uuid is unique for every node:

sudo cat /sys/class/dmi/id/product_uuid

Checking required ports

When running Kubernetes in an environment with strict network boundaries, be aware of ports and protocols used by its components and verify that they are open to allow communication between them.

On control-plane / master node(s):

nc -zv 127.0.0.1 6443 2379-2380 10250 10259 10257

On worker node(s):

nc -zv 127.0.0.1 10250 30000-32767

Disabling swap memory

Cluster components are mainly performance- and reliability-oriented. The introduction of swap memory under disk pressure makes it somewhat unpredictable to achieve this purpose.

So disable swapping on all known swap devices and files:

sudo swapoff -a

Check that the swap area has indeed been disabled:

free -h

And remove the unneeded swap space file:

sudo rm /swap.img

Finally comment the entire line in the fstab file to permanently prevent swap space from mounting at startup:

sudo vi /etc/fstab
... 
# /swap.img       none    swap    sw      0       0

Installing the container runtime

Installing Docker Engine

Next it’s necessary to install a container runtime that conforms with the Container Runtime Interface (CRI) so that pods/containers can run into each node in the cluster.

As we are choosing to install Docker Engine, add Docker’s official GPG key:

        
      
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
sudo chmod a+r /etc/apt/keyrings/docker.gpg

Set up the repository:

        
      
echo \
 "deb [arch="$(dpkg --print-architecture)" signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
 "$(. /etc/os-release && echo "$VERSION_CODENAME")" stable" | \
 sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

Update the apt package index and install packages to allow apt to use a repository over HTTPS:

        
      
sudo apt update && sudo apt install -y ca-certificates curl gnupg

Install Docker Engine, containerd, and Docker Compose for the latest version:

        
      
sudo apt install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

To install a specific version, first check available versions for each package:

        
      
apt list -a docker-ce | awk '{print $2}'
apt list -a docker-ce-cli | awk '{print $2}'
apt list -a containerd.io | awk '{print $2}'
apt list -a docker-buildx-plugin | awk '{print $2}'
apt list -a docker-compose-plugin | awk '{print $2}'

And then indicate such version in the installation command:

        
      
sudo apt install -y docker-ce=<version> docker-ce-cli=<version> containerd.io=<version> docker-buildx-plugin=<version> docker-compose-plugin=<version>

Add your user to the docker group to manage Docker as a non-root user:

        
      
sudo usermod -aG docker $USER

Re-evaluate groups to activate changes without having to log out/in:

newgrp docker

Finally verify that you can run docker commands without sudo:

docker run hello-world

You should get an output like this:

Hello from Docker!
This message shows that your installation appears to be working correctly.
...

Configure Docker to start on boot:

        
sudo systemctl enable docker.service
sudo systemctl enable containerd.service

Installing cri-dockerd

Originally, Docker Engine was integrated directly with the kubelet code. When Kubernetes moved to use the CRI layer, a temporary adapter called dockershim was added between the CRI and Docker Engine. With Kubernetes 1.24, dockershim is no longer a part of the Kubernetes core and users need to install the cri-dockerd third-party adapter to provide the integration of Docker Engine with Kubernetes.

Use the pre-built cri-dockerd package to install the binary and setup the system to run it as a service:

        
wget https://github.com/Mirantis/cri-dockerd/releases/download/v0.3.4/cri-dockerd_0.3.4.3-0.ubuntu-jammy_amd64.deb -P /tmp
sudo apt install -y /tmp/cri-dockerd_0.3.4.3-0.ubuntu-jammy_amd64.deb

Check the service is running and listening on unix:///var/run/cri-dockerd.sock (the endpoint socket by default):

systemctl status cri-docker.service
systemctl status cri-docker.socket

Installing the kube tools

Next it’s time to install the command to bootstrap the cluster (kubeadm), the component to start pods and containers (kubelet) and the command line tool to talk to your cluster (kubectl).

kubeadm will not install or manage kubelet or kubectl for you, so you will need to ensure they match the version of the Kubernetes control-plane you want kubeadm to install for you.

Update the apt package index and install packages needed to use the Kubernetes apt repository:

        
      
sudo apt update && sudo apt install -y apt-transport-https ca-certificates curl

Download the Google Cloud public signing key:

        
      
curl -fsSL https://dl.k8s.io/apt/doc/apt-key.gpg | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-archive-keyring.gpg

Add the Kubernetes apt repository:

        
      
echo "deb [signed-by=/etc/apt/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list

Update apt package index and install kubelet, kubeadm and kubectl for the latest version:

        
      
sudo apt update && sudo apt install -y kubelet kubeadm kubectl

To install a specific version, first check available versions for packages:
1 curl -s https://packages.cloud.google.com/apt/dists/kubernetes-xenial/main/binary-amd64/Packages | grep Version | awk '{print $2}' | sort -V | uniq
And then indicate such version in the installation command:
1 sudo apt install -y kubelet=<version> kubeadm=<version> kubectl=<version>

Pin kubelet, kubeadm and kubectl versions:

sudo apt-mark hold kubelet kubeadm kubectl

Enable kubectl autocompletion:

        
      
echo 'source <(kubectl completion bash)' >> ~/.bashrc

Extend shell completion to work with the k alias (optional):

        
echo 'alias k=kubectl' >> ~/.bashrc
echo 'complete -F __start_kubectl k' >> ~/.bashrc

Reload .bashrc in order to new configuration take effect in current session:

source ~/.bashrc

kubectl is supported within one minor version (older or newer) of kube-apiserver.

To know at any time which versions of kubectl and kube-apiserver are running, check the client and server version respectively from the output of the following command:
1 kubectl version

Installing the cluster control-plane components

The control-plane node (master node) is the machine where all the decisions are made, running components such as the etcd (the cluster database) and the API Server (which the kubectl command line tool communicates with).

Initialize the control-plane node specifying a suitable CIDR block for the CNI based Pod network add-on and the container runtime endpoint:

        
      
sudo kubeadm init --pod-network-cidr=10.244.0.0/16 --cri-socket unix:///var/run/cri-dockerd.sock

Take care that your pod network must not overlap with any of the host networks.

Copy the kubeadm output join command with the token and the discovery-token-ca-cert-hash to later join additional nodes to the cluster by:
1 kubeadm join <control-plane-host>:<control-plane-port> --token <token> --discovery-token-ca-cert-hash sha256:<hash>

Make kubectl work for your non-root user:

        
      
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

Installing the pod network add-on

Now it’s time to deploy a Container Network Interface (CNI) based Pod network add-on so that your pods can communicate with each other.

We’ve chosen Calico as is a widely adopted, battle-tested open source networking solution for Kubernetes, providing two major services for Cloud Native applications such as:

Network connectivity between workloads.
Network security policy enforcement between workloads.

Requirements

Create the following configuration file to prevent NetworkManager from interfering with the interfaces:

        
      
sudo mkdir -p /etc/NetworkManager/conf.d/
cat << EOF | sudo tee /etc/NetworkManager/conf.d/calico.conf
[keyfile]
unmanaged-devices=interface-name:cali*;interface-name:tunl*;interface-name:vxlan.calico;interface-name:vxlan-v6.calico;interface-name:wireguard.cali;interface-name:wg-v6.cali
EOF

Installing Calico by operator

Get the latest stable version tag from the projectcalico/calico repository:

        
      
VERSION=$(curl -sL https://api.github.com/repos/projectcalico/calico/releases/latest | jq -r ".name")

Install the Calico operator and the custom resource definitions (CRDs):

kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/$VERSION/manifests/tigera-operator.yaml

Download the custom resources necessary to configure Calico and customize the manifest for the 10.244.0.0/16 CIDR:

        
curl https://raw.githubusercontent.com/projectcalico/calico/$VERSION/manifests/custom-resources.yaml -O
sed -i 's/cidr:.*/cidr: 10\.244\.0\.0\/16/' custom-resources.yaml

Finally create the manifest in order to install Calico:

kubectl create -f custom-resources.yaml

Wait until each pod is in running status:

watch kubectl get pods -n calico-system

Checking the installation

Cluster DNS (CoreDNS) will only start up after a network is properly installed.

Verify that CoreDNS pods are running:

kubectl get pods --all-namespaces

Verify your node is also ready:

kubectl get nodes -o wide

Remove the taint on the control-plane so that you can schedule pods on it:

kubectl taint nodes --all node-role.kubernetes.io/control-plane-

In case you plan to keep the master node dedicated to the control-plane you can skip the previous step, but some worker node must be joined later in order to deploy your applications.

Install calicoctl

The calicoctl command line tool is used to manage Calico network and security policy, to view and manage endpoint configuration, and to manage a Calico node instance.

To install calicoctl as a binary on a single host:

        
      
POD=$(kubectl -n calico-system get pod -l k8s-app=calico-kube-controllers -o jsonpath="{.items[0].metadata.name}")
VERSION=$(kubectl -n calico-system describe pod $POD | grep Image: | cut -d ':' -f3)
sudo curl -L https://github.com/projectcalico/calico/releases/download/$VERSION/calicoctl-linux-amd64 -o /usr/local/bin/calicoctl
sudo chmod +x /usr/local/bin/calicoctl

Verify the command was properly installed by:

calicoctl version

Make sure you always install the version of calicoctl that matches the version of Calico running on your cluster.

Joining a new worker node

Whether or not you have allowed the deployment of pods on the master node, run this on any machine you wish to join an existing cluster:

        
      
sudo kubeadm join <control-plane-host>:<control-plane-port> --token <token> --discovery-token-ca-cert-hash sha256:<hash>

Check that the new node has been properly added by running the following command on the control-plane node:

kubectl get nodes

In case you need to retrieve the token, run the following command on the control-plane node:
1 kubeadm token list

In case you need to retrieve the discovery-token-ca-cert-hash, run the following command on the control-plane node:
1 2 openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | \ openssl dgst -sha256 -hex | sed 's/^.* //'

In case you are joining a node after token expired, run the following command on the control-plane node:
1 kubeadm token create --print-join-command

Deprovisioning a node cleanly

Everything that has a beginning has an end, if you want to deprovision your cluster and revert all changes made by the kubeadm command cleanly, you should first drain the node and make sure that the node is empty, then deconfigure the node.

To drain the node marking the node as unschedulable to prevent new pods from arriving and evicts or deletes all pods, run:

        
      
kubectl drain <node_name> --delete-emptydir-data --force --ignore-daemonsets

Then reset the state installed by kubeadm:

sudo kubeadm reset 

If you wish to reset iptables rules manually run:

        
      
sudo iptables -F && sudo iptables -t nat -F && sudo iptables -t mangle -F && sudo iptables -X 

If you wish reset IPVS tables manually run:

sudo ipvsadm -C

Now you are safe to remove the node from the cluster:

kubectl delete node <node_name>

You are now ready to start over, running kubeadm init or kubeadm join with the appropriate arguments.

Controlling your cluster from machines other than the control-plane node

In order to get a kubectl on some other computer to talk to your cluster, you need to copy the administrator kubeconfig file from your control-plane node to your workstation.

Assuming SSH access is disabled for root in your control-plane node, copy the kubeconfig file to your user home dir and set your user as owner:

        
sudo cp /etc/kubernetes/admin.conf $HOME/.
sudo chown $USER: admin.conf

Copy the kubeconfig file from your remote control-plane node and give a name of your choice:

scp $USER@<control-plane-host>:~/admin.conf ~/.kube/
mv ~/.kube/admin.conf ~/.kube/config-new

Append the kubeconfig file to your current KUBECONFIG env variable and update your current kubeconfig file:

        
export KUBECONFIG=~/.kube/config:~/.kube/config-new
kubectl config view --flatten > ~/.kube/config

Check that your new cluster is listed as a Kubernetes context and set it to control remotelly:

kubectl config get-contexts
kubectl config set-context <cluter_name>

Summary

Congratulations! At this point you must have a perfectly operational and remotely accessible Kubernetes cluster ready to receive the deployment of your favorite applications.

In future posts I will try to explain how to deploy some of them.