Moving off of docker for the container runtime interface (CRI) on Kubernetes

Kubernetes is deprecating using docker as a runtime as of 1.24

A little background: I have a Kubernetes cluster with the masters running on Raspberry Pis running Raspbian and the workers on Intel NUCs running CentOS 7.


--------You can ignore this section if you aren't running Raspberry Pis--------
My Raspberry Pis were running kernel 4.19.75-v7l+ and was hitting a bug trying to do this conversion in kernel <4.14.159 or <4.19.89. I was getting the following error:

failed to "StartContainer" for "etcd" with RunContainerError: "failed to create containerd task: OCI runtime create failed: container_linux.go:349: starting container process caused \"process_linux.go:449: container init caused \\\"process_linux.go:415: setting cgroup config for procHooks process caused \\\\\\\"failed to write \\\\\\\\\\\\\\\"100000\\\\\\\\\\\\\\\" to \\\\\\\\\\\\\\\"/sys/fs/cgroup/cpu,cpuacct/system.slice/containerd.service/kubepods-besteffort-pod624587c2245d85257936b6b4eeee084f.slice:cri-containerd:56614f4494d205c14a16e47715d68fea52c62d6d4d0f99d51d83658bff0e7f10/cpu.cfs_period_us\\\\\\\\\\\\\\\":

To fix this, I had to run rpi-update to get a new kernel and modify the /boot/cmdline.txt to include the following and reboot.

cgroup_enable=cpuset cgroup_memory=1 cgroup_enable=memory

Nothing needed to be done on CentOS for the kernel
--------End Raspberry Pi section--------

First thing to do is to drain the node you want to work on

kubectl drain node4 --ignore-daemonsets --delete-local-data

Once your node is drained you can convert.
Now we need to stop and remove docker from the box

systemctl stop docker

Debian:

apt-get remove docker-ce docker-ce-cli

CentOS:

yum remove docker-ce docker-ce-cli

Debian only: I then did an install of containerd to ensure it was there and tracked as a package I installed.

apt-get install containerd.io

Generate a default config for containerd beyond what is there by default

containerd config default > /etc/containerd/config.toml

systemctl restart containerd

Now edit kubelet to use containerd, modify /var/lib/kubelet/kubeadm-flags.env add the following to the end of KUBELET_KUBEADM_ARGS

--container-runtime=remote --container-runtime-endpoint=unix:///run/containerd/containerd.sock

systemctl restart kubelet

You will also need to change the node annotation from using the docker CRI if you used kubeadm on all nodes

kubectl annotate node node4 --overwrite kubeadm.alpha.kubernetes.io/cri-socket=unix:///run/containerd/containerd.sock

Once I had done that, the apiserver and other master-node services started up without docker running.

You can verify things are running by checking the containers with (Images, containers and other things are namespaced with containerd):

ctr -n k8s.io container ls

If you are adding a new node to the cluster with just containerd you will need to modify the system a bit since Docker isn't there

cat <<EOF | sudo tee /etc/modules-load.d/containerd.conf
overlay
br_netfilter
EOF

modprobe overlay
modprobe br_netfilter

# Setup required sysctl params, these persist across reboots.
cat <<EOF | sudo tee /etc/sysctl.d/99-kubernetes-cri.conf
net.bridge.bridge-nf-call-iptables  = 1
net.ipv4.ip_forward                 = 1
net.bridge.bridge-nf-call-ip6tables = 1
EOF

# Apply sysctl params without reboot
sudo sysctl --system