Lessons Learned Upgrading On-Prem Kubernetes Nodes from Ubuntu 20.04 to 22.04

Operating system upgrades on Kubernetes nodes sound simple until they collide with boot partitions, kernel packages, container runtimes, CNI behavior, and workloads that are not as stateless as everyone hoped.

This is a sanitized writeup of lessons learned while upgrading an on-prem Kubernetes environment from Ubuntu 20.04 to 22.04.

Start with boring inventory

Before touching a node, capture the current state.

kubectl get nodes -o wide
kubectl get pods -A -o wide --field-selector spec.nodeName=<node-name>
kubectl describe node <node-name>

On the node itself:

lsb_release -a
uname -a
df -h / /boot
systemctl status kubelet containerd --no-pager
containerd --version

The goal is not just to know whether the node is ready. The goal is to know what "normal" looked like before the change.

Check /boot before the release upgrade

One of the easiest ways to ruin your day is starting a release upgrade with a nearly full /boot.

Check it early:

df -h /boot
dpkg -l 'linux-image*' | awk '/^ii/ {print $2, $3}'

Remove old kernels carefully. Do not remove the currently running kernel.

uname -r

Keep the active kernel and at least one fallback kernel where possible.

Drain one node at a time

For worker nodes:

kubectl cordon <node-name>
kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data

For control plane nodes, be more careful. Confirm the remaining control plane and etcd members can maintain quorum and API availability before taking anything down.

Validate containerd and cgroups

After the upgrade, make sure the runtime and kubelet still agree on cgroup behavior.

stat -fc %T /sys/fs/cgroup
containerd config dump | grep -i systemd -n
systemctl status containerd kubelet --no-pager
journalctl -u kubelet -n 100 --no-pager

In many environments, the desired state is systemd cgroups for containerd and kubelet.

Watch for iptables surprises

Some environments still rely on legacy iptables behavior. After the OS upgrade, verify the active alternatives:

sudo update-alternatives --display iptables
sudo update-alternatives --display ip6tables

If your CNI or kube-proxy setup expects legacy behavior, set it explicitly.

sudo update-alternatives --set iptables /usr/sbin/iptables-legacy
sudo update-alternatives --set ip6tables /usr/sbin/ip6tables-legacy

Then restart the relevant services:

sudo systemctl restart containerd
sudo systemctl restart kubelet

Confirm the node comes back clean

kubectl get nodes
kubectl describe node <node-name>
kubectl get pods -A -o wide --field-selector spec.nodeName=<node-name>

Look for:

• Ready=True
• no unexpected NetworkUnavailable
• no CNI initialization errors
• no runtime errors
• no stuck terminating pods
• no critical DaemonSet failures

The bigger lesson

The upgrade itself is not the hard part. The hard part is proving the node is safe before and after.

The best upgrade process is one where the human is making go/no-go decisions, not manually remembering every validation command.