Worker nodes — kubelet, kube-proxy, container runtime

The kubelet runs pods from their PodSpec, kube-proxy programs network rules for Services, and a CRI runtime runs the containers.

flowchart LR
  API["kube-apiserver"] <--> KL
  subgraph NODE["Worker node"]
    KL["kubelet<br/>(systemd service)"] --> CRI["containerd / CRI-O"]
    CRI --> P1["Pod A"]
    CRI --> P2["Pod B"]
    KP["kube-proxy<br/>iptables / IPVS"]
  end

Component	Role
kubelet	The agent on each node. Gets PodSpecs through the apiserver, calls the runtime to run containers, runs probes and reports pod and node status. On a kubeadm cluster it is a systemd service, not a pod
kube-proxy	Programs network rules (iptables or IPVS) on the node so traffic to a Service IP is forwarded and balanced to the pods behind it. It works at layer 4; it is not an HTTP reverse proxy
Container runtime	CRI-compatible software that runs containers: containerd, CRI-O. Dockershim was removed in Kubernetes 1.24, so nodes need containerd/CRI-O (or cri-dockerd) rather than talking to Docker Engine directly

Pods and Services

A Pod is the smallest unit: one or more containers on the same node sharing a network namespace (one IP) and volumes.
A Service gives a stable address (ClusterIP + DNS name) for a group of pods selected by label, because pod IPs change whenever pods are recreated.

Each node's container runtime shows in the CONTAINER-RUNTIME column of kubectl get nodes -o wide.

Command	Meaning
`kubectl get nodes -o wide`	Nodes, internal IP, OS, kernel, runtime
`kubectl describe node worker-1`	Conditions, capacity, running pods, events
`systemctl status kubelet`	kubelet status (run on the node itself)

Node health

Each node reports conditions in kubectl describe node:

Condition	True means
`Ready`	kubelet is healthy and can run pods
`MemoryPressure`	Available memory is low
`DiskPressure`	Disk space or inodes are low
`PIDPressure`	Too many processes
`NetworkUnavailable`	The node's network is not configured

kubelets send heartbeats as Lease objects in the kube-node-lease namespace. If heartbeats stop for about 40 seconds the node is marked NotReady, and by default its pods are evicted and rescheduled after five minutes. Under pressure, the kubelet itself evicts pods to protect the node, lowest priority first.

Capacity and allocatable

Capacity is the node's total CPU and memory; Allocatable is what is left for pods after reserving resources for the OS and Kubernetes daemons. The scheduler only uses allocatable.

Maintenance workflow

kubectl cordon worker-1      # no new pods scheduled here
kubectl drain worker-1 --ignore-daemonsets --delete-emptydir-data   # evict pods safely
# patch or reboot the node
kubectl uncordon worker-1    # accept pods again

drain respects PodDisruptionBudgets, so it will not take down more replicas of an app than allowed.

Taints and tolerations

A taint on a node repels pods (kubectl taint nodes gpu-1 gpu=true:NoSchedule); only pods with a matching toleration can run there. Control-plane nodes carry a taint so ordinary workloads stay off them.

Debugging on the node

When kubectl is not enough, SSH to the node: journalctl -u kubelet shows kubelet errors, and crictl ps, crictl logs <id>, and crictl images talk to containerd directly. Static pods are defined by files in /etc/kubernetes/manifests; the kubelet runs them even without the apiserver, which is how kubeadm starts the control plane itself.