Chaos drill 8.C6 — restart the kubelet with mounts held¶
Claim — you can state what volume reconstruction recovers after a kubelet restart and what it cannot, and point at KEP-3756 for why.
Topology — pair, continued from 8.C5, with pods on volumes from your driver and from local-path.
Do
- Record the full mount tree and the pod-to-volume mapping before touching anything — you are about to compare against it.
systemctl restart kubeleton the worker. Time how long the node isNotReadyand what happens to the running containers.- Diff the mount tree afterwards. Everything the kubelet rebuilt from its own on-disk state is reconstruction; everything that came back from the API is not.
- Now make it harder: restart the kubelet while a pod is terminating with its volume still mounted, and see what is left behind.
- Look for orphaned pod directories under
/var/lib/kubelet/pods/and decide whether the kubelet will clean them.
Observe
ssh zain@10.10.10.131 'findmnt -R /var/lib/kubelet > /tmp/before'
ssh zain@10.10.10.131 'systemctl restart kubelet; sleep 30; findmnt -R /var/lib/kubelet > /tmp/after; diff /tmp/before /tmp/after'
ssh zain@10.10.10.131 'journalctl -u kubelet -b | grep -i "reconstruct\|orphan"'
kubectl get nodes -w
Expect — the containers keep running; the kubelet is not in their execution path. Reconstruction rebuilds the volume manager's actual-state-of-world from the mount table, and KEP-3756 exists because what it could infer before was not enough — specifically the parts of a volume's identity that live only in the API.
Write down — the list of what survived and what did not, and one sentence on why the desired/actual-state-of-world split makes this recoverable at all.
Teardown — leave it up for 8.C4, which ends the phase's cluster.