The state this post announces, and links to its own definition of, appears nowhere in the pinned documentation; the one sentence that carries the idea calls it something else on a page about kernel version floors; and what is left of it on a node is a kubelet field with no flag¶
Post — Kubernetes 1.31: Moving cgroup v1 Support into Maintenance Mode, 2024-08-14.
4,742 bytes, 102 lines, one author: Harshal Patil. Forty-seventh of 2024's 54 posts by size, 6,026
below the year's mean of 10,768, and the third shortest of the thirteen in this year's walk set.
Eleven links, of which two reach kubernetes.io and both of those are in Further reading; three go
to github.com; two to the kernel's own cgroup documentation; two to man7.org; one to systemd.io;
and one is an in-page anchor to the post's own definition of its subject. The KEP this post exists
to announce, 4569, is not among them — the only KEP it links is 2033, for rootless kubelets, cited
in passing for one of cgroup v2's benefits.
As written
:9-15 states the decision in one sentence and immediately links
#what-does-maintenance-mode-mean, an anchor inside the post itself. That link is the shape of the
whole piece: the term is new, the post knows it is new, and the post is where it will be defined.
:17-32 is a primer. Control groups are a Linux kernel feature for allocating, prioritising,
denying and managing CPU, memory, disk I/O and network bandwidth among processes, which matters most
in multi-tenant environments. Then the flat claim at :27-29: There are two versions of cgroups:
v1 and v2, each linked to its own kernel documentation. v1 provided sufficient capabilities and
had limitations that led to the development of cgroup v2.
:34-50 puts that in Kubernetes terms. Each container is placed in its own cgroup, which is how
limits get enforced, usage gets monitored, and distribution stays fair. :43-50 is a definition
list of three terms — Resource Allocation, Isolation, Monitoring — and it is the only formatted
structure in the post that is not a numbered list.
:52-65 is the direction of travel. The Linux community focuses new work on v2; major distributions
and systemd are transitioning; v2 brings a unified hierarchy, an improved interface, better resource
control, a cgroup-aware OOM killer and rootless support. Two of those five are linked to GitHub
rather than to documentation — a pull request and a KEP README anchor. :63-65 says Kubernetes is
making the move too, carefully, to avoid disrupting existing workloads.
:67-86 is the announcement. :71-76 defines the term in three numbered promises: Feature
Freeze, no new features added to cgroup v1 support; Security Fixes, critical ones still
provided; Best-Effort Bug Fixes, major bugs fixed if feasible. :78-86 gives the reason and
then the reassurance the rest of this exercise is about: It's important to note that maintenance
mode does not mean deprecation; cgroup v1 will continue to receive critical security fixes and major
bug fixes as needed.
:88-96 is the instruction to administrators — two numbered items, upgrade the operating systems
and container runtimes, test the workloads — and :98-102 is three further-reading links. There is
no feature gate named anywhere in the post, no {{< feature-state >}} marker, no version skew
table, and no command. Nothing in it is switchable.
As it runs now
The subject of this post is a word. So the first question is what happened to the word, and the pinned tree answers it precisely.
The phrase never entered the documentation. At the pin, maintenance mode occurs twelve times
in content/en, in exactly two files: nine times in this post and three times in
blog/_posts/2024/kubernetes-v1-31-release.md (:235, :249, :251), which is the release
announcement of the same release. Under docs/ the count is zero. The idea survives in the
reference documentation as one sentence, docs/reference/node/kernel-version-requirements.md:58-59,
under a heading called ## Version 2 control groups: Kubernetes cgroup v1 support is in maintained
mode starting from Kubernetes v1.31; using cgroup v2 is recommended. The word is not maintenance,
it is maintained, and none of the three promises travelled with it. What the sentence gained
instead is a recommendation the post never makes in those words, and it sits on a page about kernel
version floors, between a paragraph on kube-proxy's nftables mode and one on Pressure Stall
Information.
The one maintenance mode under docs/ belongs to something else entirely. Searched with a
hyphen instead of a space, the phrase does appear:
docs/concepts/workloads/autoscaling/horizontal-pod-autoscale.md:653 is a heading reading Implicit
maintenance-mode deactivation, and :655-660 defines it as an operator setting a workload's
replica count to zero so the HorizontalPodAutoscaler stops adjusting it. That is a different
subsystem, a different meaning and a different mechanism. It is the only thing in the Kubernetes
documentation that calls itself a maintenance mode, and it has nothing to do with this post.
The reassurance became the thing it was distinguished from. concepts/architecture/cgroups.md
carries a section marking cgroup v1 deprecated as of v1.35, four releases after this post. The
section and the kubelet field behind it belong elsewhere in this collection and are not re-explained
here; the line that matters for this exercise is cgroups.md:139 — Removal will follow [Kubernetes
deprecation policy] — because it is a link, and following it is what settles the post's central
claim. docs/reference/using-api/deprecation-policy.md:366-370 opens the rules for deprecating a
feature or behavior of the system that is not controlled by the API or CLI, which is exactly what
cgroup v1 support is. Rule #7 at :372-373: Deprecated behaviors must function for no less than 1
year after their announced deprecation. The lifecycle the policy names, at :412-417, has four
stages — alpha, beta, GA, and GA with the deprecation window complete — and a deprecation section at
:419. There is no maintenance mode in it, and there is no stage between supported and deprecated.
So the four releases this post bought started no clock under the policy the deprecation notice
points at; the one-year guarantee begins at v1.35, not at v1.31. The only part of the policy that
contemplates a state it has not defined is ## Exceptions at :493-502, which says such situations
should be discussed with SIGs and project leaders and that Exceptions will always be announced in
all relevant release notes — which is, in policy terms, an accurate description of what this post
is.
The feature freeze held, and it is legible only from the v2 side. Nothing in the pinned tree
records a feature being added to cgroup v1 and nothing records one being taken away, because that is
not how a freeze shows up in documentation. It shows up as capabilities that only ever existed on
the other version. Four places under docs/ say so in those terms:
docs/reference/instrumentation/understand-psi-metrics.md:29 requires a cgroup v2 node;
docs/tasks/configure-pod-container/resize-container-resources.md:59 requires it for resizing
memory-backed emptyDir volumes and :193 is the only sentence in the tree that says what a cgroup
v1 node does instead — in-place volume resize requests are rejected as infeasible;
docs/tasks/administer-cluster/kubelet-in-userns.md:131 requires it for running node components in
a user namespace; and docs/concepts/workloads/pods/pod-qos.md:195 requires it for Memory QoS. All
four describe work that landed at or after the release this post announces.
And the documentation disagrees with itself about the one field that is left. The single place
in the whole configuration API where cgroup v1's behaviour survives as something you can ask for is
docs/reference/config-api/kubelet-config.v1beta1.md:984-994, the kubelet's singleProcessOOMKill.
Its own description says what it is for, at :988-990: setting it true prevents memory.oom.group
from being set on container cgroups under cgroup v2, so processes are OOM-killed individually rather
than as a group, and the behavior aligns with the behavior of cgroups v1. Then three consecutive
sentences. :991: The default value is determined automatically when you don't specify. :993:
On cgroup v1 linux, only null / absent and true are allowed. :994: On cgroup v2 linux, null /
absent, true and false are allowed. The default value is false. The first sentence says there is no
fixed default; the last says the default is false. They are reconcilable only if the last one is
scoped to cgroup v2 and the automatic default on cgroup v1 is true, which the page never states.
Do not pick a reading from the page. Step 4 reads the value off a running kubelet and settles it.
One smaller thing, since the post is about what a policy change costs an administrator: the field
has no command-line flag. docs/reference/command-line-tools-reference/kubelet.md lists
--fail-cgroupv1 and --oom-score-adj and nothing for this one, so the only way to keep cgroup
v1's OOM semantics is to edit the kubelet configuration file and restart the kubelet. Step 6 does
exactly that.
What this exercise does not cover, and where it lives
cgroup v2 itself — the concept page, its requirements list, its distribution table, and the two
opposing feature-state markers the page carries for the two versions — belongs to Fifteen bullets
absorbed word for word, and the eight the page added since. That
exercise also runs the citation test on its own post and reads cgroup.controllers, memory.max
and cpu.max from inside a container. Nothing here repeats any of it.
The failCgroupV1 kubelet field, its documented default, and reading it off a running kubelet
belong to The one configuration value this post documents does not parse at the pin, the default it
names in parentheses is not the default any more, and both rows of its cgroups table are
void. This exercise reads a different field off the same
endpoint and does not touch that one. Memory QoS, memory.min and the reservation policy that sets
it belong to Fifteen releases in alpha, beta on the newest release the pin
carries, and the cgroup driver question — which component
decides whether the driver is systemd or cgroupfs — belongs to The project this post announces
survives in the pinned documentation as a tab id, a PNG filename and an AppArmor
profile.
Which task inside a cgroup the kernel picks when memory runs out, and the arithmetic that decides
it, are lab ground: The kernel kills the fattest process in the cgroup, and the pod is only marked
OOMKilled when that process happens to be the container's
init produces both outcomes on one pod spec, and the
oom_score_adj each QoS class gets, from the
formula computes the values it reads. Step 5
here reads the one file that decides whether that question has one victim or a whole cgroup of them;
it does not run the drill. Walking the cgroup tree from the node root down to a container is Walk
from the node's root cgroup down to one container, naming every
level, and reading the kernel's pressure figures
is the housekeeping drill. Pressure Stall
Information is named once below, as one of four counted requirements, and its own arc through alpha,
beta and GA is a later year's subject, not this one's.
The diff, and why
Never absorbed. The post coins a term, links to its own definition of it, and the term went
nowhere. Zero occurrences under docs/; the only survivor is one sentence on a kernel-requirements
page that spells it differently and keeps none of the three promises. The absorption test's other
half gives the same answer: nothing in the pinned tree links this post. Not the concept page it
points at, not the kernel-requirements page that carries its one surviving sentence, and not the
release announcement for its own release, which cites the enhancement issue instead. Step 10 runs
both halves. This is the seventh case in its cleanest form — a post that was read, acted on, and
then left behind without a citation.
Broke. :84-86 is the sentence the post wrote in order to be reassuring: maintenance mode
does not mean deprecation. It is false at the pin, and the way it is false is worse than a simple
reversal. Because maintenance mode was not deprecation, Rule #7's one-year clock did not start when
this post was published; it started four releases later when the deprecation was actually declared.
The sentence was true as written and the four releases of grace it describes bought no guaranteed
time at all. An administrator who read :84-86 in August 2024 and concluded there was no schedule
to plan against was reading it correctly, and that is exactly the problem.
Still right. The freeze was real. No feature has been added to cgroup v1 support in the pinned
tree, and the four capabilities counted above all exist on v2 only. The post's one instruction to
administrators, at :93-96 — make sure the operating system and the container runtime support
cgroup v2, then test the workloads — is still the entire migration, and step 7 runs it as a
checklist against a node. What the post could not say is that the second half of its own definition,
critical security fixes and best-effort bug fixes, is unobservable: there is no place in the
documentation, no field, and no metric that records whether either promise was kept. Only the first
promise left a trace.
Topology
solo, one node at 10.10.10.180 running Kubernetes v1.35,
provisioned with the standard steps and the node
baseline. One node is enough: everything here
is either a file the kernel exposes, a field the kubelet reads at startup, or a read of the pinned
checkout. The node runs cgroup v2, which is what makes it the right place to ask what is left of v1
— you are not looking for a cgroup v1 machine, you are looking for cgroup v1's remains on a machine
that does not use it. Cluster work happens in a namespace called bw-cg1. Step 6 edits the kubelet
configuration file and restarts the kubelet, so take the backup the step names before you run it.
Do
- Ground the node, and note that the kernel still registers both filesystems. The post's subject is a version the node does not use, so start by confirming the node does not use it and that the kernel has not forgotten how.
ssh zain@10.10.10.180
kubectl version -o json | jq -r '.serverVersion.gitVersion'
uname -r
systemctl --version | head -1
findmnt -no FSTYPE /sys/fs/cgroup
grep cgroup /proc/filesystems
- Read the v1 controller table.
/proc/cgroupsis cgroup v1's own interface and it is present on every kernel that has cgroups at all, whichever hierarchy is mounted. Four columns: the controller name, the hierarchy id it is attached to, how many cgroups use it, and whether it is enabled.
cat /proc/cgroups
awk 'NR>1 {n++; if ($4==1) e++; if ($2!=0) h++} END {
printf "%d controllers listed, %d enabled, %d attached to a numbered hierarchy\n", n, e, h}' /proc/cgroups
- Ask the kernel for a cgroup v1 hierarchy by hand. This is the difference between frozen and gone: a controller that is enabled in the table above may still be unavailable, because something else already holds it. Mount it in a scratch directory, read the answer, and put it back.
sudo mkdir -p /mnt/bw-cg1
sudo mount -t cgroup -o memory none /mnt/bw-cg1; echo "mount exit=$?"
findmnt /mnt/bw-cg1 || echo 'nothing mounted at /mnt/bw-cg1'
sudo umount /mnt/bw-cg1 2>/dev/null; sudo rmdir /mnt/bw-cg1
- Ask the running kubelet what
singleProcessOOMKillis on this node. Three sentences of the configuration reference disagree about the default; this is the command that settles them. Read the configuration file too, so you can tell a value the kubelet computed from a value somebody wrote.
NODE=$(kubectl get node -o jsonpath='{.items[0].metadata.name}')
kubectl get --raw "/api/v1/nodes/$NODE/proxy/configz" \
| jq '.kubeletconfig | {present: has("singleProcessOOMKill"), value: .singleProcessOOMKill}'
sudo grep -n singleProcessOOMKill /var/lib/kubelet/config.yaml || echo 'not set in the config file'
- Read the file the field controls, from inside a container. A container gets its own cgroup
namespace, so
/sys/fs/cgroupinside the pod is the container's own cgroup and no tree walk is needed.memory.oom.groupexists only under cgroup v2; a1means the kernel kills every task in the cgroup together, a0means it picks one.
kubectl create namespace bw-cg1
kubectl -n bw-cg1 run g1 --image=busybox:1.36 --restart=Never -- sleep 3600
kubectl -n bw-cg1 wait --for=condition=Ready pod/g1 --timeout=120s
kubectl -n bw-cg1 exec g1 -- cat /sys/fs/cgroup/memory.oom.group
kubectl -n bw-cg1 exec g1 -- sh -c 'ls /sys/fs/cgroup/ | head -20'
- Now ask for cgroup v1's behaviour back. Take the backup first. The field has no command-line flag, so this is the whole procedure: one line in the configuration file, a kubelet restart, and a fresh pod — the setting is applied when a container's cgroup is created, so the pod from step 5 will not change. Read the file again, then restore.
sudo cp -a /var/lib/kubelet/config.yaml /var/lib/kubelet/config.yaml.bw-cg1.bak
printf 'singleProcessOOMKill: true\n' | sudo tee -a /var/lib/kubelet/config.yaml
sudo systemctl restart kubelet
sleep 15
kubectl -n bw-cg1 run g2 --image=busybox:1.36 --restart=Never -- sleep 3600
kubectl -n bw-cg1 wait --for=condition=Ready pod/g2 --timeout=180s
kubectl -n bw-cg1 exec g1 -- cat /sys/fs/cgroup/memory.oom.group
kubectl -n bw-cg1 exec g2 -- cat /sys/fs/cgroup/memory.oom.group
NODE=$(kubectl get node -o jsonpath='{.items[0].metadata.name}')
kubectl get --raw "/api/v1/nodes/$NODE/proxy/configz" | jq '.kubeletconfig.singleProcessOOMKill'
sudo cp -a /var/lib/kubelet/config.yaml.bw-cg1.bak /var/lib/kubelet/config.yaml
sudo systemctl restart kubelet
- Run the post's instruction to administrators as a checklist.
:93-96asks for two things: that the operating system and the container runtime support cgroup v2, and that the workloads are tested. The first half is four commands, and this is the runtime's side of it rather than the kernel's.
grep '^PRETTY_NAME' /etc/os-release
containerd --version
runc --version | head -1
sudo containerd config dump | grep -i 'systemd_cgroup\|SystemdCgroup'
kubectl get node -o jsonpath='{.items[0].status.nodeInfo.containerRuntimeVersion}{"\n"}'
- Offline, in a checkout of
kubernetes/websiteat the pin. Census the phrase. Count it in the whole tree, count it underdocs/, list the files that carry it, then find the sentence that survived and the unrelated thing that shares the name.
cd /path/to/kubernetes/website/content/en
grep -rn -i 'maintenance mode' . | wc -l
grep -rn -i 'maintenance mode' docs/ | wc -l
grep -rln -i 'maintenance mode' .
grep -rn -i 'maintained mode' .
grep -rn -i 'maintenance-mode' docs/
sed -n '56,62p' docs/reference/node/kernel-version-requirements.md
sed -n '653,660p' docs/concepts/workloads/autoscaling/horizontal-pod-autoscale.md
- Still offline. Follow the link the deprecation notice points at and read what the policy actually offers. One line from the concept page, then the rules for deprecating a behaviour, then the only lifecycle the policy names, then the section that covers a state it has not defined.
cd /path/to/kubernetes/website/content/en
sed -n '139p' docs/concepts/architecture/cgroups.md
sed -n '366,373p' docs/reference/using-api/deprecation-policy.md
sed -n '409,418p' docs/reference/using-api/deprecation-policy.md
sed -n '493,502p' docs/reference/using-api/deprecation-policy.md
grep -c -i 'maintenance\|maintained' docs/reference/using-api/deprecation-policy.md || echo 0
-
Last offline step. Run the absorption test's citation half, read the field's paragraph in full, and count the four requirements that only the other version can meet.
cd /path/to/kubernetes/website/content/en grep -rn 'kubernetes-1-31-moving-cgroup-v1-support-maintenance-mode' . | grep -v '^./blog/_posts/2024/moving-cgroup' || echo 'no inbound reference' grep -rn '2024/08/14' . || echo 'nothing links the post by URL' sed -n '984,995p' docs/reference/config-api/kubelet-config.v1beta1.md sed -n '29p' docs/reference/instrumentation/understand-psi-metrics.md sed -n '59p;193p' docs/tasks/configure-pod-container/resize-container-resources.md sed -n '131p' docs/tasks/administer-cluster/kubelet-in-userns.md sed -n '195p' docs/concepts/workloads/pods/pod-qos.md grep -c 'single-process-oom-kill' docs/reference/command-line-tools-reference/kubelet.md || echo 0
Expect
Step 1. v1.35, a 6.12 kernel, systemd 257, and cgroup2 as the filesystem type at /sys/fs/cgroup.
The line that matters is the last one: /proc/filesystems lists both cgroup and cgroup2.
The kernel this node booted has cgroup v1 compiled in and registered as a mountable filesystem, and
nothing about the node using v2 changes that. Frozen is not the same as absent, and this is the
first place you can see the difference.
Step 2. /proc/cgroups prints a header and a row per controller — on a Trixie kernel expect
thirteen or fourteen of them, cpuset through misc. Every row should read 1 in the enabled
column and 0 in the hierarchy column. Those two numbers together are the whole state of cgroup v1
on this machine: the kernel offers every controller, and not one of them is attached to a v1
hierarchy, because the unified hierarchy has them. num_cgroups sitting at 1 for each is the root
cgroup and nothing else. The post's :27-29 says there are two versions of cgroups; this file is
the first one, still exporting its interface to a node that has never used it.
Step 3. Expect the mount to fail, and expect the reason to be device or resource busy rather than
an unknown filesystem type. That distinction is the finding. The kernel understands the request; it
refuses because the memory controller is already bound to the unified hierarchy, and a controller
can be attached to one hierarchy at a time. So cgroup v1 on this node is not deprecated, not removed
and not disabled — it is unreachable, and it is unreachable because of a boot-time decision made by
systemd rather than by anything in Kubernetes. If the mount instead succeeds, the node did not put
every controller on the unified hierarchy and you have just created a live cgroup v1 hierarchy in
maintenance mode with one command; unmount it and record that, because it changes what every later
step means.
Step 4. Expect present: true and value: false, and expect the configuration file not to mention
the field at all. That settles the three sentences. :991's determined automatically is the
operative rule — nobody wrote this value — and :994's The default value is false describes what
the automatic determination produces on a cgroup v2 node, which is the scoping the page leaves
implicit. If instead present comes back false, the kubelet is not serialising a computed default
and :991 is the only sentence in effect; either way, the reading in which :994 is an
unconditional statement about all nodes is the one the node rules out, because :993 forbids
false on cgroup v1 and a single unconditional default cannot be both legal and illegal.
Step 5. memory.oom.group prints 1. The kubelet set it, because singleProcessOOMKill is false,
and a 1 means the kernel treats the container as a unit when it runs out of memory: every task in
the cgroup dies together. The listing shows the other half of what changed between the versions — a
flat set of interface files, memory.max and cpu.max and pids.max and cgroup.controllers,
with no per-controller subdirectory anywhere, because the unified hierarchy has one tree instead of
one per controller. Take the 1 and hold on to it; the next step is about turning it off.
Step 6. g1 should still print 1 and g2 should print 0, and configz should now report
true. What you are looking at in g2 is cgroup v1's OOM semantics running on a cgroup v2 node, a
release after cgroup v1 was deprecated and four after it was frozen, reachable because somebody
wrote one field for it — a field with no command-line flag, no feature gate and no feature-state
marker. That is what maintenance mode turned into where it is observable: not a mode, a
compatibility switch. If g1 also flips to 0, the kubelet rewrote an existing container's cgroup
on restart rather than deciding at creation time; record which happened, because the reference text
says only that the flag is set for container cgroups and does not say when. The last two commands
put the configuration back; the teardown checks that they did.
Step 7. Debian Trixie, containerd 2.2.1, runc 1.5.1, SystemdCgroup = true, and a runtime version
on the Node object matching the binary. Both of the post's instructions to administrators are
already satisfied and were satisfied before you started, which is the ordinary case at the pin and
the reason the post reads as advice addressed to somebody else. The post could not have known that;
in August 2024 the migration it describes was real work for a large number of clusters. What the
exercise records is that the advice outlived the situation it was written for by a wide margin.
Step 8. Twelve, then zero. Two files carry the phrase and both are blog posts from the same release.
maintained mode returns exactly one line, docs/reference/node/kernel-version-requirements.md:58,
and reading :56-62 around it shows how small the landing was: a heading, that sentence, a note
about cpu.stat in Linux 5.8, and a note about runc's freezer. The hyphenated search returns the
HorizontalPodAutoscaler's Implicit maintenance-mode deactivation, which is an operator setting
replicas to zero. A term the post defined for the project is not in the project's documentation, and
the one phrase that looks like it belongs to a different subsystem.
Step 9. cgroups.md:139 is a single sentence and it is a link. Rule #7 gives one year from
announced deprecation, and the deprecation was announced at v1.35, so the guarantee this post's
readers actually have started four releases after they read it. The lifecycle list names alpha,
beta, GA and GA-with-the-window-complete, and nothing between supported and deprecated. The last
count should print 0: the deprecation policy does not contain the word maintenance or the word
maintained anywhere in its 502 lines. ## Exceptions is the only part that anticipates a state
the policy has not defined, and its answer — discuss it with the SIGs, announce it in the release
notes — is a fair description of how this post came to exist.
Step 10. No inbound reference and nothing linking the post by URL: the citation half of the
absorption test comes back empty from a tree of 767 posts and several thousand documentation files.
The field's twelve lines are worth reading in one piece rather than as quotations. The four
requirement sentences are the freeze seen from the far side, and resize-container-resources.md:193
is the only one that says what happens on a cgroup v1 node instead of what is needed on a v2 one.
The last count is 0: there is no command-line flag for the one field that preserves the behaviour
of the version this post was written about.
Read on
-
docs/reference/using-api/deprecation-policy.md:366-450— the behaviour half of the policy, read whole. Rules #7, #8, #9 and #10, the feature-gate lifecycle, and the deprecation section. It is the document that decides what a status announcement is worth, and it is short. -
docs/reference/node/kernel-version-requirements.md— the page the post's idea ended up on. Reading it end to end is the quickest way to see why one sentence about a support status looks out of place between a kube-proxy kernel floor and a note aboutCONFIG_PSI. -
docs/reference/config-api/kubelet-config.v1beta1.md:984-994—singleProcessOOMKillin full, including the two sentences step 4 settles and the one about non-Linux nodes that neither of them covers. -
Fifteen bullets absorbed word for word, and the eight the page added since — the other side of this migration, from the release that declared cgroup v2 generally available. Read together, the two posts are the same arc from both ends, and neither is cited by the page that replaced them both.
-
Unanswerable from the pin: whether the second and third promises were kept. Critical security fixes and best-effort bug fixes leave no trace in documentation — no field records them, no metric counts them, and no page says whether a cgroup v1 bug was fixed or declined between v1.31 and v1.35. Only the freeze left evidence, and only because new work went somewhere else.
Teardown
kubectl delete namespace bw-cg1 --wait=true
sudo cmp /var/lib/kubelet/config.yaml /var/lib/kubelet/config.yaml.bw-cg1.bak && echo 'kubelet config restored'
sudo rm -f /var/lib/kubelet/config.yaml.bw-cg1.bak
findmnt /mnt/bw-cg1 || echo 'no stray cgroup v1 mount'
systemctl is-active kubelet
The cmp is the important line. Step 6 appended to the kubelet's configuration file and restored it
from a copy; if the two files differ, something else wrote to the file in between and the node is
not in the state it started in. If step 3's mount succeeded and was not unmounted, findmnt will
say so — leaving a cgroup v1 hierarchy mounted on a node whose kubelet expects v2 is the one way
this exercise can damage a cluster. If the node was untainted during provisioning and the taint is
wanted back: