The page that teaches you to read the number disagrees with itself by a factor of ten, the JSON sample it disagrees inside is not JSON and is printed byte for byte in two files, the schema link beside it points at a page the pin does not contain, and four cores can check the graphs¶
Post — Kubernetes v1.36: PSI Metrics for Kubernetes Graduates to GA, 2026-05-12.
92 lines, 9,469 bytes, one author — Maria Fernanda Romano Silva, Google Cloud — one fenced block,
four figures and no tables. The thirty-seventh of 2026's fifty-nine published rows and the sixth
walk among them. It is the only walk of the year whose central claim is a performance measurement,
and the only evidence it offers for that claim is four line graphs.
As written
The opening is two paragraphs. :9-12 dates Pressure Stall Information to its 2018 implementation
in the Linux kernel and says it provides the high-fidelity signals needed to identify resource
saturation before it becomes an outage; unlike utilisation metrics, PSI tells the story of tasks
stalled and time lost, all in nicely-packaged percentages of time across the CPU, memory, and I/O.
:14 announces the graduation in v1.36 — a stable, reliable interface to observe resource
contention at the node, pod, and container levels — and promises a dive into the improvements and
performance testing that proved its readiness for production.
:16-20 is the case for the metric. :18 says monitoring CPU or memory usage alone can mislead,
because a node may report XX% (below 100%) CPU utilization while certain tasks are experiencing
severe latency due to scheduling delays. The two bullets at :19-20 are what PSI adds: Cumulative
Totals, the absolute time spent stalled, and Moving Averages over 10s, 60s and 300s windows,
which let an operator distinguish between transient spikes and sustained resource tension.
:22-28 sets up the testing. The concern named at :24 is the resource overhead of collecting and
serving the metrics, and the answer is extensive performance validation on high-density workloads
(80+ pods) across various machine types run by SIG Node. :26-28 splits it in two: scenario 1
holds kernel PSI on and toggles the kubelet feature, isolating the kubelet's cost; scenario 2 holds
the kubelet feature on and toggles kernel PSI, isolating the kernel's.
Scenario 1 is :30-37. :31 reports four-core machines with the kernel already tracking pressure
by default (psi=1), the KubeletPSI gate toggled, synchronized bursts that are practically
identical in both magnitude and frequency, and a conclusion: the collection logic is lightweight,
stays within the normal 0.1 cores or 2.5% of the total node capacity, and is safe for production.
The figure at :33 is titled (Case 1) Kubelet CPU Usage Rate Comparison and captioned Figure 2.
:35 turns to system CPU in the same run, says the enabled line follows the same pattern as the
disabled one with a slight expected increase from the baseline, and concludes that once the OS is
tracking PSI, at around 2.5 cores, the act of reading those cgroup metrics is negligible to
performance. The figure at :37 is titled (Case 1) System CPU Usage Rate Comparison and
captioned Figure 1.
Scenario 2 is :39-46, also on a four-core machine. :40 compares a cluster booted psi=1
(described as the COS default) against one booted psi=0 at 80-pod density under heavy I/O and CPU
load, and gives the system CPU delta as consistently between 0.037 cores and 0.125 cores or 0.925%
- 3.125% of the total node capacity, with a single spike to 0.225 cores, or 5.6%, brought back
down within seconds. The figure at :42 is Figure 3. :44 zooms in on the kubelet process, which
serves as the primary collector for these metrics, and reports that its usage never exceeds 0.25
cores or 6.25% of total capacity for longer than a second; the figure at :46 is Figure 4.
:48-50 is the section titled Improvements between beta (1.34) and stable (1.36), and it holds
one bullet. Previously, if the feature was on in Kubernetes but the kernel did not support PSI
(psi=0), the kubelet would emit misleading zero-valued metrics, which could trigger false
alarms when read as real metrics instead of missing values. In v1.36 the kubelet detects OS-level
PSI support via cgroup configurations before reporting, so pressure metrics are only collected and
emitted where they are supported.
:52-70 is Getting started. Two requirements at :56-57: a Linux kernel 4.20 or later on cgroup
v2, and PSI enabled at the OS level, meaning CONFIG_PSI=y and no psi=0 on the kernel command
line. :59 says that as of v1.36 the metrics are generally available and you do not need to opt in
to any feature gate. :61 tells you to scrape /metrics/cadvisor with a Prometheus-compatible
monitoring solution or query the Summary API, and notes that Windows nodes simply omit the metrics.
:63 offers the API server's kubelet proxy as a way to see real-time pressure data, :65 cautions
that proxying to the kubelet is a privileged operation, and :67-70 is the post's only fenced
block: a kubectl get --raw .../proxy/stats/summary piped into a jq filter that selects one
container by name and prints {name, cpu: .cpu.psi, memory: .memory.psi, io: .io.psi}.
The tail is :72-91. Further reading at :74-77 names the kernel's own PSI documentation, the
Understanding PSI page in the Kubernetes documentation, and cAdvisor's prometheus.go. :81
thanks SIG Node and dates the journey from alpha in v1.33, through beta in v1.34, to GA in v1.36.
:83-91 is two sets of contact details for the same group.
As it runs now
The page the post sends you to for the interpretation disagrees with itself by a factor of ten.
docs/reference/instrumentation/understand-psi-metrics.md is the Understanding PSI page named at
:76. Its :40 states the rule: the avg values represent the percentage of wall-clock time that
tasks were stalled over 10-second, 60-second and 5-minute moving averages. Its :95 applies the
rule to a worked number: a cpu.some avg10 of 0.74 means a task was stalled for 0.74% of the
time (0.0074 seconds or 74 milliseconds). Nought point seven four per cent of ten seconds is 0.074
seconds, which is 74 milliseconds. The two figures inside one parenthesis are ten times apart, and
only the second of them follows from the rule the same page gives five paragraphs earlier. Step 7
settles it against the counter that is not a percentage.
The sample it says that about is not JSON, and the identical bytes are printed in a second file.
The block at understand-psi-metrics.md:46-93 carries six trailing commas and eleven closing braces
against ten opening ones, so python3 -m json.tool refuses it at the fifteenth line and, once the
commas are removed, refuses it again at the thirty-first, where the extra brace ends the object and
leaves the io section outside it. The page introduces the block at :42-44 as what a jq filter
returns, and jq does not emit a trailing comma. The same block appears at
docs/concepts/cluster-administration/system-metrics.md:210-257, byte for byte, and the three
paragraphs of interpretation under it are byte-identical too, except that the second page adds one
sentence about Prometheus rate calculations. Two pages, one broken sample, and it is the only sample
of this API's PSI output in the tree.
The schema those samples are samples of is a link to a page the pin does not contain.
understand-psi-metrics.md:19 calls the Summary API the kubelet's Summary API and links
/docs/reference/config-api/kubelet-stats.v1alpha1/. No file under content/en renders that path:
docs/reference/config-api/ holds twenty-three files at the pin and none of them is the kubelet's
stats API. The same dead link occurs four times across three files —
docs/reference/instrumentation/node-metrics.md:13 and :53,
docs/reference/instrumentation/cri-pod-container-metrics.md:55, and the PSI page. So the JSON
block that does not parse is not a convenience beside the schema. It is the schema.
A placeholder made it into the argument. Back in the post, :18 reads a node may report XX%
(below 100%) CPU utilization while certain tasks are experiencing severe latency due to scheduling
delays. That sentence is the case for the whole feature, and its one number is the one nobody
filled in.
Figure 2 is printed above Figure 1. The figure at :33 carries caption="Figure 2: Kubelet CPU
Usage Rate Comparison" and the figure at :37 carries caption="Figure 1: Node System CPU Usage
Rate Comparison", while the prose at :35 introduces the second of them as the following graph.
Both are titled (Case 1) although the heading above them at :30 says Scenario 1, and the same
mismatch holds for :39 and the two (Case 2) figures below it. All four image files exist in the
pinned tree under static/images, so the figures render; it is their numbering that does not
survive being read in order.
Every percentage in the post divides by four cores, and the one number given without a percentage
is the one where that arithmetic would look strange. 0.1 cores is 2.5%, 0.037 is 0.925%,
0.125 is 3.125%, 0.225 is 5.6% and 0.25 is 6.25% — five conversions, all of them correct
for a four-core machine, which is the machine :31 and :40 both name. The exception is :35,
where system CPU is at around 2.5 cores and no percentage follows. On the same four cores that is
62.5%. The sentence is almost certainly describing the level both clusters sat at rather than the
cost of the feature, but it is the one place the post asks you to supply the reading, and it sits in
the paragraph that concludes the collection is negligible.
The improvement the GA release is named for cannot be observed on a release that has it. :50
says the kubelet used to emit zero-valued metrics on a node whose kernel had PSI off, and now
detects OS support before reporting. From v1.36 the gate is locked on and the detection is
unconditional, so the behaviour the bullet describes is unreachable from the release that describes
it. v1.35 is the last release that can show you the zeros, and the kernel gives a narrower lever
than a boot parameter: cgroup.pressure, one file per cgroup, which turns accounting off for a
subtree without rebooting anything. Step 8 uses it on one Pod.
Getting started is the beta announcement's Getting started with the gate step taken out. The
beta post of 2025-09-04 carries a two-item list at its :55-56 whose second item is Enable the
KubeletPSI feature gate on the kubelet; this post's list at :56-57 keeps the first item word
for word and replaces the second with the OS-level requirement. The paragraph under it is the same
paragraph: this post's :61 is the beta post's :58 with Once enabled changed to Once the OS
prerequisites are met and the closing clause about Windows nodes reworded. The beta announcement is
a read row in 2025's census, which is why it has no exercise of its own and
why this one carries the comparison.
Three releases end to end, and the middle one is named nowhere. The post's :81 dates the
journey alpha v1.33, beta v1.34, GA v1.36, and the heading at :48 calls the gap Improvements
between beta (1.34) and stable (1.36). The gate file
command-line-tools-reference/feature-gates/KubeletPSI.md records beta as running 1.34 to 1.35, so
beta lasted two releases and not one. v1.35 is the release the clusters in this lab run, and the
post's account of the feature skips straight over it.
This gate really is locked, and the file says so. KubeletPSI.md carries locked: true on its
stable stage, and two prose pages agree in the same words: Starting with Kubernetes v.1.36, the
KubeletPSI feature gate is locked to true and cannot be disabled (understand-psi-metrics.md:16
and node-metrics.md:54). That is worth saying plainly because the other v1.36 kubelet post of this
year, published eighteen days earlier, makes the identical claim about a gate whose file carries no
such key — the fine-grained kubelet authorization
row counts how many files do. The
post at :59 puts it the other way round and is also right: there is nothing left to opt into.
The two sentences that say so carry the same typo, and a third page carries another. Both write
v.1.36 with a full stop after the v, in a tree where the convention is v1.36;
system-metrics.md:265 offers realitime in a sentence that appears nowhere else. The three PSI
passages — node-metrics.md:46-64, understand-psi-metrics.md:11-29 and
system-metrics.md:178-265 — each open with the same feature-state shortcode for the same gate,
each carry a Requirements list of the same two or four bullets, and two of them close with the same
whatsnext sentence, word for word, down to a metrics pipeline that rely on these data.
The endpoint the post tells you to scrape is the one the pin has begun to deprecate. Its :61
and :76 both send a reader to /metrics/cadvisor, and the 2016 node dashboard
row reads the pinned notice that
cAdvisor-based pod and container metrics collection in the kubelet is deprecated as of v1.37, to be
served instead from the CRI runtime under the same endpoint path and the same metric names. Nothing
in the tree says whether a CRI runtime supplies container_pressure_*. The six names are written in
exactly two files, system-metrics.md:190-195 and the beta announcement's own list, and neither of
those is docs/reference/instrumentation/metrics.md, which is where the pin keeps the metrics
reference and which does not contain the string container_pressure at all.
The examples that generate the pressure pin an image three tags behind the newest the same tree
names. understand-psi-metrics.md:120, :178 and :232 all run
registry.k8s.io/e2e-test-images/agnhost:2.47. Four tags of that image appear under
content/en/docs — 2.40, 2.45, 2.47 and 2.53 — and the newest of them is used once, in
concepts/workloads/pods/advanced-pod-config.md:113. The PSI page also states at its :104 that
the image includes the stress tool and then passes stress as the container's first argument,
which is not how a tool inside an image is usually invoked and is not written down anywhere else in
the tree. That Pod is applied exactly as printed in step 3, and what happens to it is the finding.
The one command the requirements give you for checking the kernel needs a kernel option the same
page never mentions. understand-psi-metrics.md:27 and
docs/reference/node/kernel-version-requirements.md:68 both say to run zgrep CONFIG_PSI
/proc/config.gz. That file exists only where the kernel was built with its configuration embedded
and the module that exposes it is loaded, which is a second requirement, unstated, and a different
one from CONFIG_PSI. Step 1 runs the documented check and the one that works on the node beside
it.
What this exercise does not cover, and where it lives. cAdvisor, the Summary API,
metrics-server and the four endpoints the kubelet serves are the 2015 resource usage
row, and the deprecation of cAdvisor-based
collection, the gate that decides which collector answers on /metrics/cadvisor, and the metric
that tells you which one did are the 2016 node dashboard
row. cgroup v2 as a requirement,
and the kernel-floor page this post's requirements are drawn from, are the 2024 cgroup v1
maintenance row, which already reads the sentence at
understand-psi-metrics.md:29. Turning a kubelet feature gate off on a v1.35 node, and the count of
gate files that carry a lock, belong to the fine-grained kubelet authorization
row; step 9 reuses the mechanics and
does not re-derive them. Nothing here reproduces the post's graphs: four line charts over an 80-pod,
4-core load test are not a thing one afternoon on one node can redraw, and step 9 measures a single
number in the same units instead.
The diff, and why
Wrong when it was published. The sentence the reader needs most is off by ten. A cpu.some
avg10 of 0.74 is 74 milliseconds of stall in the trailing ten seconds, not 7.4, and the page the
post names for interpretation prints both numbers inside one pair of brackets. Beside it sits a
sample of the API output that no JSON parser accepts, and beside that a link to the schema that
resolves to nothing in the pinned tree. In the post itself, the motivating figure is still XX%,
the two figures of Scenario 1 are captioned in the reverse of the order they appear, and the Getting
started section is the beta announcement's with its second step swapped out. None of this waited on
a later release to go wrong.
Still right. The measurement stands up. Every percentage the post quotes is the quoted core count divided by four, and four is the core count its own methodology section gives, so the arithmetic is internally consistent and the overhead it reports is small enough that a reader is right to stop worrying about it. The metric names are correct, the endpoint it names does serve them at the pin, the some-versus-full distinction is the kernel's own, and the claim that Windows nodes simply omit the series is repeated in three places with nothing anywhere contradicting it. The lock it asserts is in the gate file this time. Step 9 reproduces the shape of the finding on one node rather than eighty pods across a fleet, and the expectation is that it agrees.
Overtaken by stasis. The improvement the release is proudest of is unobservable from the release
that ships it. A gate locked on at v1.36 cannot be turned off to show you the zero-valued metrics it
stopped emitting, and no page in the pin tells a reader how else to get a node into that state; the
kernel's own cgroup.pressure switch, which does it per cgroup and without a reboot, is not written
down anywhere under content/en. Meanwhile the endpoint all four pages send you to is the one the
pin has already announced will stop being served by the collector that fills it, one release later.
The documentation for this feature was finished at the moment it became hardest to demonstrate.
Never absorbed. Four pages tell this story and none of them is the reference. The six Prometheus
series are written in system-metrics.md and in a blog post, and container_pressure does not
occur in docs/reference/instrumentation/metrics.md at all, which is where the tree keeps the list
of what the components expose. The GA graduation added a stable stage to a gate file that is never
rendered, and the same two sentences with the same v.1.36 typo to two prose pages. What a reader
coming to PSI cold actually finds is one page repeated three times, a sample that does not parse,
and a dead schema link in four places.
The ladder
KubeletPSI:
| stage | default | locked | releases |
|---|---|---|---|
| alpha | false |
— | v1.33 |
| beta | true |
— | v1.34 – v1.35 |
| stable | true |
true |
v1.36 – |
Three releases end to end. That is short: the pin's gate files record twenty-one releases for the longest ladder in the tree, and this one is done in three. The shortness constrains the lab in one specific way. The clusters run v1.35, the last release in which the gate is settable, so step 9 can measure the kubelet with PSI collection on and with it off and subtract. From v1.36 the stable row is the only row and it carries the lock, so that subtraction is not repeatable on a newer cluster — which is also, exactly, why the post's own Scenario 1 could be run before the feature graduated and not after.
The locked column is filled in here, and it is worth being precise about what fills it: the key is
in the gate file's stable stage, the file is marked render: false so nothing on the site displays
it, and two prose pages assert the lock in their own words. That is three sources agreeing, which is
three more than the other v1.36 kubelet feature of this year has.
Topology — solo, one node at 10.10.10.180, 4096MB, 4
CPUs, 25G, provisioned and
baselined as usual, running Kubernetes v1.35.
The core count is not incidental: the post's machines have four cores, every percentage it prints is
a core figure divided by four, and a node with the same four cores is what lets step 9 quote its
result in the post's units and compare. One node is also the only honest topology here, because
pressure is a property of one kernel's cgroups and adding a second machine would only give you a
second, unrelated set of numbers. Everything created lives in a namespace called bw-psi, except
one appended block in the kubelet's configuration file and one write to a cgroup control file, which
Teardown undoes. Steps 1 to 9 run on the node over ssh zain@10.10.10.180; step 10 runs offline
against a checkout of kubernetes/website at the pin, with W set to its content/en directory.
Do
- Establish that the kernel underneath is one that can answer at all, using the check the
documentation gives and the check that works. Open one session with
ssh zain@10.10.10.180and stay in it; every fence up to step 10 is written as though you are already there. Write down the threetotalfigures from/proc/pressure— step 7 comes back for them.
kubectl version -o json | grep gitVersion
N=$(kubectl get nodes -o jsonpath='{.items[0].metadata.name}'); echo "node $N"
nproc; uname -r
zgrep CONFIG_PSI /proc/config.gz || echo "no /proc/config.gz on this node"
grep -E '^CONFIG_PSI' /boot/config-$(uname -r)
grep -w psi /proc/cmdline || echo "no psi= parameter on the kernel command line"
stat -f -c %T /sys/fs/cgroup
for r in cpu memory io; do echo "== $r"; cat /proc/pressure/$r; done
kubectl get --raw "/api/v1/nodes/$N/proxy/metrics" \
| grep 'kubernetes_feature_enabled.*KubeletPSI'
kubectl create namespace bw-psi
- Read the same numbers one layer down, where the kubelet reads them, before any workload of yours exists. The per-cgroup files carry the same four fields as the per-node ones and one extra control file that step 8 uses.
C=/sys/fs/cgroup/kubepods.slice
ls -1 $C | head -20
for r in cpu memory io; do echo "== $r"; cat $C/$r.pressure; done
cat $C/cgroup.pressure
grep -c . /proc/pressure/cpu
- Apply the Pod the documentation itself prints for generating CPU pressure, exactly as written, and find out whether the argument it passes is a thing. The image tag is the one the page pins, three tags behind the newest the same tree names elsewhere.
cat <<'EOF' | kubectl -n bw-psi apply -f -
apiVersion: v1
kind: Pod
metadata:
name: cpu-pressure-pod
spec:
restartPolicy: Never
containers:
- name: cpu-stress
image: registry.k8s.io/e2e-test-images/agnhost:2.47
args:
- "stress"
- "--cpus"
- "1"
resources:
limits:
cpu: "500m"
requests:
cpu: "500m"
EOF
sleep 20
kubectl -n bw-psi get pod cpu-pressure-pod -o wide
kubectl -n bw-psi logs cpu-pressure-pod --tail=20 || true
kubectl -n bw-psi get pod cpu-pressure-pod \
-o jsonpath='{.status.containerStatuses[0].state}{"\n"}'
- Generate the pressure with an image whose arguments are not in question, and watch the number move. One container throttled to a quarter of a core, four busy loops inside it: the stall is manufactured by the cgroup limit, not by the machine being busy.
kubectl -n bw-psi delete pod cpu-pressure-pod --ignore-not-found --wait=true
cat <<'EOF' | kubectl -n bw-psi apply -f -
apiVersion: v1
kind: Pod
metadata:
name: bw-psi-load
spec:
containers:
- name: burn
image: busybox:1.36
command: ["sh","-c","for i in 1 2 3 4; do while :; do :; done & done; sleep 3600"]
resources:
limits:
cpu: "250m"
EOF
kubectl -n bw-psi wait --for=condition=Ready pod/bw-psi-load --timeout=60s
U=$(kubectl -n bw-psi get pod bw-psi-load -o jsonpath='{.metadata.uid}')
D=$(find /sys/fs/cgroup/kubepods.slice -type d | grep -E "$U|${U//-/_}" | head -1); echo "$D"
for i in 1 2 3; do cat $D/cpu.pressure; sleep 10; done
- Run the one-liner the post sends you to, against the container you just made stall, and ask a parser what it thinks of the result. The filter is the page's; the validation is not.
N=$(kubectl get nodes -o jsonpath='{.items[0].metadata.name}')
kubectl get --raw "/api/v1/nodes/$N/proxy/stats/summary" > /tmp/bw-summary.json
jq empty /tmp/bw-summary.json && echo "summary parses"
jq '.pods[] | select(.podRef.name=="bw-psi-load") | .containers[].cpu' /tmp/bw-summary.json
jq '.pods[] | select(.podRef.name=="bw-psi-load")
| {name: .podRef.name, cpu: .cpu.psi, memory: .memory.psi, io: .io.psi}' /tmp/bw-summary.json
jq '[paths | select(.[-1]=="psi")] | length' /tmp/bw-summary.json
- Ask the Prometheus endpoint for the same thing under its other set of names, and ask the kubelet which collector answered. Six series names are what the pin writes down; count what the node serves.
N=$(kubectl get nodes -o jsonpath='{.items[0].metadata.name}')
kubectl get --raw "/api/v1/nodes/$N/proxy/metrics/cadvisor" > /tmp/bw-cadvisor.txt
grep -c '^container_pressure' /tmp/bw-cadvisor.txt
grep '^# TYPE container_pressure' /tmp/bw-cadvisor.txt
grep '^container_pressure_cpu_stalled_seconds_total' /tmp/bw-cadvisor.txt \
| grep 'pod="bw-psi-load"'
kubectl get --raw "/api/v1/nodes/$N/proxy/metrics" | grep -E '^kubelet_metrics_provider' || \
echo "no kubelet_metrics_provider series on this release"
grep -o 'container_pressure_[a-z_]*' /tmp/bw-cadvisor.txt | sort -u
- Settle the factor of ten.
avg10is a percentage of the trailing ten seconds andtotalis a monotonic microsecond counter of the same stall, so the difference intotalacross exactly ten seconds, divided by ten seconds, is the same quantity computed two ways. Read them off the same file in the same breath and do the arithmetic.
U=$(kubectl -n bw-psi get pod bw-psi-load -o jsonpath='{.metadata.uid}')
D=$(find /sys/fs/cgroup/kubepods.slice -type d | grep -E "$U|${U//-/_}" | head -1)
psi() { tr '=' ' ' < $D/cpu.pressure | awk 'NR==1{print $3, $9}'; }
set -- $(psi); A0=$1; T0=$2
sleep 10
set -- $(psi); A1=$1; T1=$2
echo "avg10 now: $A1 % delta total: $((T1-T0)) us over 10 s"
awk -v d=$((T1-T0)) 'BEGIN{printf "that is %.4f s stalled, or %.2f%% of 10 s\n", d/1e6, d/1e5}'
awk -v a=$A1 'BEGIN{printf "%.2f%% of 10 s is %.4f s = %.1f ms\n", a, a/100*10, a/100*10000}'
awk 'BEGIN{printf "the page: 0.74%% of 10 s is %.4f s = %.1f ms\n", 0.0074*10, 0.0074*10000}'
- Reproduce the state the GA release stopped emitting misleading numbers for, on a release that
still can. Turning accounting off for one cgroup is the narrow version of booting with
psi=0: the files stay, the numbers stop.
U=$(kubectl -n bw-psi get pod bw-psi-load -o jsonpath='{.metadata.uid}')
D=$(find /sys/fs/cgroup/kubepods.slice -type d | grep -E "$U|${U//-/_}" | head -1)
cat $D/cpu.pressure
echo 0 | sudo tee $D/cgroup.pressure
sleep 15
cat $D/cpu.pressure
N=$(kubectl get nodes -o jsonpath='{.items[0].metadata.name}')
kubectl get --raw "/api/v1/nodes/$N/proxy/stats/summary" \
| jq '.pods[] | select(.podRef.name=="bw-psi-load") | .cpu.psi'
echo 1 | sudo tee $D/cgroup.pressure
- Measure what the post measured, in the post's units. Eighty pods is the load its Scenario 1 used; the kubelet's own CPU counter is the overhead. Run it with collection on, turn the gate off, run it again, and divide by four cores to get a percentage.
N=$(kubectl get nodes -o jsonpath='{.items[0].metadata.name}')
kubectl -n bw-psi create deployment bw-psi-fill --image=registry.k8s.io/pause:3.10 --replicas=80
kubectl -n bw-psi rollout status deployment/bw-psi-fill --timeout=300s
K() { kubectl get --raw "/api/v1/nodes/$N/proxy/metrics" \
| awk '/^process_cpu_seconds_total/{print $2}'; }
A=$(K); sleep 120; B=$(K)
awk -v a=$A -v b=$B 'BEGIN{printf "gate on: %.4f cores, %.2f%% of 4\n",(b-a)/120,(b-a)/120/4*100}'
sudo cp /var/lib/kubelet/config.yaml /var/lib/kubelet/config.yaml.bw
printf 'featureGates:\n KubeletPSI: false\n' | sudo tee -a /var/lib/kubelet/config.yaml
sudo systemctl restart kubelet; sleep 60
kubectl get --raw "/api/v1/nodes/$N/proxy/metrics/cadvisor" | grep -c '^container_pressure' || true
A=$(K); sleep 120; B=$(K)
awk -v a=$A -v b=$B 'BEGIN{printf "gate off: %.4f cores, %.2f%% of 4\n",(b-a)/120,(b-a)/120/4*100}'
-
Leave the cluster and count the documentation. Run this in a checkout of
kubernetes/websiteat the pin, withWset tocontent/en. Every number the As it runs now section claims is here; the first command is the one that decides whether the sample a reader is meant to learn from is data or prose.cd /path/to/kubernetes/website/content/en W=$(pwd) sed -n '47,92p' $W/docs/reference/instrumentation/understand-psi-metrics.md > /tmp/a.json sed -n '211,256p' $W/docs/concepts/cluster-administration/system-metrics.md > /tmp/b.json python3 -m json.tool < /tmp/a.json || echo "the sample does not parse" python3 -c "import re,json,sys; t=re.sub(r',(\s*[}\]])',r'\1',open('/tmp/a.json').read()); json.loads(t)" cmp /tmp/a.json /tmp/b.json && echo "byte identical" python3 -c "import re; s=open('/tmp/a.json').read(); print(len(re.findall(r',\s*[}\]]',s)))" grep -o '{' /tmp/a.json | wc -l; grep -o '}' /tmp/a.json | wc -l grep -rn 'kubelet-stats.v1alpha1' $W --include='*.md' | wc -l grep -rln 'v\.1\.36' $W --include='*.md' grep -c 'container_pressure' $W/docs/reference/instrumentation/metrics.md grep -rho 'agnhost:2\.[0-9]*' $W/docs --include='*.md' | sort | uniq -c grep -n 'XX%' $W/blog/_posts/2026/psi-metrics-ga.md
Expect
Step 1 prints v1.35.x, four cores and a Debian kernel well past 4.20. The documented check fails:
/proc/config.gz does not exist on a stock Debian cloud image, because the kernel is built without
CONFIG_IKCONFIG_PROC, which is a different option from the one the command is grepping for and
which the pin never mentions. The /boot/config-$(uname -r) line is the one that answers, and it
prints CONFIG_PSI=y; CONFIG_PSI_DEFAULT_DISABLED will be there too, set to n, which is what
makes the absent psi= parameter harmless. stat -f prints cgroup2fs. All three /proc/pressure
files answer with a some line and, for memory and io, a full line; cpu has a full line as
well on a recent kernel and its values will be zero. The kubernetes_feature_enabled series reports
KubeletPSI at 1 — it is beta and defaults on at v1.35 — and the node is idle enough that every
avg figure is 0.00.
Step 2 shows the same four fields per cgroup. cpu.pressure, memory.pressure and io.pressure
all exist under kubepods.slice, which is the answer to whether the kernel is accounting per cgroup
and not just per node, and cgroup.pressure contains 1. On an idle node the kubepods.slice
figures are zero or very close to it. grep -c . on the node-level cpu file prints 2 on a kernel
new enough to report full for CPU and 1 on an older one; either is fine and nothing downstream
depends on it.
Step 3 is the one prediction here that could go either way, and the prediction is that it fails.
stress is not an agnhost subcommand the pin documents anywhere else, and a binary that dispatches
on its first argument normally exits non-zero on an argument it does not know, printing the list it
does know. With restartPolicy: Never the Pod does not loop: it goes straight to Failed, with
terminated and a non-zero exit code in the container state, within a second or two of starting.
The logs are the evidence either way. If instead the container stays up and burns half a core, the
page is right and this exercise is wrong about it — write down which happened, because it is the one
claim in this file that cannot be settled by reading the pin.
Step 4 is where the number first moves. Four spin loops inside a container limited to 250 millicores
means the cgroup is throttled roughly fifteen sixteenths of the time, so cpu.pressure's some
avg10 climbs within one ten-second window and settles somewhere above 80. The three prints ten
seconds apart should show it rising and then flat. full for CPU will be well above zero too,
because every task in the cgroup is stalled at once when the quota is exhausted, which is exactly
what full means and is the distinction the pin explains but never demonstrates.
Step 5 produces real output in the shape the broken sample is trying to show. The summary parses —
jq empty is silent — and the container's cpu object carries a psi sibling alongside
usageNanoCores. The count of psi keys in the document is one per resource per container, plus
one per resource per pod, plus one per resource for the node: on a cluster with a handful of pods it
is a two-digit number, and the point of printing it is that every one of those objects has the
structure the documented sample fails to close a brace on.
Step 6 answers with the Prometheus spelling. The count of container_pressure lines is non-zero and
the # TYPE lines name the six series the pin writes down — stalled and waiting, for cpu, memory
and io — as counters in seconds rather than the kernel's microseconds, and the sorted list at the
end of the step prints exactly those six and nothing more. The bw-psi-load line has a value that
keeps climbing between scrapes. kubelet_metrics_provider will almost certainly not be there: it
belongs to the CRI-collection path and this node is on the cAdvisor one, which is the path the
post's instructions assume and the path the pin has announced it is moving off.
Step 7 is the whole question in nine lines. total is microseconds and ten seconds is ten million
of them, so the delta expressed as a percentage of ten million is the same quantity avg10 reports,
and the two will agree to within the sampling jitter — for a cgroup stalling this hard, both land in
the eighties. Follow the same arithmetic down to the page's worked figure: 0.74% of ten seconds is
0.074 seconds, and the last line prints 74.0 ms. The printed 0.0074 seconds in the pinned page
is ten times smaller than the 74 milliseconds it is offered as a restatement of, and the counter
says which one is right.
Step 8 reproduces the zeros. Writing 0 to cgroup.pressure stops accounting for that cgroup; the
file stays readable and its avg fields decay to 0.00 within the first ten-second window while
the container carries on being throttled exactly as hard as before. total freezes rather than
resetting. The Summary API then reports a psi object full of zeros for a container that is
stalling constantly — which is precisely the misleading output the v1.36 note says the kubelet
stopped producing, obtained on a release that still produces it, at cgroup granularity instead of
node granularity. Writing 1 back restores accounting and avg10 climbs again.
Step 9 should land where the post landed. Eighty pause pods on four cores is a lot of bookkeeping
and very little work, so the kubelet's own CPU sits in the low hundredths of a core in both runs and
the difference between them is smaller than the noise of a single 120-second sample. Expect
something in the region of 0.03 to 0.06 cores, which is 0.75% to 1.5% of four — the same range the
post's figures fall in — and expect the gate-off run to be lower by an amount you would not bet on.
The container_pressure count with the gate off is 0, which is the one unambiguous confirmation
that the flip took. The kubelet restarts once here and once more in Teardown; on a single-node
cluster each restart is a short outage of the only node and nothing else.
Step 10 confirms the documentation findings offline. json.tool refuses the sample at line 15
column 5, and with the six trailing commas properly removed the second command still refuses it, at
line 31 column 4, where the extra closing brace ends the object and leaves io outside it. The two
extracted blocks are byte identical, so cmp is silent; the trailing-comma count prints 6 and the
brace counts 10 and 11. The dead schema link appears four times. The v.1.36 grep names two
files. container_pressure in the metrics reference counts 0. The agnhost tally puts 2.47 at
three occurrences, all in the PSI page, against one occurrence of the newer 2.53 elsewhere in the
tree. And XX% is still there in the post.
Read on
-
Resource Usage Monitoring in Kubernetes — cAdvisor, the Summary API and the four endpoints the kubelet serves, which is where
/stats/summaryand/metrics/cadvisorcome from and why they carry different spellings of the same data. -
Visualize Kubelet Performance with Node Dashboard — the deprecation that moves pod and container metrics collection from cAdvisor to the CRI runtime, the gate that decides which one answers, and the series that tells you which one did.
-
cgroup v1 maintenance mode — why cgroup v2 is a requirement rather than a preference, and the kernel-floor page this post's requirements list is drawn from.
-
Fine-grained kubelet API authorization — appending a
featureGatesblock to a kubelet configuration and restarting into it, and the count of gate files that carry a lock on their stable stage. -
Unanswerable from the pin: whether the
container_pressure_*series survive the move to CRI collection. The deprecation notice promises the same endpoint and the same metric names from the runtime instead of cAdvisor, and PSI arrives at the kubelet through cAdvisor's cgroup reader. No page in the tree says which of the six names a CRI runtime is expected to supply, and no runtime's documentation is in reach of the pin to check.
Teardown
The namespace takes the two pods and the eighty-replica deployment. The one thing that outlives it
is the appended featureGates block in the kubelet's configuration file: step 9 takes the backup
and never puts it back, so the restore below is the only thing that does. Run it even if step 9 was
interrupted, because a node left in that state collects no pressure metrics at all. The last command
is the check that it worked — a non-zero count.
ssh zain@10.10.10.180 "kubectl delete namespace bw-psi --wait; \
sudo mv -f /var/lib/kubelet/config.yaml.bw /var/lib/kubelet/config.yaml; \
sudo systemctl restart kubelet; sleep 30; \
kubectl get --raw /api/v1/nodes/\$(kubectl get nodes \
-o jsonpath='{.items[0].metadata.name}')/proxy/metrics/cadvisor \
| grep -c '^container_pressure'"
The cgroup.pressure write from step 8 needs no undoing beyond the 1 the step already puts back,
and it disappears with the cgroup when the namespace goes. The node itself is
destroyed in the usual way when you are finished with
it.