Skip to content

The per-Pod mitigation in this post is still the pinned docs' own first security example and is no longer sufficient, the rule name in its global one survives in one line of the whole tree, and the project's answer was to make the post's premise false and then stop checking

Post — Runc and CVE-2019-5736, 11 February 2019, by the Kubernetes Product Security Committee. 101 lines, 6,399 bytes.

The only post in this year's walk that announces no feature. It is an incident bulletin, published the morning an embargo lifted, and it is the one kind of post where being out of date is the intended outcome: the fix shipped, the distributions patched, the advisory links went stale on purpose. What is worth walking is not the vulnerability. It is the fact that a security bulletin has to give advice in the API of the day, and this one gave three pieces, and seven years later they have come apart in three different directions.

The post's own frontmatter records that the project already noticed. Line :4 reads evergreen: false # mentions PodSecurityPolicy — a machine-readable admission, in the file itself, that one of the three no longer holds.

As written — runc is "the low-level tool which does the heavy lifting of spawning a Linux container" (:13), Docker, Containerd and CRI-O sit on top of it, and "while no part of Kubernetes itself is vulnerable, most Kubernetes installations are using runc under the hood" (:15).

The vulnerability, still embargoed at the time, is stated as a premise rather than a mechanism (:19):

when running a process as root (UID 0) inside a container, that process can exploit a bug in runc to gain root privileges on the host running the container. This then allows them unlimited access to the server as well as any other containers on that server.

Then the boundary conditions (:21): the vulnerability does not apply if the process is trusted or "is not running as UID 0", and "It can also be prevented by SELinux, if an appropriate policy has been applied." The stated source of risk (:23) is "attacker-controller container images, such as unvetted images from public repositories" — the typo is the post's.

The advice (:27-61) is upgrade runc, or mitigate. Three mitigations, in the order the post gives them. First, per Pod, because "the exploit requires UID 0 within the container" (:29):

---
apiVersion: v1
kind: Pod
metadata:
  name: run-as-uid-1000
spec:
  securityContext:
    runAsUser: 1000
  # ...

Second, the same thing globally — "This can also be enforced globally using a PodSecurityPolicy" (:43):

---
apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
  name: non-root
spec:
  privileged: false
  allowPrivilegeEscalation: false
  runAsUser:
    # Require the container to run without root privileges.
    rule: 'MustRunAsNonRoot'

with a sentence of advocacy after it (:59): "Setting a policy like this is highly encouraged given the overall risks of running as UID 0 inside a container." Third (:61), vet your images, "by building all your images yourself, or by vetting the contents of an image and then pinning to the image version hash", and the post writes the form inline as image: external/someimage@sha256:7832659873hacdef.

The rest (:63-101) is the perishable half: safe package versions for seven platforms, four provider bulletins, and contact addresses.

As it runs now — take the three mitigations in the post's order, because they have not aged at the same rate.

The first one still applies, and more than applies: it is the pin's own opening example of a security context. content/en/examples/pods/security/security-context.yaml — the file that security-context.md:61 renders as the first code sample on the task page — sets securityContext.runAsUser: 1000 at :7, exactly the post's field with exactly the post's value, and allowPrivilegeEscalation: false at :22, which is the post's second fence's field. Between them the post's two mitigations supply two of the four fields in the pin's canonical example, and security-context.md:63 still explains runAsUser in the present tense.

And it is no longer sufficient, on a field the post never names. The successor to the post's global mechanism validates namespaces against the Pod Security Standards, and the restricted profile's Running as Non-root control at pod-security-standards.md:396-414 restricts spec.securityContext.runAsNonRoot with an Allowed Values list of exactly one entry, true. Undefined is not in that list. A separate control, Running as Non-root user (v1.23+) at :417-431, is the one that governs runAsUser, and its allowed values are "any non-zero value" and undefined/null. So the post's Pod satisfies the control about the field it sets and violates the control about the field it does not, and the violated one is the control whose name is the post's own argument.

The post did name the concept, once, and only in the global fence: rule: 'MustRunAsNonRoot'. At the pin that string occurs in the entire content/en/docs tree exactly once, at psp-to-pod-security-standards.md:143, and it occurs there as the left-hand side of a translation: the runAsUser row of a table mapping PodSecurityPolicySpec fields onto profiles, whose Restricted cell reads "rule is MustRunAsNonRoot". The rule name in the post survives only inside the document written to translate it away.

The second mitigation is a dead API, and this walk has already been there. policy/v1beta1 PodSecurityPolicy has not been served since v1.25, and the removal, the migration table, and the tense the deprecation guide uses about it are walked in 2018 against a post that carried its own PodSecurityPolicy fragment. Nothing here re-reads those pages. What is this post's own is narrower and stranger: this bulletin's fence is the reason the tree still contains a MustRunAsNonRoot, and step 1 goes and finds it.

The third mitigation is the one that has become more documented rather than less. The post writes digest pinning as a parenthesis; the pin gives it a section. images.md:40-43 distinguishes tags from digests — "Tags can be moved to point to different images, but digests are fixed" — :52-53 names the format, and :59-68 lists six forms of image name, of which two are the post's and one is a form the post does not mention at all: registry.k8s.io/pause:3.5@sha256:1ff6c18fbef2045af6b9c16bf034cc421a29027b800e4f9b68ae9b1cb3e9ae07, described as "Image name with tag and digest. Only the digest will be used for pulling." Seven years on, the only one of the three mitigations that needs no translation is the one the post threw in at the end.

The boundary condition has aged best of all. The post's SELinux clause is the single sentence of this bulletin that the pinned documentation still repeats, and it repeats it with the same CVE number: linux-kernel-security-constraints.md:215-219, a note under the table of Linux kernel security modules, says "Mechanisms like AppArmor and SELinux can provide protection that extends beyond the container. For example, you can use SELinux to help mitigate" — and then links CVE-2019-5736. That is the only occurrence of the string CVE-2019-5736 anywhere in content/en/docs.

And here is the part the post could not have written. The project's eventual answer to root in a container is root on the host was not to enforce non-root harder. It was to make the premise false, and then to stop checking. user-namespaces.md:73 gives the opt-in as pod.spec.hostUsers, and :255-273 states the consequence for admission: for Linux Pods that enable user namespaces, Kubernetes "relaxes the application of Pod Security Standards in a controlled way", and the eight fields it then lists as "not checked" under Baseline or Restricted are the four runAsNonRoot fields and the four runAsUser fields. Both of the post's fields. The justification (:260-263) is the negation of the post's :19:

This behavior does not present a security concern because root inside a Pod with user namespaces actually refers to the user inside the container, that is never mapped to a privileged user on the host.

The post says UID 0 in a container is UID 0 on the host, so do not be UID 0. The pin says if UID 0 in the container is not UID 0 on the host, we will not ask whether you are UID 0. Same sentence, opposite direction, and the second one is why the first one is optional.

What this exercise does not cover, and where it lives. The removal of PodSecurityPolicy, the field-by-field mapping table read as a whole, and the tense the deprecation guide uses about it are covered in 2018, which is the exercise that owns them; step 1 opens that mapping table at one line and reads no further. The PodSecurity and UserNamespacesSupport ladders, and the reframing of non-root advice that the second one implies, are covered in 2016; the ladder below is a third gate that neither of them names. The runtime layer the post's :13 describes — what sits on top of runc, and what the pin documents in its place — is covered in 2017. And the labelling scheme that turns a namespace into an enforced profile is described in both of those security exercises; steps 3 onward use kubectl label as a tool and do not re-explain it.

The diff, and why — this is a bulletin, so the interesting cases are not the ones about features.

Mitigation two broke, in the ordinary way an API version breaks, and its breakage is the one the post's own frontmatter flags. Nothing subtle happened: the kind was withdrawn, the manifest no longer resolves, and a reader who applies it gets an error naming the kind.

Mitigation one is still right and no longer enough, which is the case this walk sees least often. It is not that the field changed meaning or that the advice was wrong. The advice was correct, was adopted, became a docs example, and was then joined by a second field expressing the same intent more precisely — and the enforcement mechanism that replaced the post's was written against the second field. The post's Pod is admitted by baseline and rejected by restricted, and the rejection is not a disagreement about security; it is a disagreement about which field carries the claim. runAsUser: 1000 asserts a number. runAsNonRoot: true asserts a property. The successor checks the property, because a number in a Pod spec can be overridden by an image whose USER directive the API server cannot see, and the property cannot.

Mitigation three is still right, unchanged, and better documented, which in this corpus is rare enough to be worth stating plainly. The post's throwaway parenthesis is now a six-item list with a documented precedence rule.

And the whole post was retired by being agreed with, twice over, at two different layers. Its recommendation became a namespace-level default that nobody has to switch on — the gate for that is already laddered in 2016. Then the premise it argues from was itself dismantled, and the dismantling came with its own gate, which is the one below, and which was deleted for the most interesting reason in the whole gate directory.

The ladder — one gate, and it is the only gate in this year's set that never reached beta.

UserNamespacesPodSecurityStandards

stage default locked releases
alpha false — v1.29 – v1.34

The file declares removed: true. Six releases at alpha, no beta row, no stable row, and gone. Its body is the relaxation described above — "Enable Pod Security Standards policies relaxation for pods that run with namespaces. You must set the value of this feature gate consistently across all nodes in your cluster, and you must also enable UserNamespacesSupport to use this feature." — and then, unusually, the file explains its own deletion:

This feature gate is not part of Kubernetes v1.35 (and onwards) because, for those versions of Kubernetes, all supported kubelet versions will fail to create a pod if it requests a user namespace, but the node does not.

Read that against the four-column table and it is a fourth kind of ending. The other retirements in this year are switches deleted because the answer became unconditional. This one is a switch deleted because the hazard it guarded became impossible: the gate existed so that admission would not relax a check on the strength of a hostUsers: false that the node might silently ignore, and once every supported kubelet fails closed on that request instead of ignoring it, there is nothing left for the gate to guard. The relaxation did not go away — user-namespaces.md:255-273 still describes it, ungated. The doubt went away.

There is no PodSecurityPolicy gate file to ladder: the mechanism the post's second fence uses was an admission controller and an API version, never a feature gate, so its removal leaves no four-column trace at all. That absence is worth noticing before step 8 goes looking for it.

Topology — solo: one node at 10.10.10.180. Every check below is either a read of the pinned checkout or an admission decision, and admission decisions are the API server's, so one node carries all of them; step 7 is the only one that touches the kubelet, and it wants a node whose /etc/subuid you are willing to look at and leave alone. Bring it up with the five provision steps, substituting topology=solo, then ssh zain@10.10.10.180. Steps 1, 2, 5, 8 and 9 read the checkout and need no cluster.

Do

  1. Find the post's rule name in the tree, and stop where it stops:
W=/path/to/pinned/website/content/en/docs
grep -rn 'MustRunAsNonRoot' $W | wc -l
grep -rn 'MustRunAsNonRoot' $W
sed -n '139,145p' $W/reference/access-authn-authz/psp-to-pod-security-standards.md

One line in the whole tree. Read the seven lines around it and note which column the post's string is in, and which column the reader is being sent to. Do not read further down that table; the table as a whole belongs to another exercise.

  1. Read the pin's own first security example against the post's two fences:
E=/path/to/pinned/website/content/en/examples
cat $E/pods/security/security-context.yaml
sed -n '61p;63,66p' $W/tasks/configure-pod-container/security-context.md

Write down which fields of the post's two YAML fences appear in the example, and which field of the example appears in neither fence.

  1. Build three namespaces, one per profile, and use the post's Pod as the probe. Nothing is created: every apply here carries the flag from the exercise before this one.
for L in privileged baseline restricted; do
  kubectl create ns psa-$L
  kubectl label ns psa-$L pod-security.kubernetes.io/enforce=$L
done
cat > /tmp/post-pod.yaml <<'YAML'
apiVersion: v1
kind: Pod
metadata:
  name: run-as-uid-1000
spec:
  securityContext:
    runAsUser: 1000
  containers:
    - name: c
      image: busybox:1.28
      command: ["sleep", "3600"]
YAML
for L in privileged baseline restricted; do
  echo "--- $L"
  kubectl -n psa-$L apply --dry-run=server -f /tmp/post-pod.yaml; echo "exit $?"
done

The # ... in the post's fence is where a container list has to go, so the file above is the post's Pod with the smallest container that will make it a valid Pod. Three answers, and the third is the finding: read which field the message names.

  1. Now the pin's own example, into the same three namespaces:
for L in privileged baseline restricted; do
  echo "--- $L"
  kubectl -n psa-$L apply --dry-run=server -f $E/pods/security/security-context.yaml
done

Count the violations in the restricted message. Every one of them is a control the docs' first security example does not satisfy, on a page that page's own reader has no reason to distrust.

  1. Read the four controls that produced those messages, in the order the profile lists them:
for N in 380 396 417 435 459; do
  echo "--- $N"
  sed -n "${N}p" $W/concepts/security/pod-security-standards.md | sed 's/<[^>]*>//g;s/^[ \t]*//'
done
sed -n '406,414p' $W/concepts/security/pod-security-standards.md | sed 's/<[^>]*>//g;s/^[ \t]*//' | grep -v '^$'
sed -n '427,431p' $W/concepts/security/pod-security-standards.md | sed 's/<[^>]*>//g;s/^[ \t]*//' | grep -v '^$'

Five control names, then the Allowed Values of the two that concern the post. One of the two lists has a single entry and does not include undefined; the other includes undefined explicitly. That asymmetry is the whole of step 3's third answer.

  1. Add the field the post never names, and find out what is still missing:
cp /tmp/post-pod.yaml /tmp/nonroot-pod.yaml
sed -i 's/    runAsUser: 1000/    runAsUser: 1000\n    runAsNonRoot: true/' /tmp/nonroot-pod.yaml
kubectl -n psa-restricted apply --dry-run=server -f /tmp/nonroot-pod.yaml
cat >> /tmp/nonroot-pod.yaml <<'YAML'
      securityContext:
        allowPrivilegeEscalation: false
        capabilities:
          drop: ["ALL"]
        seccompProfile:
          type: RuntimeDefault
YAML
kubectl -n psa-restricted apply --dry-run=server -f /tmp/nonroot-pod.yaml; echo "exit $?"

Two more rounds. The first message is shorter than step 4's and still not empty. The second should be silent — and if it is not, read what is left against the list from step 5 before changing anything.

  1. The premise, removed. Look at the node first, because admission and the kubelet answer different questions:
cat /etc/subuid; echo "exit $?"
cp /tmp/post-pod.yaml /tmp/userns-pod.yaml
sed -i 's/^spec:/spec:\n  hostUsers: false/' /tmp/userns-pod.yaml
kubectl -n psa-restricted apply --dry-run=server -f /tmp/userns-pod.yaml; echo "exit $?"
kubectl -n psa-restricted apply -f /tmp/userns-pod.yaml
kubectl -n psa-restricted get pod run-as-uid-1000 -o wide
kubectl -n psa-restricted describe pod run-as-uid-1000 | sed -n '/Events:/,$p'

The Pod is the post's Pod — still no runAsNonRoot — with one field added. Admission is one verdict and the kubelet is another; record both, and do not predict either from the other. If /etc/subuid has no entry for the runtime's user, the second verdict is the one the gate below used to exist for.

  1. The gate, and the gate that is not there:
G=$W/reference/command-line-tools-reference/feature-gates
cat $G/UserNamespacesPodSecurityStandards.md
ls $G | grep -ic 'podsecuritypolicy'; true
grep -c 'PodSecurityPolicy' $W/reference/command-line-tools-reference/kube-apiserver.md; true

One file that explains its own deletion, then two searches that come back empty. The post's global mitigation was never a feature gate, so there is no ladder to read and no removed: true to find; the only trace of it in this directory is the absence.

  1. Ask the running server which admission plugins it actually has:
kubectl get --raw /metrics \
  | grep '^apiserver_admission_controller_admission_duration_seconds_count' \
  | grep -o 'name="[^"]*"' | sort -u

metrics.md:21-26 describes this histogram as STABLE, labelled by name, operation, rejected and type, and exported by kube-apiserver. Find PodSecurity in the list. Then look for the post's plugin, and note that a plugin the binary does not have cannot appear here whatever any page says about it.

  1. The mitigation that needed no translation. Read the pin's list, then pin an image by both tag and digest and ask the cluster what it pulled:

    sed -n '40,43p;52,53p;59,68p' $W/concepts/containers/images.md
kubectl -n psa-privileged run digest --image=busybox:1.28 --restart=Never \
  --command -- sleep 3600
kubectl -n psa-privileged wait --for=condition=Ready pod/digest --timeout=90s
kubectl -n psa-privileged get pod digest -o jsonpath='{.spec.containers[0].image}'; echo
kubectl -n psa-privileged get pod digest \
  -o jsonpath='{.status.containerStatuses[0].imageID}'; echo

    The first command is the six forms and the precedence sentence. The last two are the string you asked for and the digest the node resolved it to. The post's advice is to write the second in place of the first, and the difference between those two outputs is why.

  2. The clause that outlasted the rest of the post:

    sed -n '215,219p' $W/concepts/security/linux-kernel-security-constraints.md
grep -rn 'CVE-2019-5736' $W | wc -l

    One note, one CVE, and a count of one. Read it against the post's :21 and say which half of that sentence the docs kept.

Expect

Step 1: 1, then a single line, psp-to-pod-security-standards.md:143. The seven-line window shows a table row whose first cell is runAsUser, whose second says Mutating & Validating, and whose third gives Baseline: Anything and Restricted: "rule is MustRunAsNonRoot". The post's string is the input to that mapping, and the output is a profile name.

Step 2: the example sets runAsUser: 1000, runAsGroup: 3000, fsGroup: 2000 and supplementalGroups: [4000] at the Pod level and allowPrivilegeEscalation: false on the container. The post's first fence supplies runAsUser; the post's second fence supplies allowPrivilegeEscalation. The field in the example that appears in neither fence is fsGroup, and the field in neither the example nor the fences is runAsNonRoot.

Step 3: psa-privileged and psa-baseline accept the Pod and report it as created; psa-restricted refuses it, and the message names runAsNonRoot — a field the file does not contain — together with seccomp and capabilities. The post's Pod is refused by the profile that its own global fence translates to.

Step 4: privileged accepts, baseline accepts, restricted refuses with three violations — runAsNonRoot, seccompProfile, and capabilities.drop. The docs' canonical example fails the docs' strictest profile, and nothing on either page says so.

Step 5: Privilege Escalation (v1.8+), Running as Non-root, Running as Non-root user (v1.23+), Seccomp (v1.19+), Capabilities (v1.22+). Then Running as Non-root's Allowed Values: one entry, true. Then Running as Non-root user's: any non-zero value and undefined/null. Undefined is allowed for the number and forbidden for the property.

Step 6: the first message drops runAsNonRoot and keeps seccomp and capabilities. The second is accepted with exit 0. Four fields to satisfy a profile that the post's one field was aimed at, and three of them post-date the post.

Step 7: /etc/subuid either exists with a mapping or does not, and its exit code is the honest answer. Admission accepts the Pod despite the missing runAsNonRoot, because hostUsers: false puts both of the post's fields on the not-checked list at user-namespaces.md:266-273. Whether it then runs is the kubelet's answer, and on a node with no subuid range for the runtime it will not: the Pod stays out of Running and the events name the user-namespace configuration, which is the failure mode the deleted gate was there to prevent from being silent.

Step 8: the gate file, with removed: true, one alpha row, and the paragraph explaining why v1.35 does not have it. Then 0 from the directory listing and 0 from the API-server help. The post's global mitigation leaves no trace in the feature-gate directory at all.

Step 9: a sorted list of admission plugin names containing PodSecurity, and no PodSecurityPolicy in it. The list is the server's, not a page's.

Step 10: six image forms, the sentence "Tags can be moved to point to different images, but digests are fixed", and the note that with both a tag and a digest only the digest is used. Then an imageID containing an @sha256: digest, and a spec.containers[0].image that is still busybox:1.28. The node resolved the tag to a digest and recorded both; the post's advice is to write the resolved form yourself so that nobody else can change what the tag resolves to.

Step 11: the note, the CVE link, and the count 1. The docs kept the SELinux half of the post's :21 and dropped the "is not running as UID 0" half into the profile machinery of steps 3 to 6.

Read on

  • The 11-ways exercise — for PodSecurityPolicy's removal and the mapping table read in full. It closes with an open question about why the deprecation guide writes about the PodSecurityPolicy admission controller in the future tense. Step 9's output is from the server rather than a page: does it settle that question, or only narrow it?
  • The security best practices exercise — for the PodSecurity and UserNamespacesSupport ladders, and for a 2016 post whose advice was the same as this one's. Given that its UserNamespacesSupport ladder ends locked: true, what is the earliest release in which step 7's hostUsers: false could not be switched off by a cluster operator?
  • The dry-run exercise — for the flag that every apply in steps 3, 4, 6 and 7 uses. Those steps ask admission a question about a Pod that is never created; read what that exercise measures about defaulting, and say whether a --dry-run=server verdict from the restricted namespace would still hold if a mutating webhook were installed.
  • The containerd exercise — for the layer this post's :13 describes. The post names Docker, Containerd and CRI-O as the things that sit on runc; read what the pin documents in container-runtimes.md and say which of the three the kubelet can still talk to without a shim.

Teardown

kubectl delete ns psa-privileged psa-baseline psa-restricted
rm -f /tmp/post-pod.yaml /tmp/nonroot-pod.yaml /tmp/userns-pod.yaml

Leave the guest up; the next exercise in this year reuses it. Everything created lived in the three namespaces, so the deletion covers the one Pod from step 7 and the one from step 10. Nothing in this exercise changed the node: step 7 reads /etc/subuid and does not write it, and if you want a node that can actually run a user-namespaced Pod, that is a node change and belongs in a topology, not in a walk.