Phase 11 — Synthesis¶

2–3 weeks. No new subsystem, no new tool, no new area. This phase reads across everything already read: it walks one request — kubectl run nginx — from the terminal to a running container, through every area entered in P2–P10, and it cites each seam to source. Then it uses the whole picture for the two operations that only make sense once you have it: a live minor-version upgrade with nothing dropped, and scaling under real load. This is the loop P0 opened in week one, closed. The range is planning information. The gate at the bottom decides when the phase is finished.


Prerequisites	All of P2–P10 — this is the only phase that is prerequisite on the whole descent. The trace crosses Area 2 (P3), Area 1 (P2), Area 3 (P5), Area 7 (P6) and Area 5 (P7) in a single request, and lands on the P0 syscalls. An unentered area is an unreadable seam.
Unlocks	P12 — the climb back up. Everything before this phase asks how does Kubernetes work; having answered it end-to-end, P12 asks what you build on top so other people don't have to. The synthesis is the last thing that must be true before abstraction is honest.
Source	The whole corpus, re-read along one path rather than by subsystem — the three cross-area traces, of which this phase attempts trace #1. No new file is opened that a prior phase didn't already read; what is new is reading them in the order a single pod-create visits them, which is the order no single phase could.
Build track	None — build mechanics lists no P11 artifact. Everything this phase runs was built or installed earlier; synthesis produces a trace and two incident write-ups, not a binary.
Ecosystem	None — this phase installs nothing new. The ecosystem distribution gives P11 no big rock by design: it is synthesis, not another tool. The distribution comparison that does appear (kubeadm vs k0s upgrade paths) is a property of the cluster you already run, folded into the upgrade capstone, not a new install.
Lab	`pair` for the trace — two nodes are enough to watch scheduling pick one. `workhorse` (3 nodes) for the upgrade-and-scale beat, because a zero-downtime upgrade means draining one node at a time while the other two keep serving, which two nodes cannot demonstrate honestly. Switching topology mid-phase is itself the point: the trace needs a pod, the upgrade needs a quorum-plus-spare.
Labs	`labs/11/` — 12 exercises, in order. The unit is the seam, not the hop; the trace half runs on `pair`, then one switch to `workhorse` for the upgrade-and-scale capstones, and the index says which two exercises need no cluster.
Strands	talks · chaos · archaeology

1. Objectives¶

Every one is falsifiable — a cited trace, a timed production, or a claim a hostile reader could check against source or a running cluster. understand and know appear nowhere; a trace you cannot cite is not a trace.

By the end you can:

Walk kubectl run nginx from the terminal to a running container, naming every area boundary it crosses and citing a file:line (or commit sha) at each — the six seams, not the endpoints.
Name the exact handoff at each seam — how the object leaves the client and enters the apiserver, leaves the apiserver and enters etcd, leaves etcd and reaches the scheduler, leaves the scheduler and reaches the kubelet, leaves the kubelet and becomes a container — as a mechanism, not a diagram label.
Run a minor-version cluster upgrade with zero workload downtime, proven by a load generator that records 0 dropped requests / 0 5xx across the whole upgrade window, not by "it looked fine."
Scale a workload under real load and name, from observation, which component acted at each step (metrics → HPA → scheduler → kubelet) and the measured latency of each hop.
Re-watch the DNS postmortem and, at each layer it descends, name the phase that taught that layer — the talk's four-week descent is the curriculum's spine read backwards.
Produce a one-page incident write-up in the format P1/P6/P8 escalated: symptom, objects inspected in order, mechanism cited to source, fix.

2. Modules¶

The corpus offers no new reading here — every file below was opened in an earlier phase. The rule is unchanged and now at its strictest: every hop resolves to a file:line a hostile reader could open, and the deliverable is those citations. The archaeology drill standard applies with no slack: a seam you can describe but not cite is a seam you did not actually cross. Live-verify every path before trusting it — the tree moves (stale paths).

Module 11.1 — The trace on paper, before the cluster (~3 days)¶

You assemble the map from eight phases of notes before instrumenting anything, so the live trace confirms a prediction rather than discovers a path.

Question to answer from the source: the trace crosses five areas. At each of the four internal seams, which single function hands the object to the next area — the last frame in area N and the first frame in area N+1? Name both.

Labs — The map drawn from notes, before a single command

Module 11.2 — Seam A: client → apiserver → etcd (~4 days)¶

Areas 2 (P3) and 1 (P2), read as one continuous handoff.

Question to answer from the source: handlers/create.go runs admission before the object reaches registry/generic/registry/store.go. Cite the line where admission is invoked, and the line in storage/etcd3/store.go where the object becomes an etcd Txn — and state what guarantees the object is validated before it is durable, not after.

Labs — Seam A: kubectl to an etcd Txn, cited · 11.C1 — a create wedged at admission

Module 11.3 — Seam B: watch cache → scheduler → Binding (~3 days)¶

Area 3 (P5) — how the persisted-but-unscheduled pod becomes a scheduled one.

Question to answer from the source: the scheduler does not mutate the pod's node field directly — it POSTs a Binding. Cite the line in schedule_one.go that issues the bind, and explain why binding is a separate write and not an in-place update of the pod the scheduler already holds in cache.

Labs — Seam B: spec.nodeName empty to set, and the Binding

Module 11.4 — Seam C: kubelet → CRI → CNI → the syscalls (~4 days)¶

Areas 7 (P6) and 5 (P7) — the pod becomes a process, and the loop closes on P0.

Question to answer from the source: computePodActions decides what to do; cite the line where it decides a container must be created, and follow it to the CRI call. Then name the CNI ADD result — the veth/netns — and point at the P0 syscall (clone/setns/unshare) it corresponds to. The whole trace ends on a syscall you once made yourself.

The three sub-paths (11.2/11.3/11.4) joined are the capstone.

Labs — Seam C: kubelet → CRI → CNI → the syscalls · 11.C4 — kill one component, read the frame it owns

Module 11.5 — The postmortems, re-read with the internals in hand (~3 days)¶

The debugging corpus watched differently now. The DNS talk was planted in P1 for its method; you re-watch it here for its mechanism, because you have now read every layer it descends into.

Read/watch — the debugging strand, the DNS four-week-incident talk first. Map its descent — apparent DNS failure → conntrack overflow → martian-packet drops → kernel source → a gRPC reconnect bug — onto the phases: conntrack is P7, the kernel drop is P0, the reconnect is a client the apiserver work in P3 made legible.

Question to answer (from the talk against your own trace): the talk's resolution was three lines of code, found four layers below the symptom. For each layer it descended, name the phase that taught you to read it — and name the one layer, if any, this curriculum still leaves you unable to read.

Labs — The DNS talk, re-read for mechanism · 11.C3 — the conntrack table filling under a rollout

3. Chaos drills¶

Anchored in chaos.md#manual-drills. These are the drills the strand marks permanently manual — upgrade, botched rollout, drain — because the failure modes are procedural, not injected, and each needs the whole picture the earlier phases could not yet supply. They are the rehearsals the §5 upgrade-and-scale capstone assembles.

#	Drill	What you must produce afterwards
11.C1	Wedge a create at admission	The create failing at the admission line cited in 11.2, diagnosed from apiserver logs alone
11.C2	Botched rollout during the upgrade window	The rollout wedged, recognised from the controller's behaviour, rolled back with `kubectl rollout undo`
11.C3	Rolling update under conntrack watch	The conntrack table filling — the DNS-talk mechanism, observed on your cluster
11.C4	Kill one trace component mid-flight	The pod stuck at the exact frame the missing component owns — `Pending`, `ContainerCreating` — cited

Every drill here is by hand and stays that way (manual-drills): the skill is recognising your own procedure's failure from the system's behaviour, and injecting the fault externally removes exactly that skill. The drain drill lives in the capstone because it is the upgrade.

4. Talks¶

Full entries under Cluster debugging and postmortems — the strand weights these most generously, and this is the phase they were saved for.

★ Logs Told Us It Was DNS… It Wasn't DNS (Bernaille & Andrews, EU 2022) — the best debugging talk in the corpus, planted in P1 for method, re-watched here for mechanism. A four-week incident descended four layers to three lines of code; you can now read every layer. The module 11.5 spine.
★ How the OOM-Killer Deleted My Namespace (Bernaille, NA 2020) — a node-level event cascading into a control-plane outcome. The lesson is coupling and blast radius across components that look independent — which, having traced them as one path, you can now see are not.
★ Kubernetes Failure Stories and How to Crash Your Clusters (Jacobs, EU 2019) — the breadth-first survey of how clusters actually fail, from the curator of k8s.af. The pattern-catalogue against which your own two incidents are single data points.
★ 10 Ways to Shoot Yourself in the Foot with Kubernetes (Bernaille & Boll, EU 2019) — ten production failures, each traced to a specific mechanism (conntrack, limits, DNS, controller assumptions). Effectively a failure-mode catalogue with root causes attached — read it as a checklist of seams to distrust.

5. Capstone¶

Two capstones. The first proves you can read the whole machine; the second proves you can operate it without stopping it.

Capstone 1 — corpus trace #1, `kubectl run nginx` to a running container¶

Join the three sub-paths from modules 11.2–11.4 into one cited trace, terminal to container:

kubectl → request filters → handlers/create.go → admission chain → registry/generic/registry/store.go → storage/etcd3/store.go → etcd Txn → watch cache → scheduler informer → schedule_one.go → Binding → kubelet config/apiserver.go → pod_workers.go → computePodActions → CRI → CNI → the P0 namespaces and cgroups.

Cite file:line a hostile reader could check at every seam — this is trace #1's spec, and the deliverable is the citations, not the diagram. A reader clones k/k at your stated commit and opens every line. "The apiserver validates it" fails the gate; create.go:NNN (at sha abc123) passes. Attempted only now, after every area is behind you — and named in P0 as the target from week one, so this closes the loop the curriculum opened.

Lab — Capstone 1: the three sub-paths joined, terminal to container

Capstone 2 — a minor-version upgrade with nothing dropped, then scale under load¶

On workhorse, with a load generator running against a Service the whole time:

Upgrade one minor version — kubeadm upgrade plan/apply (and note where k0s differs), draining one node at a time so the other two keep serving, watching kubectl drain fight PDBs (manual-drills #4/#7). Prove 0 dropped requests / 0 5xx across the window from the load generator's own log — a re-readable artifact, the "file:line" of an operation with no source line.
Scale under load — drive load past the HPA threshold and name each component as it acts: metrics → HPA decision → new pod persisted (Seam A) → scheduled (Seam B) → running (Seam C) → in the Service's EndpointSlice (P7). Record the measured latency of each hop; when a node can't satisfy the request, name the Pending frame.

The two capstones are one claim from two sides: you can read the machine top to bottom, and you can change it underneath a running load without it noticing.

Lab — The upgrade that drops nothing · Scale under load, narrated by component

6. Checklist¶

Concrete, demonstrable, grouped by evidence type. No item says understand or know.

The trace (cited to source): - [ ] Seam A — kubectl → filters → create.go → admission → store.go → etcd3/store.go → Txn with a line at each arrow + the raw etcd key (11.2). - [ ] Seam B — watch cache → informer → schedule_one.go → Binding with the bind line + spec.nodeName before/after (11.3). - [ ] Seam C — config/apiserver.go → pod_workers.go → computePodActions → CRI → CNI with the P0 syscall it ends on (11.4). - [ ] The three sub-paths joined into one trace, every seam cited at a stated commit sha (Capstone 1).

Operations (proven by artifact): - [ ] A minor-version upgrade with the load generator showing 0 5xx across the window (Capstone 2). - [ ] A scale-under-load run with per-hop component and latency named, including the Pending frame when a node is full (Capstone 2).

Written artifacts (each is a module's Write-down): - [ ] The paper trace-map, corrected in red against the live run (11.1). - [ ] The DNS talk's descent as a layer-to-phase table + the transferable method one-liner (11.5). - [ ] A one-page incident write-up from 11.C2, in the escalated P1/P6/P8 format.

Falsifiable claims — write, then verify: - [ ] Why admission runs before the object is durable, not after (11.2). - [ ] Why the scheduler POSTs a Binding instead of updating the pod in place (11.3). - [ ] The one layer, if any, this curriculum still leaves you unable to read (11.5).

7. Gate¶

You may advance to P12 when:

The full trace is cited, terminal to container — every one of the six seams resolves to a file:line at a stated commit that a hostile reader opens and confirms. If any seam is a description rather than a citation, the area behind it did not land — go back to that phase, not forward. This is the gate the whole curriculum was built to reach.
The upgrade dropped nothing — the load generator's own log shows 0 5xx across a real minor-version upgrade on workhorse, and you can name which node was draining at each moment. "It seemed fine" is not the artifact; the log is.
You scaled under load and narrated it by component — metrics → HPA → Seam A → B → C → EndpointSlice, with the frame that stalls when a node is full. If scaling is still "Kubernetes handled it," the trace did not transfer to operations.

The descent is over. P12 is the inverse of everything before it — eleven phases removed abstraction; the last one builds it back, deliberately, for people who will never read the trace you just cited. You can only hide a machine well once you have seen all of it, which is why this gate comes first.