Labs — Phase 7, networking, the L3/L4 datapath¶
Thirty-four exercises in the order they are meant to run, for the only phase whose two
build artifacts have no fast loop: the strand lists both the CNI plugin and the eBPF
program as on-node-only, so there is no
stage 1 to edit-and-rerun against a live API from outside. What replaces it is named twice
and built twice — a cnitool harness in exercise 7 and a
veth in a namespace in exercise 28 — and
both are legitimate only because forge runs the nodes' kernel, which is why
exercise 2 checks that before anything is
compiled.
Each file states one claim to test or one artifact to produce, links its topology rather than restating a footprint, and ends with a teardown that does two things: deletes what that exercise created, then says whether the topology stays or goes.
The framing — why each module exists, what to read and the question to answer from it —
stays in phases/07-networking.md. These files hold only
what you type and what you should see.
| # | Exercise | Why it exists |
|---|---|---|
| 1 | The four communication paths, three of them already built in P0 | The model's four paths mapped onto P0's veth exercises, so nothing here re-teaches a veth pair. |
| 2 | forge and the nodes must share a kernel |
The precondition made an exercise: the repair is a reprovision, so week four is the wrong place to find out. |
| 3 | What the runtime hands a plugin | A real ADD captured by shimming the installed binary — the spec met as evidence rather than as prose. |
| 4 | Ready, Serving and Terminating for five pod states |
Fifteen cells predicted from the comments, scored against a pod with a 90-second preStop. |
| 5 | Shrink the slice limit to five | KEP-752's write amplification made visible on a two-node cluster by one flag. |
| 6 | A Service pointed outside the cluster, twice | The mirroring controller, the three managed-by values, and what KEP-4974 is removing. |
| 7 | The fast loop for an artifact with no stage 1 | Builds the harness, proves it with somebody else's plugin, and names the three things it cannot test. |
| 8 | Build artifact 1: ADD, DEL, and cnitool as judge |
The plugin, with the gate made to fail twice on purpose before it is allowed to pass. |
| 9 | An allocator, and the leak DEL cannot free |
Frees by container id rather than by address, and kill -9 mid-ADD proves why GC exists. |
| 10 | Stage 2: real pods, addressed by your allocator | Discovery order, three failure modes, and the worker-only install that keeps the control plane known-good. |
| 11 | Two nodes, two pod CIDRs, no route | The row P0 could not build; each command maps to host-gw, overlay, BGP or eBPF, and none of it survives a reboot. |
| 12 | A malformed result at the CRI seam | The error surfaces in containerd first and the kubelet second — P6's named seam, paid. |
| 13 | A second plugin in the chain | prevResult is byte-for-byte yours, DEL runs backwards, and a chained failure leaks an address. |
| 14 | kube-proxy's model in six small files | The unit of work is a service port; read the model before any backend, or every backend looks arbitrary. |
| 15 | reconciler.go, cited |
P6's first owed citation, verified against the running commit because staging paths move. |
| 16 | One ClusterIP followed to its KUBE-SEP |
Four chains, the probability arithmetic worked, and two hashes with two different scopes. |
| 17 | The same Service as a verdict map | Switch the backend and diff the dumps: two structural reasons, written as claims rather than as faster. |
| 18 | The tracker between two syncs | P6's second owed citation, with minSyncPeriod widened so the pending counter can be caught above zero. |
| 19 | 7.C3 — one wrong DNAT | A third of requests failing with three Ready endpoints and no event anywhere; timed against the resync. |
| 20 | 7.C1 — one partition, two primitives | Checks the strand's one-primitive-per-fault claim instead of repeating it. Chaos Mesh arrives here. |
| 21 | An address that exists only as an ARP reply | externalTrafficPolicy measured both ways, with the bastion NAT recorded as a limit rather than worked around. |
| 22 | Three backends, one table | IPVS read and never run; the row that earns it is weight-zero draining, which the other two cannot do. |
| 23 | A policy nobody enforces | A perfect deny-all changes nothing, and nothing in the cluster reports that — the negative result is the finding. |
| 24 | Ten policy questions, sealed | Predictions committed to git before an enforcer exists, so exercise 32 scores a claim rather than a transcript. |
| 25 | One curl by name, counted in queries |
The ndots trap measured, and DNS shown to be a client of module 7.3's datapath. |
| 26 | 7.C2 — a resolver that lies | The checklist run against a fault four of its five checks cannot see; the second action succeeds and is wrong. |
| 27 | Three rules NetworkPolicy cannot express | KEP-2091 read, three manifests written, and banked to P10 rather than installed here. |
| 28 | Build artifact 2: five verbs, each proven | Load, attach, read, detach — plus the CO-RE program the kernel gate exists to protect. |
| 29 | The borrowed drill: a type-layout mismatch | Three endings — loud at load, loud at attach, silent at read — and only the third is the dangerous one. |
| 30 | A drop decided by a map entry | Your program on the veth your plugin made, at a hook that sees the ClusterIP and not the backend. |
| 31 | The same DNAT, a third time | Chains, verdict map, cilium bpf lb list — one Service, three representations, from your own saved dumps. |
| 32 | Ten predictions opened and scored | Every miss traced to a citation, and a column naming which component actually answered each row. |
| 33 | 7.C4 — three policies that read correct | Blind-picked and timed; the over-allow is the slow one, because nothing is broken and nothing shows up. |
| 34 | The capstone | A pod on your own plugin, a drop with a map entry behind it, and P6's three citations re-verified live. |
Which cluster is running when¶
Exercise 1 needs no cluster — it runs on forge
with ip netns and nothing else, which defers the phase's first provision by a day.
pair comes up at
exercise 2 and is held to the end, destroyed at
the capstone. It is
provisioned for the kernel check, before there is any Kubernetes work to do on it, and
that ordering is the point of exercise 2.
forge never changes size, and P7 is the phase that proves the standing 1536MB is
enough. Nothing here links a control-plane binary — the biggest compile is clang on one
small C file, well under the 564 MiB the strand measured
— so the P5 resize has no analogue here. What this
phase needs from forge is the kernel, not the RAM.
| Exercises | Topology | forge |
Inside the guests | Total | Margin |
|---|---|---|---|---|---|
| 1 | none | 1536MB | — | 1.5GB | 8.0GB |
| 2–19 | pair 5.0GB |
1536MB | — | 6.5GB | 3.0GB |
| 20 | pair 5.0GB |
1536MB | Chaos Mesh 582Mi | 6.5GB | 3.0GB |
| 21–30 | pair 5.0GB |
1536MB | + MetalLB 250–300Mi | 6.5GB | 3.0GB |
| 31–34 | pair 5.0GB |
1536MB | Chaos Mesh out, Cilium in | 6.5GB | 3.0GB |
against the 9.5GB ceiling. The guest total never moves, and neither does the margin.
The one real pressure in this phase is tenancy, not totals, and it is flagged here rather
than softened. Three DaemonSet-shaped tenants wanted to be resident at once inside
pair's 5.0GB: Chaos Mesh at 582Mi, MetalLB at 250–300Mi,
and Cilium — against the phase's own note that
Cilium is not co-resident with the heavy tooling on this host. The smallest change is
ordering, and that is the change taken: Chaos Mesh goes in at
exercise 20, both drills that need it are done by
exercise 26, and it is uninstalled at the start of
exercise 31 before Cilium is installed. No guest
is resized and no drill is dropped — the two remaining drills, 7.C3 and 7.C4, are by-hand
and need nothing installed. If the worker is still under pressure after Cilium,
exercise 31 names MetalLB as the next removal,
in advance, so that the decision is not made under time pressure.
Two CNIs, one cluster, on purpose. From exercise 10
to exercise 30 your plugin owns .131 and the stock
one owns .130. That is what makes a cross-node failure a comparison rather than an outage,
keeps kubectl working while pod networking on the worker is deliberately broken, and
guarantees exercise 23's negative result instead of hoping
for it. It ends at exercise 31, which also says
what the arrangement implies for a real CNI migration.
Exceptions¶
- Exercise 1 needs no cluster at all, and exercises 7, 8, 9 and 13 need none either — they are
forge, a netns and thecnitoolharness, withpairup and idle beside them. - Exercises 28 and 29 are also
forge-only.pairstays up and idle rather than being destroyed, because exercise 30 needs a real pod's veth and a reprovision costs more than the idle RAM. - Exercises 22, 24 and 27 create nothing. Two are reading with a table as the artifact; the third is a prediction committed to git and scored at exercise 32.
- Exercise 2 is a precondition exercise, not a check. It provisions the topology solely to compare two kernels, and its two named fixes are both reprovisions — which is why it is second rather than in module 7.5.
- Exercise 29 substitutes for the strand's drill rather than running it exactly: it produces three of the four failure modes on one kernel and says, in the exercise, which one only a genuine second kernel and a reboot can produce.
- Drills 7.C3 and 7.C4 use no chaos tooling — the phase says why: reading the rule that broke is the exercise, and a tool that injects the break also names it.
- Exercise 21 records a limit instead of engineering around it. A request from the Mac arrives NAT'd by the bastion, so the "preserved" client address is the bastion's; the deliverable is the pair of results, and the note says so.