Phase 7 — Networking, L3/L4 datapath¶
4–5 weeks. The area where the specs are more readable than the code — the CNI spec is clear prose,
proxier.gois a 61 KB rule-generation machine — so you lead with specs and API types and enter the dataplane code only throughsyncProxyRules. The build track resumes here with two artifacts loaded into a real kernel, and this phase repays a debt: P6's capstone trace could only watch the EndpointSlice/kube-proxy far end; here you read it in source. The range is planning information. The gate at the bottom decides when the phase is finished.
| Prerequisites | P6 — its capstone trace stopped at the node's edge (nft list ruleset-observable but uncited); this phase opens that far end in source and closes the seam. P0 — pod networking is ip netns + veth + bridges + routes; the CNI plugin you build is those primitives automated, and the phase is barely legible without them. |
| Unlocks | P8 — the last descent phase; storage is the one major subsystem left below the abstraction line. P9 — the service mesh is an L7 layer on top of the L3/L4 datapath built here, and Cilium (entered here) is where mesh and CNI start to merge. |
| Source area | Area 5 — Networking, entry point discovery/v1/types.go. kube-proxy in iptables and nftables, IPVS comparative-only (KEP-5495 deprecates it; nftables is GA and slated default). |
| Language | Go — two build artifacts (CNI plugin, eBPF program), both on-node-only: unlike the webhooks and CSI driver, neither has an out-of-cluster stage 1, because you cannot wire a netns or attach a tc hook without a node. The eBPF build carries the kernel-lockstep gate. |
| Lab | pair — Cilium is a DaemonSet, so two nodes beat three. forge never changes size this phase: what the two artifacts need from it is the nodes' kernel, not RAM. |
| Labs | labs/07/ — 34 exercises, in order. One topology for 33 of them, and three tenants that cannot all be resident at once — the index gives the order they arrive and leave in. |
| Strands | source reading · build · talks · eBPF talks |
1. Objectives¶
Every one is falsifiable — an artifact, a timed production, or a claim a hostile reader could check against source or a running cluster. understand and know appear nowhere.
By the end you can:
- Wire a pod's network by hand —
ip netns, veth pair, bridge, routes — and then say exactly what a CNI plugin automates, citing the CNISPEC.mdoperations. - Read
discovery/v1/types.goand explain what EndpointSlice replaced and whyEndpointsdidn't scale (O(n) watch fan-out on every pod churn), citing KEP-752 — the far end of P6's trace, now cited in source. - Build a CNI plugin (ADD/DEL, JSON on stdin, wires a netns) that gives pods a
ping-able network. Build artifact 1. - Compare
syncProxyRulesin iptables and nftablesproxier.goand name the structural difference (chain-per-service vs verdict maps) — the comparison is the lesson. - Trace a Service packet from ClusterIP to the kube-proxy rule/map that DNATs it to an endpoint, shown in a live
iptables-save/nft list rulesetdump — repaying P6's far half in source (endpointschangetracker→syncProxyRules). - Compile, load, attach, read a map from, and detach an eBPF program via
cilium/ebpf, with theforge/node kernel lockstep verified. Build artifact 2. - Prove a NetworkPolicy at the datapath — the specific rule or map entry that drops the packet, not just that the connection failed (k8s ships no enforcer; the CNI does).
- Diagnose a DNS failure by method — CoreDNS,
resolv.conf, the Service behind it — the It Wasn't DNS procedure.
2. Modules¶
Reading is Area 5, and its rule is the corpus's clearest: specs and API types before the proxier.go machines, the small backend-agnostic proxy files before any big backend. Each cited item carries a question to answer — no bare links.
Module 7.1 — Networking from first principles, and EndpointSlice (~1 week)¶
The Linux primitives first, by hand — then the data structure the whole service dataplane consumes.
Read
| Item | Answer from it |
|---|---|
CNI SPEC.md (item 1) + CONVENTIONS.md (item 2) |
The spec: ADD/DEL/CHECK/GC/STATUS, the JSON config, chaining, and how the runtime invokes plugins as executables with env vars. No Kubernetes needed. What does the runtime hand a plugin on stdin? |
network/networking.md design doc (item 3) |
The IP-per-pod model, no NAT between pods, the four communication paths. What are the axioms every CNI must satisfy? |
discovery/v1/types.go (item 4, ⭐) + KEP-752 (item 5) + KEP-4974 (item 6) |
The entry point: Endpoint, EndpointConditions{Ready,Serving,Terminating}, the addressType split. Why did one Endpoints object per Service melt the apiserver, and what does slicing fix? This is the data structure P6's trace passed through. |
Labs — 1 the four paths, three of them P0's · 2 the kernel both sides must share · 3 what the runtime hands a plugin · 4 Ready/Serving/Terminating predicted · 5 the packing heuristic made visible · 6 the API that is being deleted.
Module 7.2 — Build artifact 1: the CNI plugin (~1 week)¶
CNI demystified: a spec for an executable that receives JSON on stdin and wires a netns. Module 7.1 done by hand; now automate it.
On-node-only — there is no out-of-cluster stage 1, because you cannot wire a netns without a node, which is why the artifact table lists none for it. What replaces the fast loop the other nine artifacts get is the first exercise of the module, not a footnote.
Labs — 7 what replaces stage 1 · 8 artifact 1, judged by cnitool · 9 an IPAM that does not leak · 10 stage 2, real pods on the worker · 11 two pod CIDRs, no route · 12 a malformed result at the CRI seam · 13 a second plugin in the chain.
Module 7.3 — The Service dataplane: kube-proxy (~1 week)¶
Where P6's trace debt is repaid in source. Read the small model files first — the routinely-skipped correct on-ramp.
Read
| Item | Answer from it |
|---|---|
pkg/proxy/{serviceport.go, endpoint.go, servicechangetracker.go, endpointschangetracker.go, topology.go} (item 10, first) |
kube-proxy's backend-agnostic model. How are changes accumulated between syncs, and how does topology.go filter endpoints? endpointschangetracker is the P6-trace hop you could only watch. |
endpointslice/reconciler.go (item 8) + endpointslice_controller.go (item 9) |
The pod→endpoint mapping and churn-minimising slice packing — the most readable substantial networking code in the tree. This is the reconciler P6's trace named as owed. |
iptables/proxier.go syncProxyRules only (item 15) + nftables/proxier.go syncProxyRules (item 16) + KEP-3866/5343 |
KUBE-SERVICES/KUBE-SVC-*/KUBE-SEP-* chains vs nftables verdict maps. Read each beside a live dump. What is the structural reason nftables scales where iptables' chain-per-service does not? |
KEP-265 + KEP-5495 (items 13–14) + external-lb-source-ip-preservation.md (item 30) |
IPVS as comparative/historical only (it's being deprecated), and why externalTrafficPolicy: Local exists and what it costs on factory's NAT'd bridge. |
Labs — 14 the model files before the machine · 15 reconciler.go, cited · 16 a ClusterIP followed to its KUBE-SEP · 17 the same Service as a verdict map · 18 the tracker between two syncs · 19 drill 7.C3 · 20 drill 7.C1 · 21 an address claimed by ARP · 22 three backends, one table.
Module 7.4 — NetworkPolicy and DNS (~4 days)¶
The semantics live in the type comments; enforcement lives in the CNI, not in Kubernetes.
Read
| Item | Answer from it |
|---|---|
networking/v1/types.go NetworkPolicy types (item 20) + network-policy.md (item 21) |
The normative semantics are the comments: the default-allow→default-deny flip on selection, ingress/egress rule union, and podSelector+namespaceSelector AND-vs-OR — the classic mistake. Where does the doc say k8s ships no enforcer? |
| KEP-2091 AdminNetworkPolicy (item 22) | Cluster-scoped, explicitly-ordered Allow/Deny/Pass — fixing NetworkPolicy's inability to express a real deny. Strong CKS material — banked toward P10. |
| CoreDNS + the It Wasn't DNS method (talk) | How a pod resolves a Service name: resolv.conf, the ndots trap, the CoreDNS Service behind it. What are the first three things the diagnostic method checks? |
Labs — 23 a policy nobody enforces · 24 ten questions sealed before an enforcer exists · 25 a name resolved in five hops · 26 drill 7.C2 · 27 AdminNetworkPolicy, read and banked.
Module 7.5 — eBPF, and build artifact 2 (~1 week)¶
The datapath's future, loaded into a real kernel — so Cilium's "it uses eBPF" becomes personal in module 7.6.
Read
| Item | Answer from it |
|---|---|
cilium/.../bpf/architecture.rst + progtypes.rst (item 29) |
The eBPF primer: program types, maps, the verifier, tail calls, JIT. Readable without Cilium context. What does the verifier reject, and why? |
cilium/.../network/ebpf/lifeofapacket.rst (item 28) |
"Life of a Packet": which hooks (XDP, tc ingress/egress, socket) a packet traverses and where policy is enforced. iptables.rst contrasts explicitly with kube-proxy. |
Gate — kernel lockstep: CO-RE resolves against /sys/kernel/btf/vmlinux, so forge's kernel must match the node's. uname -r on forge and the target node must be equal before the program loads — a mismatch is a stop-and-fix, not a warning, because the failure is either a verifier rejection or a silently wrong field offset.
Labs — 28 artifact 2, five verbs each proven · 29 the mismatched-BTF drill · 30 a drop decided by a map entry.
Module 7.6 — Cilium (~4 days)¶
Entered after the eBPF artifact exists — detailed as ecosystem in §5, this is its hands-on. It is also the module that changes the cluster's tenancy: Cilium is not co-resident with the heavy tooling on this host, so it arrives only once the drills that need Chaos Mesh are done and Chaos Mesh has come out. Ordering, not a bigger guest — the lab index has the sequence.
Labs — 31 kube-proxy replaced by map lookups · 32 the prediction scored at the datapath · 33 drill 7.C4.
3. Chaos drills¶
Chaos Mesh is now installed (since P6); network faults are exactly where its one-Linux-primitive-per-fault design pays off — NetworkChaos is netem on a qdisc, DNSChaos is CoreDNS running the k8s_dns_chaos plugin. The two by-hand drills stay by-hand because reading the broken rule is the lesson.
| # | Drill | What you must produce afterwards |
|---|---|---|
| 7.C1 | Network partition | Which of the four communication paths broke, and the tc -s qdisc show evidence |
| 7.C2 | DNS failure | Every name-based connection failing while IPs still work — the It Wasn't DNS signature |
| 7.C3 | Corrupt kube-proxy rules | The half-broken Service mapped to the specific missing KUBE-SEP/verdict-map entry |
| 7.C4 | A NetworkPolicy that lies | Why the policy reads correct but over-/under-allows, proven at the datapath |
7.C3 and 7.C4 are by-hand because the exercise is reading the rule that broke.
4. Talks¶
Full entries under Networking and eBPF.
- ★ Container Networking From Scratch (Jacobs, NA 2018) — the P0 networking talk: builds it live from
ip netns, veth, bridges and routes, no Kubernetes. Everything in module 7.1 becomes derivable afterwards. Reproducible on one Debian VM. - ★ Kubernetes Networking: How to Write a CNI Plugin From Scratch (Yanay, EU 2019) — CNI as a spec for an executable that receives JSON and wires a netns — module 7.2 in talk form.
- ★ Kubernetes Networking Intro and Deep-Dive (Du & Hockin, EU 2020, 80 min) — the canonical Service/EndpointSlice data model and kube-proxy modes from the people who designed them. The 80 minutes replace three shorter talks; watch against module 7.3.
- ★ eBPF and Kubernetes: Little Helper Minions (Borkmann, EU 2020) — the reference eBPF talk from a kernel maintainer: verifier, maps, program types, tc/XDP hooks — why eBPF is fast, not just that it is. Watch before module 7.5.
- ★ Logs Told Us It Was DNS… It Wasn't DNS (Bernaille & Andrews, EU 2022) — the best diagnostic-method talk in the corpus, and DNS is where this phase's failures actually live. The procedure behind drill 7.C2.
5. Ecosystem¶
Cilium — entered after the eBPF artifact exists, so "Cilium uses eBPF" is a statement about something you have personally loaded into a kernel, not a slogan.
- Hands-on: module 7.6 — install it in kube-proxy-replacement mode and read its Life of a Packet against your own program.
- Internals note (required): k8s ships no NetworkPolicy enforcer — Cilium is one, in eBPF. It replaces kube-proxy's iptables/nftables
syncProxyRuleswith eBPF map lookups: the ClusterIP→endpoint DNAT you traced in module 7.3 becomes a map entry, and policy enforcement is a program at the tc hook rather than a chain. This is the same job you read in source, done at a different layer — which is the only way to judge the claim that it's faster. - Maturity: CNCF graduated (2023) — the first CNI to graduate, and the project that drove eBPF into mainstream Kubernetes networking. DaemonSet-shaped and not co-resident with Istio/Falco/Prometheus on this host (#8); its L7 and mesh capabilities are where P9 picks up.
6. Capstone¶
Your CNI plugin giving pods a ping-able network, plus a NetworkPolicy enforcement proof read at the iptables/eBPF layer — the rule or map entry that drops the packet, not just that the connection failed.
Two parts, plus the debt repaid:
- The CNI plugin (module 7.2): real pods with connectivity from a plugin you wrote —
ADD/DEL, veth, IPAM, routes. - The enforcement proof (module 7.4/7.6): a NetworkPolicy whose drop you can point at — the
KUBE-SEPabsence, the iptablesDROP, or the Cilium policy-map entry /cilium monitorevent. "The connection failed" is not a proof; the packet's fate in the datapath is. - P6's trace debt repaid (module 7.3): the far half of corpus trace #2 —
EndpointSlice reconciler → endpointschangetracker → syncProxyRules— now cited in source, closing the seam P6 could only observe.
Cite file:line a hostile reader could check — at minimum: the syncProxyRules DNAT site (name the backend); the reconciler.go pod→endpoint mapping; and the datapath location of your policy drop. Every path verified live per P2's archaeology standard — the proxier.go files are 60+ KB, so a copied line number rots fast.
Lab — exercise 34, which is also where the topology goes.
7. Checklist¶
Concrete, demonstrable, grouped by evidence type. No item says understand or know.
Live against a running cluster:
- [ ] Two ip netns ping each other with hand-wired veth/bridge/routes (7.1).
- [ ] Real pods get a ping-able network from your CNI plugin (7.2).
- [ ] A Service packet traced to its DNAT rule in both iptables and nftables dumps (7.3).
- [ ] An eBPF program loaded, attached, its map read, and detached — kernel lockstep verified (7.5).
- [ ] Cilium installed; a policy drop shown at the eBPF layer (7.6).
Build artifacts:
- [ ] Artifact 1: the CNI plugin (ADD/DEL, on-node).
- [ ] Artifact 2: the cilium/ebpf program (compile/load/attach/read-map/detach).
Written artifacts (each is a module's Write-down):
- [ ] The by-hand netns wiring, mapped to what the CNI automates (7.1).
- [ ] The ADD handler steps mapped to the by-hand commands (7.2).
- [ ] The ClusterIP→endpoint DNAT path with syncProxyRules file:line, both backends (7.3).
- [ ] The NetworkPolicy AND-vs-OR rule and the DNS diagnostic checklist (7.4).
- [ ] The eBPF load→attach→read-map→detach sequence + the uname -r gate (7.5).
- [ ] The policy-drop map entry / cilium monitor event (7.6).
Falsifiable claims — write, then verify:
- [ ] Why Endpoints didn't scale and slicing fixed it (KEP-752).
- [ ] The structural reason nftables scales where iptables' chain-per-service doesn't.
- [ ] Why NetworkPolicy needs a CNI to enforce it (k8s ships no enforcer).
8. Gate¶
You may advance to P8 when:
- The CNI plugin gives pods a
ping-able network, and you can name every step it automates against the by-handip netnswiring. If the plugin works but you can't map it back to the primitives, P0 didn't land — stay here. - A NetworkPolicy drop is proven at the datapath — you point at the rule or map entry that drops the packet, cited. "The connection failed" does not pass this gate.
- P6's trace debt is repaid and the eBPF program ran in a real kernel — trace #2's far half is now cited in
syncProxyRules/reconciler.go, and Cilium's "it uses eBPF" is a claim about something you personally loaded. The seam P6 could only watch is closed in source.
Networking is the last major subsystem below the abstraction line except one. P8 opens storage — the CSI datapath, the PVC-bound-and-mounted trace — and then closes the drill block with CKA, the first exam since CKAD at P1.