Phase 4 — Controllers & reconciliation¶

4 weeks. The area with the best on-ramp in the corpus — a complete controller in one commented file — and the phase where the watch machinery you read in P2–P3 becomes machinery you write. The range is planning information. The gate at the bottom decides when the phase is finished.


Prerequisites	P3 — the watch cache you read there is the far end of the informer's watch here, and the `ErrCompacted → 410 → relist` chain (etcd P2, apiserver P3) is exactly what the reflector's `watchHandler` survives. This phase is where that chain stops being a fact and becomes a code path you own.
Unlocks	P5 — the scheduler is a controller with an unusually interesting `syncHandler`, and its informers and workqueue are the ones you build here. P12 — Flux's four controllers are the reconcile loop you hand-write, so GitOps practice there rests on the mechanism here. This also closes the gap P1 opened: Helm had no controller and did not reconcile; a controller does.
Source area	Area 4 — Controllers, entry point `sample-controller/controller.go`. Read its reflector→DeltaFIFO→indexer→workqueue diagram before any `tools/cache` source — the other order is the specific failure mode the area warns about (`reflector.go` and `shared_informer.go` are miserable cold, delightful once the pattern is known).
Language	Go — two build artifacts, hand-wired `client-go` then `kubebuilder`, both gated by `envtest` so the comparison is a `git diff`, not an anecdote.
Strands	source reading · build · talks
Labs	`labs/04/` — thirty-four exercises, in order

1. Objectives¶

Every one is falsifiable — an artifact, a timed production, or a claim a hostile reader could check against source or a running cluster. understand and know appear nowhere.

By the end you can:

Draw reflector → DeltaFIFO → indexer → workqueue and name the file implementing each stage — derived from sample-controller before opening tools/cache, which is the correct reading order.
State why a handler enqueues a key, not the object, and why the worker re-reads from the lister rather than trusting the event payload — citing sample-controller/controller.go and controllers.md. This is level-triggered reconciliation, not a style choice.
Show the reflector surviving a 410 Gone with a relist, citing reflector.go's watchHandler — the client end of P2's ErrCompacted and P3's watch-cache ring buffer.
Hand-wire a client-go operator — informer, lister, rate-limiting workqueue, CRD, finalizer, status subresource, all by hand — that passes envtest.
Explain the ControllerExpectations mechanism and why a diff-driven controller over-creates without it, citing controller_utils.go and replica_set.go.
Diagnose a namespace wedged in Terminating to the specific finalizer, and name what Foreground/Background/Orphan propagation each do — citing the GC graph and the namespace controller.
Run two replicas with and without leader election and show the LeaseLock renew-deadline-vs-lease-duration behaviour, citing leaderelection.go.
Produce a git diff of the kubebuilder scaffold against your hand-wired operator, naming what controller-gen, zz_generated.deepcopy.go and envtest produced that you had written by hand.

2. Modules¶

Reading is Area 4, and the sequencing rule is load-bearing: the picture and sample-controller first, tools/cache in earnest only after. Each cited item carries a question to answer — no bare links.

Module 4.1 — The reconcile principle, and the picture first (~4 days)¶

Before any client-go source, install the one idea everything else serves.

Read

Item	Answer from it
`controllers.md` (item 1, read first, always)	Why do you re-read from the cache instead of trusting the event, and what makes a reconcile safe to run twice? State the idempotency rule in one sentence.
`sample-controller` README + the `client-go-controller-interaction` diagram (item 2)	The reflector→DeltaFIFO→indexer→workqueue picture with prose. Ten minutes that saves ten hours — draw it from memory afterwards.
`sample-controller/controller.go` (item 3, ⭐)	The Rosetta Stone: one file with informer setup, key-enqueuing handlers, the worker loop, `syncHandler`, owner refs and status update. Which line enqueues a key rather than the object, and why?

Labs — Draw the pipeline before you open any of it · The line that enqueues a key, and the line that pays for it · The Rosetta Stone, running against a cluster you can break · One log line per stage, and the order they fire in · Two edits: one gets stomped, one survives

Module 4.2 — The machinery, in reading order (~1 week)¶

Now tools/cache, in the order the area prescribes — small and elegant first, the 55 KB files last and only in part.

Read

Item	Answer from it
`util/workqueue/queue.go` (item 4)	The best first piece of `client-go` source. How do the dirty and processing sets give you de-duplication and single-worker-per-key at once?
`util/workqueue/{delaying,rate_limiting}_queue.go` (item 5)	`AddAfter` and exponential per-item backoff. What symptom does a broken controller show when its rate limiter is wrong — and how would you recognise the hot loop?
`tools/cache/delta_fifo.go` (item 7)	The conceptual crux. Why does the queue hold deltas per key, and what does `Replace` do on a re-list that a plain FIFO could not?
`tools/cache/reflector.go` — `ListAndWatch`, `watchHandler` only (item 8)	The file that connects controllers back to API machinery. Where is `ErrResourceExpired`/`410` handled, and what does the reflector do next? This is objective 3.
`tools/cache/shared_informer.go` — `Run`, `HandleDeltas` (item 9)	One watch shared by many handlers. What does `HasSynced` promise, and why must a controller wait for it before its first reconcile?

Labs — Add one key a hundred times, count the reconciles · The gap between retries, measured against the limiter that sets it · One Pop, one key, several deltas · The delete you never saw, delivered anyway · The third layer of a failure you have met twice · HasSynced is a promise about one moment, not about being current · 4.C4 — a controller with an empty cache concludes nothing should exist · The same drawing, now with the file names checked

Module 4.3 — Build artifact 1: the hand-wired operator (~1 week)¶

The phase's centre. Everything above becomes something you wire yourself, modelled on sample-controller.

Gate — envtest: a real kube-apiserver and etcd out-of-cluster, so the reconciler meets genuine optimistic concurrency, watch delivery, defaulting and validation — not a fake client that agrees with whatever you wrote. This is the objective gate a hand-wired controller otherwise lacks, and the reason module 4.6's diff is a comparison rather than a story.

Labs — The API type you write, and the four things generated from it · Build artifact 1: every moving part wired by hand · The gate: four assertions a fake client would pass and a real API server fails you on · 4.C1 — SIGKILL mid-syncHandler, twice, with one flag changed · The error you returned, and the retry that never came · Stage 2: the same binary, and a ClusterRole discovered one 403 at a time

Module 4.4 — Exemplars, owner refs, GC and finalizers (~1 week)¶

The in-tree controllers, read for the patterns your operator glossed — and the source behind the two nastiest real-world failures.

Read

Item	Answer from it
`pkg/controller/replicaset/replica_set.go` (item 11)	The most readable exemplar in-tree. How does `manageReplicas` compute actual-vs-desired, and where does adopt/orphan happen via owner refs?
`controller_utils.go` — `Expectations` types only (item 12)	The mechanism that stops a controller over-creating while its cache is stale. Why would a naive diff-driven controller create too many pods without it? (This is objective 5, and a bug your hand-wired operator may have.)
`controller-ref.md` + `garbage-collection.md` (items 18–19)	Adoption/orphaning, and why exactly one owner may have `controller: true`. `Orphan`/`Background`/`Foreground` propagation — what does each do to children?
`garbagecollector/graph.go` + `namespace/` + KEP-5080 (items 20–21)	The owner-reference graph, and the `kubernetes` finalizer — the single most-encountered finalizer problem. What makes a namespace stick in `Terminating`?

Labs — manageReplicas: the subtraction, and the two policies wrapped around it · Make your own operator over-create, then name the two defences against it · Three things an owner reference will not do for you · The same delete, three times, with three different meanings · 4.C2 — stuck in Terminating, cleared twice, once wrongly · A namespace that will not go, and the condition that says why · The garbage collector matches on UID, and it checks before it deletes

Module 4.5 — Leader election (~3 days)¶

Self-contained, well-commented, and directly demonstrable — the answer to "what happens when you run two replicas".

Read

Item	Answer from it
`tools/leaderelection/leaderelection.go` (item 16)	`LeaseLock`, the renew deadline versus the lease duration, and the `OnStoppedLeading` contract. What must a controller do the instant it loses the lease, and why is the renew deadline shorter than the lease?
KEP-4355 Coordinated Leader Election (item 17)	`LeaseCandidate` and skew-aware leader selection — the modern extension. When does uncoordinated election pick the wrong leader during an upgrade?

Labs — Two replicas, one Lease, and a failover you can time · Why the renew deadline is shorter than the lease, and the window it does not close · 4.C3 — two replicas, no lease, and the fight you can measure

Module 4.6 — Build artifact 2: kubebuilder, and the diff (~4 days)¶

The scaffold, met after the hand-wiring, so controller-gen, Reconcile(ctx, req), envtest and zz_generated.deepcopy.go land as recognition, not magic (#9 — order fixed, do not swap).

Labs — Scaffold the same operator, and change nothing about the test · The same kill, and where the framework actually helps · The diff: every generated file against the line you wrote by hand

3. Chaos drills¶

Hand-driven, every one — Chaos Mesh is not introduced until P6, deliberately after several phases of manual failure so the tool reads as a scripted wrapper over primitives already met. These drills are all convergence under interruption — the level-triggered guarantee tested, not asserted.

#	Drill	What you must produce afterwards
4.C1	Kill the controller mid-reconcile	That it converges on restart with no lost work, and the line that requeues on error — the capstone's core
4.C2	A finalizer that never completes	The specific finalizer wedging the object/namespace in `Terminating`, and the correct clearance (fix cleanup, not force-remove)
4.C3	Two replicas, no leader election	The concrete symptom of the fight (double-create, status stomp), then the lease that ends it
4.C4	A stale informer cache	Why acting before sync is wrong, and how re-read-from-cache-plus-resync recovers — the reason objective 2 exists

4.C1 is the capstone's restart test; 4.C3 is module 4.5's payoff.

4. Talks¶

Full entries with runtimes under Controllers.

The Life of a Kubernetes Watch Event (Zhang & Cai) — the most load-bearing talk in the corpus, and doubly so now: it follows one change from the etcd watch through the apiserver cache, the reflector, DeltaFIFO, indexer and workqueue — the exact pipeline you build this phase. Watch it first; it is modules 4.1–4.2 in one sitting.
client-go: The Good, The Bad and The Ugly (Cosic) — the honest tour of the moving parts and their sharp edges, aimed precisely at someone wiring a reconcile loop by hand. The edges in this talk are the ones you will hit in module 4.3.
Don't Write Controllers Like Charlie Don't Does (Young) — the current catalogue of controller bugs: status-vs-spec confusion, non-idempotent reconciles, hot loops, requeue misuse. Watch it after 4.3, as a review of the operator you just wrote.

5. Ecosystem¶

Flux — the reconcile loop you just hand-wrote, running in production (#6 settled Flux over Argo).

Hands-on: read Flux's controller source immediately after building your own operator — the single best available reality-check on what you wrote. Its four controllers (source, kustomize, helm, notification) are the same informer→workqueue→reconcile shape, with real reconcile intervals and real drift correction. The lab is Someone else's reconcile loop, read the day after you wrote yours.
Internals note: Flux closes the gap P1 opened. There, helm upgrade did not reconcile — delete a resource it made and it stayed gone. Flux's helm-controller does reconcile: it re-reads desired state (a git commit) against actual and corrects drift, on an interval, forever. That difference — a controller versus a client-side templater — is the whole GitOps argument, and now you can point at the loop that makes it true.
Maturity: CNCF graduated. GitOps splits across two phases by design: its mechanism is here ("how does a controller make the cluster match a git commit"); its practice is P12 ("how do you run a delivery model on that").

6. Capstone¶

The hand-wired operator managing a CRD end to end, surviving a restart mid-reconcile with no lost work — plus a written git diff of what kubebuilder scaffolded against what you wrote by hand.

Two artifacts, checked together:

The operator (module 4.3): a CRD reconciled through a hand-wired informer/lister/workqueue, with a finalizer and a status subresource, passing envtest. The no-lost-work proof is the point: SIGKILL it mid-syncHandler (drill 4.C1) and it converges on restart, because reconcile is level-triggered and re-reads from cache — not because it buffered the event. Demonstrate this, do not assert it.
The diff (module 4.6): the kubebuilder version of the same operator, with a written mapping of every generated artifact to its hand-written equivalent.

Cite file:line a hostile reader could check — at minimum: - the key-enqueue-not-object line in your controller, against its sample-controller/controller.go model; - the requeue-on-error line that makes restart lossless; - the ErrResourceExpired/410 handling in tools/cache/reflector.go that your informer relies on; - the Expectations types in controller_utils.go if your operator needed them to avoid over-creating.

Lab — The capstone: two operators, one proof, four citations, which assembles the two artifacts, the transcript and the citations.

Every path verified live per P2's archaeology standard — the two 55 KB tools/cache files are exactly where a copied line number rots.

7. Checklist¶

Concrete, demonstrable, grouped by evidence type. No item says understand or know.

Live against a running cluster / envtest: - [ ] The hand-wired operator reconciles a CRD, with finalizer and status subresource, passing envtest. - [ ] SIGKILL it mid-reconcile; it converges on restart with no lost work. - [ ] Two replicas fight without leader election, then cooperate with it — timed failover shown.

Build artifacts: - [ ] Artifact 1: the hand-wired client-go operator (envtest-gated). - [ ] Artifact 2: the kubebuilder operator, same job, same envtest. - [ ] The git diff mapping every scaffolded file to its hand-written equivalent.

Written artifacts (each is a module's Write-down): - [ ] The reflector→DeltaFIFO→indexer→workqueue diagram from memory, file per stage (4.1). - [ ] The 410-to-relist path citing reflector.go, tied to P2's ErrCompacted (4.2). - [ ] The five hand-wired parts mapped to sample-controller, and the requeue line (4.3). - [ ] The Terminating diagnosis — finalizer, cause, propagation policy (4.4). - [ ] The observed leader-election failover timing citing leaderelection.go (4.5). - [ ] The capstone diff (4.6).

Falsifiable claims — write, then verify against source: - [ ] Why a handler enqueues a key, not the object. - [ ] Why a diff-driven controller over-creates without Expectations. - [ ] Why the renew deadline is shorter than the lease duration.

8. Gate¶

You may advance to P5 when:

The hand-wired operator survives a mid-reconcile kill with no lost work, and you can point at the requeue line and say why level-triggered reconciliation makes the restart safe. If it loses work, the reconcile is trusting the event — stay here, because P5's scheduler is this same loop with a harder syncHandler.
The kubebuilder diff is done and every generated file has a named hand-written equivalent. The scaffold must read as recognition — if any of it is still magic, the hand-wiring did not land.
The 410-to-relist chain is reflexive across all three layers — etcd ErrCompacted (P2), apiserver watch-cache (P3), and your own reflector relisting. This is the third time you have met it; from here it is assumed, never re-taught.

The reconcile loop is the pattern the rest of the control plane is built from. When P5 opens the scheduler, its informers, its workqueue, and its "re-read, don't trust the event" discipline are all machinery you have now written by hand and watched survive being killed.