The third layer of a failure you have met twice¶
Claim — you can make your own reflector receive 410 Gone, catch the line that handles it, and show what it does next — completing a chain whose first two layers you already produced: etcd's compaction and the apiserver's watch cache. Neither of those mechanisms is restated here. This exercise adds only the client's end, which is the one that has to survive.
Rests on — the watchable_store.go reading question for what compaction does to a slow watcher, and module 3.5 for the ring and the prediction of its capacity. Do not re-derive either; produce the event and read the client.
Topology — pair, continued, with the probe running on forge.
Do
- Find the handling before you cause it, and answer one question from the source that the log will not tell you:
cd ~/src/kubernetes && git grep -n 'ErrResourceExpired\|IsResourceExpired\|isExpiredError\|too old resource version' \
-- staging/src/k8s.io/client-go/tools/cache/reflector.go
The question: after the expiry, which resourceVersion does the next LIST ask for — the last one the reflector held, 0, or none at all? Read the function that chooses it, and say in one sentence what that choice costs and what it buys. There is a real trade there between a stale answer served cheaply and a fresh one served expensively, and the reflector picks a side.
- Freeze the probe so its watch falls behind:
- Churn the resource it is watching, hard enough to push its position off the end of the ring. Start with 500 and go up if the thaw produces no expiry:
for i in $(seq 1 500); do
kubectl -n academy-lab create configmap churn-$i --from-literal=n=$i >/dev/null
done
Five hundred creates is deliberate: creates and deletes both count as events on the ring, but creates leave something behind you can count afterwards, so a run that failed to overflow is distinguishable from a run that overflowed and recovered.
- Thaw it and read the log:
Observe — the reflector's own log lines around the thaw, and the POP burst that follows.
Expect — a warning naming the resource type and the expiry, then a re-list, then Replaced deltas for everything that still exists. The controller does not crash, does not miss objects, and cannot say which specific changes it missed — it re-derives the world instead of replaying the events, which is the same property exercise 5 showed at the reconcile layer, here at the cache layer.
If 500 objects produce no expiry, the ring grew; that growth is module 3.5's subject and the number to raise is the churn, not the freeze time. Confirm the failure is the one you wanted by checking that the message names an expired resource version and not a closed connection — the two look alike in a log and are not the same failure.
Write down — the checklist's 4.2 artifact: the 410-to-relist path with the reflector.go file:line, tied back to the two layers below it by name. This is objective 3 and it is gate condition 3; after this it is assumed and never taught again.
Footprint note — 500 ConfigMaps is a few MB in etcd on a cluster with nothing else in it, and they are deleted in the teardown. This is the phase's only deliberate churn.
Teardown
Stop the probe. The topology stays — exercise 11 restarts it against a clean namespace, which is precisely the state it needs.