Skip to content

2.C2 — stop two of three and meet a cluster that cannot decide anything

Claim — with two of three members stopped, the survivor serves stale serializable reads and nothing else: no writes, no linearizable reads, no member changes, no leader. Recovery is not a restart; it is a decision to abandon the old cluster's membership, and --force-new-cluster is the flag that makes it.

Rests on — exercise 15, which took one member away. This takes two, and the difference is not one of degree: at one loss the cluster is fine and one member is stranded, at two losses there is no cluster. This is drill 2.C2 and manual drill 3, and it is capstone artifact 1.

Topology — etcd-only, all three healthy and hashes agreeing after exercise 26.

Setup

Take a snapshot first. You are about to do something with a real chance of ending with a cluster you cannot recover, and the next exercise is the other half of this one:

etcdctl put /quorum/before yes
etcdctl snapshot save /tmp/before-quorum-loss.db
scp /tmp/before-quorum-loss.db zain@10.10.10.160:~   # keep a copy off the member you will keep
etcdctl member list -w table                          # you will need these IDs when member list stops working

Do

  1. Stop two members. Stop the guests, not the process — the strand says why: a killed process is a lesson about supervision, and this drill is about a machine being gone. On the Proxmox host:
qm stop <vmid-of-.161>
qm stop <vmid-of-.162>
  1. From .160, try everything, and predict each result first:
etcdctl --endpoints=http://127.0.0.1:2379 put /quorum/after no
etcdctl --endpoints=http://127.0.0.1:2379 get /quorum/before
etcdctl --endpoints=http://127.0.0.1:2379 get /quorum/before --consistency=s
etcdctl --endpoints=http://127.0.0.1:2379 member list
etcdctl --endpoints=http://127.0.0.1:2379 member remove <id-of-.161>
etcdctl --endpoints=http://127.0.0.1:2379 endpoint status -w table

  1. member remove is the one to think about. It is the operation that would fix the problem — reducing the cluster to one member makes one member a quorum. Say why it is refused, and why a system that permitted it would be unsafe.

  2. Read what the survivor says about itself:

ssh zain@10.10.10.160 'journalctl -u etcd -n 40 --no-pager'
  1. The recovery. Stop the survivor, and restart it declaring itself the whole cluster:
ssh zain@10.10.10.160 'sudo systemctl stop etcd'
ssh zain@10.10.10.160 'sudo etcdutl snapshot restore --help; sudo etcd --help 2>&1 | grep -i force-new-cluster'

Add --force-new-cluster to .160's unit, start it, and read what it did:

ssh zain@10.10.10.160 'sudo systemctl daemon-reload && sudo systemctl start etcd'
etcdctl --endpoints=http://10.10.10.160:2379 member list -w table
etcdctl --endpoints=http://10.10.10.160:2379 put /quorum/after yes

  1. Take the flag back out immediately and restart. A member that keeps --force-new-cluster in its unit will discard the cluster's membership every time it restarts, which is a foot-gun that lies dormant until a reboot months later.

  2. Rebuild to three. Start the two stopped guests, and notice that they come back believing in the old cluster — they have the old membership on disk and it no longer exists:

qm start <vmid-of-.161>
ssh zain@10.10.10.161 'journalctl -u etcd -n 30 --no-pager'

Fix them the way exercise 24 established: member add on the survivor, wipe the data dir, --initial-cluster-state existing, start. Do one at a time and check health between.

Observe — step 2's write, linearizable read, member list and member remove all fail with a timeout or a no-leader error; the serializable read succeeds and returns the value from before the loss.

Expect — --force-new-cluster rewrites the surviving member's membership so that it alone is the cluster, at the cost of discarding any entry the survivor had not yet received. That is the trade and it must be stated out loud: this is not a repair, it is an assertion that whatever the two dead members knew and this one did not is now acceptably lost. On a Kubernetes cluster that is objects created in the last few seconds.

Step 3's answer is the same safety rule as exercise 15, one level up. A membership change is a Raft log entry like any other, so changing membership requires a quorum of the membership you are changing — a cluster cannot vote itself smaller when it cannot vote at all. --force-new-cluster is not an exception to that rule; it is an operator stepping outside the protocol and taking responsibility for the consequence.

Footprint note — qm stop frees 2.0GB for the duration, which is unhelpfully the wrong direction: the phase's whole 3.0GB was never the constraint. What it does buy is honesty — no chaos tool can express this, so the fault is a machine that is off.

Write down — the six outcomes from step 2, the answer to step 3, and what --force-new-cluster costs. Then the ordered recovery, written so someone else could follow it. That procedure plus the restore is capstone artifact 1.

Teardown — confirm --force-new-cluster is out of .160's unit, confirm three healthy members with agreeing hashkv, etcdctl del --prefix /quorum/. Keep /tmp/before-quorum-loss.db — the restore drill wants a snapshot that is deliberately stale. The topology stays.