Consistency in Replicas

Concept. Replicas keep copies of the data on several machines, so it survives a dead disk or a dead node. Every write has to reach enough replicas to keep the copies in sync. Systems do that one of two ways: send the write to every replica, or route it through one elected leader.

Intuition. When Spotify updates its "current Top 50", every copy has to end up with the same list. Broadcast the write to all copies and write latency is tied to the slowest machine. Funnel it through a leader that needs only a majority to agree, and the cluster stays fast and keeps working when a machine dies.


Option 1: Write to All Replicas

Any one copy can serve a read. A write goes to every copy, then waits for every one of them to confirm before the client is told it is safe.

A client sends one write to all three replicas; two acknowledge but the third is slow or crashed, so the write blocks and the client cannot reply OK, because every copy must confirm.

Figure 1. Writing to all replicas has no fault tolerance. A write is safe only once every copy acknowledges, so a single slow or crashed replica (amber) stalls every write while the client waits. In a cluster where machines fail routinely, waiting on all of them is unusable.


Option 2: Write to a Leader

Reads are already fast. The cluster elects one leader and routes every write through it, and the leader forwards each write to the followers. A write is safe once a majority confirms it, so a minority can be down. The leader also orders every write, so there is no separate vote per write: running a fresh majority round on every write would cost about 200 ms each, while a standing leader drops that to about 50 ms.

The Majority: What "Enough" Means

A write under Option 2 commits once a majority of the copies have it, ⌊N/2⌋ + 1. That single rule makes the leader both fault-tolerant (a minority can be down and a majority still forms) and safe (two majorities always overlap, so two leaders can never both win).

Nodes Majority Tolerate Use Case
3 2 1 failure Dev/Test
5 3 2 failures Production
7 4 3 failures Critical

Formula: Majority = ⌊N/2⌋ + 1

Five nodes; one majority is any three of them, and any other majority must share at least one node, so a node that votes once per term cannot back two winners and split-brain is impossible.

Figure 2. Any two majorities of a five-node cluster must overlap on at least one node. Since each node casts one vote per term, that shared node cannot put two candidates over the top, so at most one leader is ever elected. That same overlap lets a majority both tolerate failures and prevent split-brain.

Read and Write Quorums

The same overlap idea tunes read and write consistency, not just leader election. Give every copy a vote on the data too. A write waits for W replicas to ack before it returns (the write quorum); a read gathers R replicas and takes the newest value it sees (the read quorum). With N copies, one rule decides consistency:

R + W > N forces every read quorum to share at least one node with the last write quorum, so a read always sees the latest write. That is strong consistency, and the majority rule (W = R = ⌊N/2⌋ + 1) is its symmetric case. You can tune the split instead: W = N, R = 1 makes reads one-hop fast and writes slow; W = 1, R = N does the reverse. Either stays strongly consistent as long as R + W > N.

Drop below the line and the guarantee goes. R + W ≤ N lets a read quorum miss the last write entirely, so reads can return stale data (eventual consistency). Setting W = 1, R = 1 on five copies (R + W = 2 ≤ 5) is the fast, weakly consistent extreme.

Durability is a separate axis. Even at W = 1 the write is durable: the one replica that acks logs it to its write-ahead log first, exactly as in single-node recovery. A small W weakens consistency and failure tolerance, not durability.

Replacing a Dead Leader

The leader itself can die. The survivors run an election, and the same majority rule guarantees only one winner.

Leader election timeline: when the leader fails, followers detect, vote, and elect a new leader

Figure 3. Raft elects a new leader after the old one fails. Each node votes at most once per term, so only one candidate can clear a majority, which rules out split-brain. The timeout that triggers the election is randomized per node, so followers rarely become candidates at the same instant. Production Raft systems like etcd and Consul elect a new leader within about 200 ms of the failure.


Copying to Followers: Synchronous or Asynchronous

Option 2 routes every write through the leader, which leaves one last choice: when the leader tells the client the write is safe. After a follower confirms it, or as soon as the leader has it.

Synchronous replication waits for a follower to acknowledge before replying OK to the client, losing nothing on failover but adding latency; asynchronous replication replies OK immediately and ships to the follower later, which is fast but can lose the un-shipped tail if the leader fails.

Figure 4. The two modes trade latency against durability. Synchronous replication forwards each write to a follower and waits for its acknowledgement before replying OK, so a failover to that follower loses no committed write, at the cost of a full round trip per write. Asynchronous replication replies OK as soon as the leader has the write and ships it to followers in the background, which is fast but means a leader crash can lose any writes that had not yet shipped. The gap between leader and follower is replication lag.

Reading from a replica is stale, not wrong

Read replicas soak up read traffic, but an asynchronous replica is always a little behind. A query there sees a consistent snapshot (MVCC still gives it one self-consistent view), just an older one, as of whatever the replica has applied. The practical trap is read-your-writes: you write to the leader, immediately read from a replica, and your own write is not there yet because it has not replicated. The data is consistent, just not current.

Picking a mode

Need zero data loss on failover (payments, the system of record)? Replicate synchronously and pay the latency. Serving analytics or feeds where a few seconds behind is fine? Replicate asynchronously and read from replicas. Most large systems do both: synchronous within a region for durability, asynchronous across regions for reach.


Key Takeaways

1. Core Problem

Split-brain is the failure: two nodes both act as leader and corrupt the data. Every node votes once per term, so two candidates can never both clear a majority, and at most one leader is ever elected for a term.

2. The Raft Way

Three states, Follower → Candidate → Leader, with one rule: win a majority to lead. But a majority only forms when enough machines can still reach each other. When the network splits, neither side may have one, and that is the CAP tradeoff on the next page.