Sharding vs Replication
Concept. Sharding splits one logical dataset across many machines so the cluster can hold and serve more data than any single node can; replication keeps multiple copies of each shard on different machines so a single failure doesn't drop a slice of the data offline. Real systems do both.
Intuition. When Spotify's 10 TB Listens table outgrows one machine's disk, sharding by user_id splits it into disjoint slices across nodes (ten shards of about 1 TB here), and replication keeps 3 copies of each slice on 3 different machines, so any one disk failure is invisible and any user's data can be served from any of 3 places.
Figure 1. Sharding and replication solve different problems and almost always run together. Sharding splits a 10 TB table into disjoint slices across machines, each row on exactly one shard, to scale capacity past one machine. Replication keeps two or three copies of the same data on different machines to survive single-machine failures and serve more reads. Production systems shard then replicate each slice, taking capacity from sharding and availability from replication, as Cassandra, sharded MongoDB clusters, and BigQuery's storage layer do.
Production Synthesis
Sharding handles capacity; replication handles availability.
Figure 2. In production, both at once. Spotify's 10 TB Listens is sharded ten ways for capacity, then each shard is replicated three times on different racks. A user's shard lives on three machines, so a read can hit any of them and a dead machine loses nothing.
Sharding hurdles: cross-shard queries • rebalancing • hotspots Replication limits: adds read throughput, not write capacity or storage. Replication's cost: three copies mean 200% storage overhead. Colossus later cuts that to about 50% for the same durability.
Implementation reality
Sharding is the concept. Enforcing it across a cluster without losing track of which node holds which rows is the mechanism. Next: Hash Partitioning and Consistent Hashing, the math that makes modern sharding work.
Common Confusions
Figure 3. Why both are needed. Sharding without replication (left): when a machine dies its shard goes offline, locking out one in ten users. Replication without sharding (right): every machine must hold the whole 10 TB, which never fits on a 1 TB disk. Only running both closes both gaps.
Q: Can replication help with scale?
Replication boosts read scale, not write capacity or data size.
Q: Why not just replicate instead of sharding?
Replication doesn't solve physical limits. A 10TB dataset won't fit on a 1TB drive, no matter how many copies you make. Sharding is essential for overcoming capacity constraints.
Q: How to choose?
Data too big? → Shard. Need high availability? → Replicate. Both? → Implement both.