Distributed File Systems

Concept. A distributed file system splits each large file into fixed-size chunks, replicates each chunk across many cheap commodity servers, and gives clients a single-namespace view, so the cluster delivers sequential read throughput proportional to its size while tolerating constant disk and machine failures.

Intuition. When Spotify stores a 64 GB audio-feature export, GFS splits it into 1,024 chunks of 64 MB each, places 3 copies of every chunk on different machines, and lets a job read all 1,024 chunks in parallel from 1,024 disks. Even if a disk dies mid-read, two other replicas of the dead chunk are already on standby.

The Pattern, and What Changes Across Generations

The GFS case study already gave us the pattern: split each file into chunks, replicate them across cheap machines, and track it all on a single Master that holds every file's metadata in RAM. Every distributed file system since reuses that chunks-and-replicas substrate; what evolved is how each one escapes GFS's single-Master bottleneck, and that one choice cascades into scale, latency, and consistency.

Four columns by year: GFS (2003) and HDFS (2006) keep all metadata on one pink Master or NameNode capped near petabyte scale; S3 (2006) spreads metadata across leaderless pink nodes scaling without limit; Colossus (2010+) shards metadata with Reed-Solomon to exabyte scale; grey boxes are the chunk servers underneath

Figure 1. Four generations share one chunks-and-replicas substrate but coordinate metadata differently, and that single choice sets each scale ceiling. GFS (2003) and HDFS (2006) keep all metadata on one Master or NameNode (64 MB / 128 MB chunks, 3× replication, CP), so the single leader caps them near petabyte scale. S3 (2006) drops the leader for distributed metadata (redundant across ≥3 availability zones, designed for 11 nines of durability) and scales effectively without limit; since 2020 it is strongly read-after-write consistent (earlier, eventually consistent). Colossus (2010+) shards metadata across many nodes with Reed-Solomon erasure coding (~50% overhead versus 200% for 3× replication, CP) and reaches exabyte scale.

HDFS (2006): Open-Source GFS

HDFS is the open-source re-implementation of GFS. One NameNode plays the Master role; DataNodes store 128 MB blocks (twice GFS's chunk size). Replication is rack-aware, so a rack losing power never takes every copy of a block. Same architecture as GFS, so the same single-NameNode bottleneck, which HDFS Federation later sharded to push past.

S3 (2006): No Leader at All

Amazon S3 took a different turn. Instead of one Master, S3 has no central metadata coordinator. Every metadata operation hits a distributed key-value store, and the storage backend is a partitioned cluster spanning availability zones.

  • No leader bottleneck. Throughput scales horizontally with cluster size.

  • 11 nines of durability (99.999999999%) by keeping data redundant across ≥3 Availability Zones (erasure-coded, not literal full replicas).

  • Strong read-after-write consistency since December 2020. A PUT is immediately visible to every subsequent GET, LIST, and metadata read, automatically and at no extra cost. (Before 2020, S3 was eventually consistent: a write could take a few seconds to be visible to every reader.)

  • Effectively infinite scale. Customers store exabytes; the largest single buckets hold petabytes.

S3 stays leaderless, favoring availability and scale, with metadata distributed rather than parked on a single namenode. The 2020 change closed the old eventual-consistency window, so reads no longer have to tolerate stale writes.

Colossus (2010+): GFS Without the Bottleneck

Google replaced GFS with Colossus to break the Master bottleneck without giving up consistency.

  • Distributed metadata. Metadata is sharded across many servers; no single node holds the full chunk map.

  • Reed-Solomon erasure coding instead of 3× replication, dropping storage overhead from 200% to ~50% for the same fault tolerance.

  • Sub-millisecond latencies for metadata operations.

  • Exabyte-plus scale at Google.

Colossus splits data into 6 data blocks and 3 parity blocks computed from them; any 3 of the 9 can be lost and rebuilt from the surviving 6, tolerating 3 failures at 50% storage overhead versus 200% for three copies.

Figure 2. Reed-Solomon erasure coding. Colossus splits data into data blocks and computes parity blocks from them, for example six data plus three parity. Any three of the nine can be lost and rebuilt from the six that survive, so it tolerates three failures at 50% storage overhead, where three full copies cost 200% and survive only two failures.

Quick Comparison

System Year Metadata Chunk Size Replication CAP Scale
GFS 2003 Single Master 64 MB 3× CP PB
HDFS 2006 Single NameNode 128 MB 3× rack-aware CP PB
S3 2006 Distributed Variable ≥3 AZs (erasure-coded) Strong RAW since 2020 (earlier eventual) ∞
Colossus 2010+ Distributed Variable Reed-Solomon CP EB+

Real-World Usage

HDFS

The open-source analytics storage layer of the 2010s. Facebook ran a 600 PB+ warehouse on it; Yahoo clusters reached 40,000+ nodes.

Amazon S3

The default cloud object store. Netflix and Spotify serve large catalogs from it, alongside peers like Google Cloud Storage and Azure Blob.

Google Colossus

Google's internal store for YouTube, Gmail, and BigQuery, holding exabytes behind sub-millisecond metadata.

Next

MapReduce → Now that the storage layer can hold petabytes across cheap hardware, the next question is how to compute on that data without moving it across the network. MapReduce gives you a programming model that works directly on top of HDFS-style chunks.