How Google Stored Web-Scale Data on Cheap, Failing Hardware
Concept. GFS splits large files into 64 MB chunks, distributes those chunks across commodity machines, and stores three replicas of each chunk. One Master tracks the metadata while clients transfer data directly to chunkservers.
Intuition. A 100 TB file cannot fit on a 1 TB disk. GFS divides it into about 1.6 million chunks and spreads three copies of each chunk across the cluster. When a disk fails, clients read another replica while the Master restores the missing copy.
Cheap hardware fails
In the early 2000s, Google stored hundreds of terabytes across thousands of disks on more than a thousand commodity machines. The cluster used spinning disks rather than today's SSDs.
At that scale, disks and machines fail routinely. Google designed GFS to keep serving data while components fail.
Figure: Cheap hardware at scale: failure becomes routineGoogle chose commodity disks over a few expensive enterprise drives. Across thousands of disks, components fail routinely, so GFS keeps serving data through those failures. The figure shows the choice (commodity chosen over enterprise), the scale (a grid of ten thousand disks), and the reality (some disks fail every day). Color key: pink is the commodity choice and its consequence, red is a failed disk, grey is the working disks.
Figure 1. Google linked thousands of commodity disks instead of buying a few costly enterprise drives. At that scale, components fail routinely, so GFS keeps serving data while disks and machines fail.
Sharding supplies capacity
Consider a 100 TB crawl and 1 TB disks. No single disk can hold the crawl.
GFS splits each file into 64 MB chunks. A 100 TB file produces about 1.6 million chunks, which GFS distributes across the cluster.
Figure: Sharding: capacity beyond one diskA 100 terabyte file cannot fit on a 1 terabyte disk, so GFS splits it into 64 megabyte chunks and scatters them across many machines. The figure shows the mismatch (file far larger than a disk), the split into millions of fixed-size chunks, and the chunks spread across machines so total capacity grows with the cluster.
Figure 2. Sharding for capacity. A 100 TB file cannot fit on a 1 TB disk, so GFS splits it into 64 MB chunks, about 1.6 million of them, and scatters them across machines. Total capacity then grows with every machine added to the cluster.
GFS uses one Master and many chunkservers.
-
Chunkservers store the 64 MB chunks on local disks.
-
The Master keeps the file namespace, chunk handles, and replica locations in memory. A client sends a file name and byte offset. The Master maps the offset to a chunk, returns its handle and replica locations, and lets the client read directly from a chunkserver. The Master also places new replicas to balance disk use and load.
Figure: GFS: one metadata Master over commodity chunkserversA client, one Master, and three chunkservers. The Master keeps an in-memory directory mapping each file to its ordered chunks and each chunk to the servers holding a replica. To read, the client first asks the Master for a chunk's replica locations. The Master replies with the chunk handle and server list, then the client reads the chunk directly from a chunkserver. GFS stores every chunk as three replicas on different servers, shown by chunk C7 appearing on all three.
Figure 3. The client asks the Master for a chunk handle and its replica locations, then reads directly from a chunkserver. The Master keeps only metadata in memory. Three replicas let reads continue when one disk fails.
Replication handles failures
The 100 TB example contains about 1.6 million chunks across many chunkservers. Losing every replica of one chunk would make part of the file unavailable.
GFS stores three replicas of each 64 MB chunk on different chunkservers and spreads replicas across racks.
When a chunkserver fails, clients read a surviving replica. The Master schedules a new copy on a healthy server and restores the target replica count.
Figure: Replication: self-healing storageGFS stores every 64 megabyte chunk as three identical replicas on different chunkservers across racks. When one server dies, two replicas still serve, and the Master schedules a copy on a healthy server to restore three copies. The figure shows the three replicas, a server failing with two surviving, and the Master re-replicating to a new server.
Figure 4. GFS stores three replicas of each chunk across racks. When one chunkserver fails, clients read a surviving replica while the Master schedules a new copy and restores the target replica count.
The impact
GFS combined sharding for capacity with replication for fault tolerance. Its 2003 design used a single Master, replicated the Master's operation log, and kept shadow Masters available for reads. Later systems changed the metadata design to remove or distribute that bottleneck.
Read the Original Paper
The original 2003 paper documents the architecture and the workload assumptions that shaped it.
If the embed doesn't load in your browser, open the paper directly: The Google File System (research.google).
Where This Leads
GFS combines the mechanisms this module covers, and each limit it hit maps to another page.
-
One Master limits metadata scale. Distributed File Systems shows how Colossus and HDFS Federation shard metadata.
-
Three replicas consume 200% extra storage. Distributed File Systems shows how erasure coding cuts that overhead to about 50%.
-
Changing cluster membership moves data. Distributed Sort & Hash explains how consistent hashing limits that movement.
Takeaway: GFS combines 64 MB chunks, one metadata Master, and three replicas to build scalable, fault-tolerant storage from commodity machines.