Case Study: How did Google store the entire internet on cheap, dying hard drives?

Reading Time: 8 mins

How did Google store the entire internet on cheap, dying hard drives?

Goal: Understand how sharding, replication, and leader election address the capacity and reliability limits of single machines.

Problem 1: Cheap, Failing Hardware

In the early 2000s, Google had to store petabytes of web crawl data. High-end supercomputers from legacy vendors cost too much, so they linked thousands of cheap, consumer-grade hard drives instead. Back then these were spinning disks, not today's SSDs.

The Catch: Consumer HDDs are unreliable. Their Mean Time To Failure (MTTF) is low, so across 10,000 spinning disks a failure every day was a certainty. Google needed software that treats failing hardware as normal.

Google chose thousands of cheap commodity disks over a few costly enterprise drives; across a grid of 10,000 disks, several fail (red) every day, so the software is built to treat failure as routine.

Figure 1. Cheap hardware at scale. Google linked thousands of commodity disks instead of a few costly enterprise drives. Across ten thousand disks a failure every day is a certainty, shown by the red disks in the grid, so the software treats failure as routine rather than an emergency.

Problem 2: The Capacity Problem (Sharding)

The Problem: The web crawl reached 100 Terabytes, while the largest hard drive was just 1 Terabyte. A single machine couldn't handle it.

The Solution (Chunks / Sharding): Google's engineers split the massive file into 64-Megabyte pieces, known as "Chunks."

A 100 TB file cannot fit on a 1 TB disk, so it is split into 64 MB chunks (about 1.6 million of them) and the chunks are scattered across machines, so total capacity grows with every machine added.

Figure 2. Sharding for capacity. A 100 TB file cannot fit on a 1 TB disk, so GFS splits it into 64 MB chunks, about 1.6 million of them, and scatters them across machines. Total capacity then grows with every machine added to the cluster.

The Two-Tier Architecture: Google developed a two-tier system with a Leader and numerous Chunk Servers.

  • Chunk Servers: These are the cheap machines storing the 64MB data chunks.

  • Leader: This central node keeps an in-memory directory. For every file it stores the ordered list of chunk handles; for every chunk handle it stores the set of Chunk Servers holding a replica. A client sends a (file name, byte offset); the Leader turns the offset into a chunk index, looks up the matching chunk handle and its replica locations, and hands them back. The client then reads from a Chunk Server directly. Where each chunk physically lands is the Leader's own placement decision, balancing disk usage and load across the cluster, not a function of the file's name.

A client, a leader, and three chunk servers: the client asks the leader for a chunk's replica locations, the leader replies with the chunk handle and server list, then the client reads the chunk directly from a chunk server; every chunk is stored as three replicas on different servers.

Figure 3. The GFS lookup. Data never streams through the Leader: the client asks it only for a chunk's replica locations (steps 1 and 2), then reads the chunk directly from a Chunk Server (step 3). The Leader holds just the directory, small enough to keep in memory. Every chunk is stored as three replicas on different servers, here chunk C7 on all three, so a dead disk costs nothing.

Problem 3: The Reliability Problem (Replication)

The Problem: With 1,000 chunks across 1,000 Chunk Servers and low MTTF, server failures were inevitable. Losing even one chunk could corrupt the entire 100TB file.

The Solution (Replicas): Store each 64 MB chunk as three identical copies (Replicas) on different Chunk Servers, ideally on separate power racks.

The Result is Self-Healing: When a Chunk Server fails, the Leader redirects requests to a surviving replica and copies the chunk to a healthy server.

Every chunk is stored as three replicas on different racks; when one server dies, two replicas keep serving and the Leader copies the chunk to a fresh healthy server, restoring three copies.

Figure 4. Replication and self-healing. Every chunk is stored as three identical replicas on different racks. When a server dies, two replicas keep serving, and the Leader copies the chunk to a fresh healthy server to restore three copies.

Summary: The Impact

By combining Sharding for capacity and Replication for reliability, the Google File System (GFS) showed you can build a scalable, reliable system from unreliable hardware. 2003 GFS leaned on a single master, kept available through external monitoring and fast restart; replacing that single master with the consensus-based leader election (Paxos/Raft) covered earlier in this module came later in practice. This architecture laid the groundwork for Hadoop (HDFS) and the modern cloud.

Read the Original Paper

The original 2003 Google File System paper started the Big Data era. Focus on the overall architecture.

If the embed doesn't load in your browser, open the paper directly: The Google File System (research.google).

Where This Leads

GFS combines everything this module covers, and each limit it hit maps to another page. Two of those threads you have already seen; two are just ahead.

  • Master as a single point of failure → Consensus & Replication, earlier in this module, replaces one Master with a Paxos/Raft-coordinated council.

  • Resharding pain when machines join or leave → Distributed Sort & Hash, earlier in this module, gives consistent hashing (the ring), used by Dynamo and Cassandra.

  • 3× replication's storage overhead → Distributed File Systems traces the move to erasure coding in Colossus (~50% overhead instead of 200%).

  • Master metadata RAM ceiling → Distributed File Systems covers the move to sharded metadata (Colossus, HDFS Federation).

The next page picks up the thread from here, generalizing GFS into the four-generation pattern (GFS → HDFS → S3 → Colossus).