Case Study: Key-Value Stores

Why does Spotify store songs and lyrics in a key-value database?

Goal: Learn when to use key-value stores.

The Data Spectrum: Structured vs. Blobs

1. The Structured Core (Tabular) Imagine a world of neatly typed columns: integers, timestamps, booleans.

• The Reality: SQL engines are built to slice and dice billions of these tiny rows. This is your core product logic, tracking who listened to what song, and when.

2. The Massive "Blobs" (Unstructured & Semi-structured) Now think of a 10 GB video file or a sprawling JSON blob with a podcast's details.

• The Reality: SQL engines excel at scanning rows in memory; huge blobs disrupt that efficiency. It's like cramming a video into a spreadsheet cell. Better to store the video elsewhere and just track its location.

This is why Spotify divides its architecture.

The Spotify Strategy: Key-Value Stores

For the unstructured and semi-structured data, Spotify turns to Key-Value databases like Google’s Bigtable or AWS’ DynamoDB.

How it works: Picture a vast locker room. Each locker has a unique number (the Key) and inside is the item (the Value).

• Example: Key = song_123, Value = audio_file.mp3.

• Retrieval: Hit play, and Spotify uses the song's ID to swiftly fetch the audio from the key-value store and stream it to you.

Alternatively, the actual file might sit in a distributed file system (like Amazon S3). The key-value database acts as an index (Key = song_id, Value = S3-location). This setup scales billions of files while keeping performance sharp.

SQL vs. Key-Value: The Trade-offs

Takeaway: Storing each line of music lyrics in separate SQL columns isn't practical. Instead, store it as one semi-structured blob in a Key-Value store: {key: song_id, value: lyrics_blob}. (Or in JSONB in modern SQL databases.)

Beyond Simple Keys: Cassandra's Wide Rows

A pure key-value store returns one opaque value per key. Wide-column stores like Cassandra (and Google's Bigtable) refine that model by splitting the key into two parts, and that split makes feeds, timelines, and inboxes fast:

• Partition key decides which node stores the data. Cassandra hashes it, so all rows sharing a partition key live together on one node.

• Clustering columns decide the sort order within that partition. Rows are stored physically sorted, ready to read in order.

The classic shape is the per-user inbox: one user's items all share a partition key, sorted by time inside the partition.

CREATE TABLE plays_by_user (
    user_id   text,
    played_at timestamp,
    song_id   text,
    PRIMARY KEY ((user_id), played_at)
) WITH CLUSTERING ORDER BY (played_at DESC);

-- "Mickey's 20 most recent plays"
SELECT song_id, played_at FROM plays_by_user
WHERE user_id = 'mickey' LIMIT 20;

Because every one of Mickey's plays lives in a single partition already sorted newest-first, "most recent N for this user" is a single-partition scan: one node reads the first 20 rows and stops. No scatter-gather across the cluster, no global sort. In a SQL database the same query is an ORDER BY played_at DESC LIMIT 20 that, without exactly the right index, scans and sorts a shared table. Modeling the access pattern into the key is the whole game in a wide-column store.

The 2026 Perspective: Converging Worlds

Modern Hybridization: These are traditional distinctions. While Spotify's scale justifies dedicated Key-Value stores, for most companies, these lines blur in 2026.

Next

Data Security → Everything you have built across this module, the lake, the pipeline, the specialized stores, is now an attack surface. Next: what happens when these systems are attacked, and how Zero Trust defends them.