Spotify End-to-End: How Everything Connects

Concept. Follow a play event through writes, batch processing, and analytics.

The Journey

Spotify end-to-end architecture: one click triggers three systems. Act 1 OLTP (LSM tree write), Act 2 ETL (Spark hash partition, BigSort, broadcast join, columnar write), Act 3 OLAP (star schema query with broadcast and columnar scan).

Figure 1. LSM writes feed a batch pipeline that produces columnar data for analytics.

Act 1 · The Click (OLTP)

You tap play. In ~8 ms, this happens:

INSERT INTO listens (user_id, song_id, timestamp)
VALUES (123456, 789, NOW());

The write lands in an in-memory MemTable, returns, and is flushed to disk later as part of an SSTable. No random disk I/O on the hot path. That's why Spotify can sustain 100 K writes/sec per node.

→ Algorithm: LSM Tree (lsm-indexes.html)

8 ms response · 100K writes/sec · 8.6B events/day

Act 2 · The Transform (ETL)

At 2 AM, Spark starts on yesterday's 8.6 billion events and loads them into the analytics warehouse.

# 1. Hash Partition  →  spread across 1000 nodes
listens.repartition(1000, "user_id")

# 2. BigSort         →  sort within each partition
       .sortWithinPartitions("timestamp")

# 3. Broadcast Join  →  small dim tables to every node
       .join(broadcast(songs), "song_id")

# 4. Write Columnar  →  compress 100 TB → 10 TB
       .write.parquet("s3://warehouse/")

Each step is one of the algorithms from M3:

Step Algorithm Page
Repartition by user_id Hash Partitioning hash-partitioning.html
Sort within partitions BigSort big-sort.html
Join dim tables Broadcast Join distributed-query.html
Write Parquet Columnar storage storage-layout.html

3 hours total · 1000-node Spark cluster · 100 TB processed per day

Act 3 · The Dashboard (OLAP)

CEO opens the mobile app. The "Top Artists" dashboard renders in 2 seconds:

-- "Top Artists by Country This Month"
SELECT artist, country, COUNT(*) AS plays
FROM fact_plays f
JOIN dim_artists a ON f.artist_id = a.artist_id   -- broadcast (10 MB)
WHERE f.date >= '2024-01-01'
GROUP BY artist, country
ORDER BY plays DESC
LIMIT 100;

A broadcast sends the 10 MB dim_artists to all 1000 nodes, so there is no shuffle. The scan reads the 1 PB fact_plays columnar, touching only artist_id, country, date out of 20 columns.

→ Algorithms: Broadcast Join + Columnar Scan (distributed-query.html, storage-layout.html)

2-second query · 3 of 20 columns scanned · 1 PB warehouse

The Pattern

One pipeline, opposite optimizations at each end. Act 1 is write-bound, so the click lands in an LSM tree with no random I/O and returns in under 10 ms. Act 3 is scan-bound, so the dashboard reads compressed columnar storage and answers in about 2 seconds. Act 2 is the 3-hour nightly ETL that converts the first format into the second. No single structure wins everywhere; each stage uses the one built for its workload.