Case Study 1.2: How ChatGPT Stores Its Chats

Concept. The world's most-used AI product runs on SQL. ChatGPT stores every conversation in Postgres: each message is a row you write once and read many times. Reads dominate, so OpenAI keeps one writable primary and fans reads out to about 50 read replicas.

Intuition. One ChatGPT question is a single write. Then every follow-up reloads the whole conversation to give the model its context, and the sidebar reloads past chats, so hundreds of reads pile on that one write. It is the same Listens table you have been querying, scaled to about 800 million users.

Case Study 1.2 Reading Time: 5 mins

Everything so far was the grounding: you define the term, you verify it, and you keep it, while the agent drafts and runs the SQL. That is the top of the stack. This page is the bottom layer, the other reason SQL matters: nearly every AI product still runs on a SQL database, and ChatGPT is the clearest example.

The Read-Heavy Constraint

Scaling to 800 million users looks different for every application. For an email service like Gmail, the system sustains a high, continuous volume of writes. But for ChatGPT, the primary operational bottleneck is read throughput. As detailed in their engineering post "Scaling PostgreSQL to power 800 million ChatGPT users", OpenAI sustains this load using standard PostgreSQL.

One ChatGPT question is one write (save the new message), then hundreds of reads (fetch history, load the sidebar, build the LLM context). Reads are the bottleneck, not writes, so the system scales reads.

Isolating Read and Write Traffic

ChatGPT's traffic has a large read/write imbalance: one write, then hundreds of reads. Every time you ask a single question (one write), the system fetches your entire conversation history to give the model its context (hundreds of reads), and the sidebar keeps reloading past chats.

OpenAI deployed a single primary database dedicated exclusively to saving new writes. To handle the read volume from context windows and the UI, they provisioned 50 globally distributed read replicas. This architecture physically separates read traffic from write traffic. It maintains the ACID constraints of standard SQL for saving new data, while horizontally expanding read capacity.

Writes from new messages go to a single writable primary that holds the truth with ACID guarantees. The primary replicates to 50 globally distributed read replicas, and all the read traffic (history, sidebars) fans out to those replicas, separating read traffic from write traffic.

(Note: we cover sharding and replication in depth in Module 5, and the strict safety guarantees in Module 4.)

The Bottom Line

For read-heavy workloads, a single primary plus read replicas scales the reads horizontally. Optimization at this level requires carefully structured data models and clear query patterns, not custom database engines.

However, when a system exhausts horizontal read replication or encounters high-volume unstructured parallel writes, this architecture degrades. We cover circumventing these hard limits using Key-Value stores and Data Lakes in Module 6.