Case Study: Claude's KV Cache
Concept. An LLM agent sends a long, nearly identical prefix on every turn: system prompt, tool schemas, and conversation history. The KV cache stores the model's per-token attention state for that prefix, one key vector and one value vector per token at each layer. Keeping it across turns is the biggest lever on an agent's cost and latency. Paging, quantization and offload all manage it, and each is a Module 2 primitive renamed.
Intuition. A Claude agent can carry thousands of tokens of system prompt and tool definitions that stay constant, while the history grows only at the end. Without caching, each new turn runs the entire accumulated prefix through the model again. Prompt caching writes the prefix's computed state once and reuses it on later turns for about a tenth of the price, so later turns pay mainly for the new message.
Every turn is mostly the same prefix
A request to an agent is mostly prefix. The system prompt and tool schemas never change, the history only grows at the end, and the new message is short. So turn after turn, the model re-reads almost exactly the same thing, and recomputing it every time is the single biggest cost in serving an agent.
Figure 1. An agent resends most of its prefix every turn, so caching it saves the most. A request is a long prefix that only grows at the end (system prompt, tool schemas, then the conversation so far) plus a short new message; without caching, the model recomputes the whole prefix each turn, so cost climbs as the conversation grows (grey). Prompt caching stores that prefix's attention state and reads it back on every later turn, so each turn computes only the new message and cost stays flat (slate). It is the same trade as keeping your working set in RAM from Module 2: a cache hit is cheap, recomputing is not.
The fix is to keep what the model computed instead of recomputing it, and that cache is a key-value store of its own, the same idea you just built, specialized for prefixes: the key is the prefix, the value is its KV cache, the intermediate results the model produced while reading the prefix (not its text, and not the tokens it later generates). A repeated prefix becomes a lookup, not a recompute.
Figure 2. What the KV cache stores. The key is the prefix (its tokens); the value is the model's intermediate work from reading it, a key vector and a value vector for each token, at every layer. It is the precomputed state the model would otherwise rebuild each turn, not the token text and not the output, so caching it skips the expensive part: you reuse the work instead of running the prefix through the network again.
That is prompt caching. Anthropic's version (launched August 2024) prices it exactly like a cache: a write costs about a 25% premium, a hit costs about a tenth of the price, and the entry lives five minutes by default. For an agent that resends the same ten-thousand-token prefix hundreds of times, reading it from cache instead of recomputing cuts the cost of that prefix by about ninety percent on every later turn.
Paging: fit many caches in one GPU
Once you cache prefixes, a server is juggling many chats at once, each with a KV cache that grows a token at a time, all sharing one GPU's memory. The move that makes that work is paging.
Figure 3. R1, R2, and R3 are three chats served on one GPU at once, each holding a KV cache (the keys and values the model computed, not the prompt text) that grows one token at a time. The naive way (left) gives each request one contiguous chunk of GPU memory sized to its longest possible length, so most of it is reserved and never used, wasting 60 to 80 percent of GPU memory. PagedAttention (right) splits each request into fixed-size blocks, packs them with no gaps, and keeps a page table mapping each request to its blocks, which can sit anywhere (request one is in blocks 0 and 2, not adjacent), dropping waste under 4 percent. This is the page-table half of Module 2 paging, packing one memory tier rather than moving data between disk and RAM.
PagedAttention (the engine behind vLLM) stores each chat's cache in fixed-size pages of GPU memory instead of one contiguous chunk, with a page table mapping logical positions to scattered GPU memory. That is the fixed-size-page idea from Module 2 paging, used here to pack one scarce memory tier, not to move data across the disk-to-RAM gap. Once you decide to cache prefixes, this is the mechanism that makes it fit.
The rest is Module 2
Once the cache lives in scarce GPU memory, the rest of the playbook is exactly what you already learned.
Quantize it. Drop the keys and values from full precision to a few bits. It is the same quantization lever as TurboQuant, and it shows up twice in an agent, at two memory tiers.
Figure 4. An agent keeps two kinds of memory, and one lever shrinks both. Long-term memory is the vector store it retrieves from, kept small by TurboQuant (Case Study 2.4); working memory is the KV cache of the current context, kept small by KV-cache quantization. Both are dense-vector piles too large to hold at full precision, so both use the same move: drop precision, keep the signal.
Precision is not the only lever. Many chats open with the same system prompt, so a server stores that prefix's KV once and points every chat at it, which is dictionary encoding from Module 2: keep the repeated thing once, reference it. And when a context runs long, eviction keeps only the tokens that still carry weight and drops the rest. (Run-length encoding is the one that does not fit here: these are dense vectors, not long runs of a repeated value.) Quantize, dedup, prune, the same compression toolkit aimed at attention vectors.
Offload it, and price it. When even paging cannot fit everything, push cold blocks from GPU to CPU to disk, the storage hierarchy. And the whole decision to cache rather than recompute is a cost-model call, a cheap hit weighed against an expensive recompute. Line all four up, and every one is a Module 2 idea:
Figure 5. The techniques that keep an agent's KV cache small and fast are Module 2 primitives under new names. PagedAttention breaks the KV cache into fixed-size pages addressed through a page table, so scattered GPU memory packs tight instead of demanding one contiguous chunk; it is the same fixed-size-page idea as Module 2 paging, borrowing the page-table half rather than the disk-to-RAM movement. Prefix and prompt caching turn a recompute into a cheap cache hit, priced by the same cost-model reasoning as Module 2 (here the expensive operation is compute, not a disk read). KV-cache quantization is drop-precision quantization, the same lever as TurboQuant in Case Study 2.4, and offloading the cache across GPU, CPU, and disk is the storage hierarchy. Nothing in the right column is new to you.
And it scales the way every store you have built does. On one machine the cache sits in GPU memory; as it grows you tier it down the hierarchy, GPU to CPU RAM to NVMe disk. Across a fleet, the prefix cache becomes a distributed key-value store: a Redis cluster or a cluster-wide pool (LMCache, Mooncake), sharded and replicated so any machine can reuse a prefix another already computed. The KV cache is a key-value store all the way down.
Takeaway
You started this course writing SELECT against a SQLite file, and watching a model write SQL back to you. You end it looking at how that model is served, and finding the same systems you spent the course on: paging, caching, the cost model, quantization, the storage hierarchy. New hardware, new names, same fundamentals. The systems ideas outlast the systems that use them.