Case Study: Claude's KV Cache
Concept. An LLM agent reuses a long prefix across turns: the system prompt, tool schemas, and conversation history. The KV cache stores the attention keys and values already computed for those tokens. Prompt caching reuses that state on later turns. Paging, quantization, sharing, and offload reduce its memory cost.
Intuition. A Claude agent may resend thousands of unchanged tokens on every turn. Without caching, the model processes that prefix again. Anthropic charges 25% more to write a five-minute cache entry and 90% less to read it, so repeated prefixes quickly become cheaper.
Every turn is mostly the same prefix
A request to an agent often contains a long repeated prefix. The system prompt and tool schemas remain fixed while the conversation grows at the end. Recomputing that prefix on every turn adds substantial cost and latency.
Figure: The KV cache across agent turnsTop: one request is a bar split into system prompt, tool schemas, and history (the slate prefix, reused every turn) plus a small new message that must be computed; the prefix grows each turn but is reused, not recomputed. Bottom: a cost chart over four turns. The grey bars, recompute the prefix every turn, rise turn after turn; the slate bars, prompt caching, pay once to write the cache then stay low as each turn only reads the prefix and computes the new message. Color key: slate is prompt caching; grey is recomputing the prefix every turn.
Figure 1. Without caching, the model recomputes the growing prefix on every turn. Prompt caching reuses the prefix and processes the new suffix, so later turns stay relatively inexpensive. The cost still grows as the uncached suffix grows.
The model avoids recomputation by saving the attention key and value vectors produced for each token. That is where the name KV cache comes from. A serving system can then index this cached state by its prompt prefix. As a systems analogy, the prefix acts as the lookup key and the saved tensors act as the value.
Figure: What the KV cache storesThe processed prefix tokens appear in grey. For each token, the cache stores an attention key vector and value vector at every model layer, shown in slate. Reusing these tensors avoids processing the same prefix again.
Figure 2. The cache stores an attention key vector and value vector for every processed token at every model layer. Reusing these tensors avoids processing the same prefix again.
That is prompt caching. Anthropic launched its version in August 2024. According to its prompt-caching documentation, a five-minute cache write costs 25% more than normal input, a hit costs 10% of the normal input price, and each hit refreshes the five-minute lifetime. For an agent that repeatedly sends the same ten-thousand-token prefix, every cache hit cuts the input cost of that prefix by 90%.
Paging: fit many caches in one GPU
Once you cache prefixes, a server is juggling many chats at once, each with a KV cache that grows a token at a time, all sharing one GPU's memory. The move that makes that work is paging.
Figure: PagedAttention pages the KV cache into fixed-size blocks instead of reserving a chunk per requestR1, R2, and R3 are three concurrent chats sharing one GPU, each with a KV cache of computed keys and values that grows one token at a time. Two panels. Left, the naive way: each request reserves one contiguous chunk sized to the maximum length, but only a few cells hold real data (slate) and the rest are reserved and never used (dashed), wasting 60 to 80 percent of GPU memory. Right, PagedAttention: each request's real tokens are split into fixed-size blocks packed together with no gaps, and a page table maps each request to its blocks, which can sit anywhere, for example request one lives in blocks 0 and 2, wasting under 4 percent. Color key: slate is a block of real KV data; a dashed outline is memory reserved but never used.
Figure 3. In the systems evaluated by the vLLM authors, fragmentation and over-reservation wasted 60 to 80 percent of KV-cache memory. PagedAttention reduced that waste to under 4 percent by allocating fixed-size blocks on demand and mapping each request to blocks that need not be adjacent.
PagedAttention (the engine behind vLLM) stores each chat's cache in fixed-size pages of GPU memory instead of one contiguous chunk, with a page table mapping logical positions to scattered GPU memory. That is the fixed-size-page idea from Module 2 paging, used here to pack one scarce memory tier, not to move data across the disk-to-RAM gap. Once you decide to cache prefixes, this is the mechanism that makes it fit.
Reuse the Module 2 toolkit
Once the cache occupies scarce GPU memory, the same Module 2 choices apply.
Quantize it. Store the keys and values at lower precision. This uses the same quantization trade-off as TurboQuant: spend less memory while preserving enough signal.
Figure: An agent's two memories, one quantization leverTwo memory tiers with one shared lever between them. On the left, long-term memory is the vector or RAG store, shrunk by TurboQuant from Case Study 2.4. On the right, working memory is the KV cache of the current context, shrunk by KV-cache quantization. The slate lever in the middle, quantize (drop precision, keep the signal), feeds both, because both are dense-vector piles too large at full precision. Color key: slate is the two memory tiers and the shared quantize lever; grey is the supporting detail.
Figure 4. Quantization can shrink both an agent's vector store and its KV cache. The two systems store different vectors, but both trade numerical precision for memory.
Precision is one lever. Prefix sharing resembles deduplication: the server stores a shared block once and lets multiple requests reference it. When memory fills, an eviction policy removes cold blocks or selected tokens. These systems apply the same Module 2 choices to attention state: compress repeated data, trade precision for space, and keep the hottest data nearby.
Offload it, and price it. When even paging cannot fit everything, push cold blocks from GPU to CPU to disk, the storage hierarchy. And the whole decision to cache rather than recompute is a cost-model call, a cheap hit weighed against an expensive recompute. Line all four up, and every one is a Module 2 idea:
Figure: Every KV-cache memory trick is a Module 2 ideaA two-column map. Each row pairs an agent KV-cache technique on the left (slate) with the Module 2 concept it reuses on the right: PagedAttention with paging, prefix and prompt caching with the cache-hit-versus-recompute cost model, KV-cache quantization with the quantization of Case Study 2.4 (TurboQuant), and GPU-to-CPU-to-disk offload with the storage hierarchy. Color key: slate is the LLM-serving technique; grey is the Module 2 concept it reuses.
Figure 5. PagedAttention, prompt caching, quantization, and offload reuse four Module 2 ideas: paging, cache cost models, compression, and the storage hierarchy.
On one machine, cached state can move from GPU memory to CPU RAM and then to NVMe storage. Across a fleet, systems such as LMCache and Mooncake pool KV-cache state across workers and storage tiers. A worker can reuse state that another worker already computed instead of processing the same prefix again.
Takeaway: Prefix reuse turns repeated attention work into a cache hit. Paging, quantization, sharing, eviction, and offload determine how much cached state fits near the GPU.