Case Study 2.4: Compressing a Billion Embeddings
Concept. A float32 embedding index stores more precision than search needs. Naive rounding fails because the embedding's variance concentrates in a small number of directions. TurboQuant applies a random rotation so each coordinate has the same predictable distribution in high dimensions, independent of the original data. After the rotation, one scalar quantization rule applies to every coordinate. The index shrinks by 8 to 32 times without training a codebook.
Intuition. The Bay Area varies mostly north-south, with little east-west variation. Coarse rounding on axis-aligned coordinates preserves north-south detail and collapses east-west differences, so San Francisco and Berkeley can round to the same description. A random rotation changes the coordinate system without changing points or distances, so the east-west difference spreads across multiple coordinates. A few bits per coordinate then preserve separation.
The demo worked. Production did not.
A team ships semantic search. On a laptop with 100,000 documents it is instant, and the demo lands. Then it goes to production and the corpus is 10 million documents, and the raw float32 index no longer fits in memory. What breaks is not the algorithm; it is the memory.
Figure 1. Float32 storage size. Embeddings often have about a thousand dimensions or more. OpenAI's smaller embedding model is 1,536-D and the large one is 3,072-D. At 1,536-D float32 that is about 6 KB per vector, so 10 million vectors take ~60 GB in RAM for fast search. The large model uses double. Quantizing each coordinate to a few bits shrinks the index 8 to 32 times and brings it under the RAM limit.
Most of the information hides in a few directions
To shrink the index you round each coordinate to a few bits. The catch is which coordinates, because an embedding's information is lopsided: most of its variation runs along a few directions and the rest of the coordinates barely move.
An embedding has hundreds or thousands of dimensions, which are hard to reason about, so take the smallest example that shows the idea: two dimensions. Picture the Bay Area. It runs north-south, the long way, and most of the variation in where things sit runs along that north-south axis; the east-west cross axis barely moves. Round each axis coarsely and you spend all your resolution on north-south and have almost none left for east-west. So San Francisco and Berkeley, which sit across the bay from each other and differ mainly east-west, round to the same description even though they are genuinely apart. Their difference fell in the direction the rounding threw away.
TurboQuant applies a random rotation to each vector before rounding. Picture it as turning the grid instead, which is the same thing: the points do not move and no distance changes, only the axes turn. But against the turned grid the variation no longer piles onto one axis, so San Francisco and Berkeley fall in different cells again.
Figure 2. Eight frames from 0° to 58°. The three cities sit at identical coordinates in every frame, so you can check for yourself that only the grid moved. San Francisco and Berkeley begin in one cell and end in different ones. Stanford, much further off, is untouched.
Figure 2b. The same idea at 256 dimensions, where it earns its keep. A 4-bit budget is 16 levels. Before the rotation the coordinates are so uneven that only 5 levels are ever used, which is 2.3 bits of information in the 4 bits you are paying to store. After it, 15 levels are in play at 3.9 bits. Storage is 4 bits per coordinate either way; only the useful fraction changes.
This map is only a 2-D picture, for intuition; it can make the trick look geographic. It is not. Real embeddings have hundreds or thousands of dimensions and the rotation turns all of them at once, which is also why it can be trusted: a single spin in two dimensions is hit or miss, but across hundreds of dimensions every coordinate reliably lands on that same spread. The picture hints at it; the math only works because the space is big.
Then just drop the precision
With the variation spread evenly, the compression itself is the easy part: round each coordinate to fewer bits, because no single coordinate is carrying the whole answer.
Figure 3. Scalar quantization of one coordinate. 0.4327 stored in 32 bits maps to a 2-bit bucket 10 and decodes to about 0.4. One coordinate contributes little signal, but a distance aggregates hundreds of coordinates. In high dimensions, quantization errors across coordinates behave close to independent and tend to cancel. The rotation enforces a shared distribution across coordinates, so the same quantizer works across coordinates and across datasets without training. Two bits gives 16× compression, and one bit gives 32×.
Like LSH, the rotation is random and needs no look at your data; but where LSH uses randomness to bucket similar items, TurboQuant uses it to spread the information so rounding is cheap. That is TurboQuant (Google Research, ICLR 2026): spin, then round.
Try it yourself
What the rounding costs in recall depends on your data, so the only way to know is to measure it on your own corpus. FAISS ships this exact recipe, a random rotation followed by a scalar quantizer, in a few lines, so it is a one-afternoon experiment. You will get a reason to: later in the course you build a memory engine for an AI agent (Project 2, NanoMem, Systems track) where a tight memory budget makes this trade a real lever.
Takeaway: Spin, then round. The order is the whole trick: rounding first wrecks the few coordinates that carry the meaning, while spinning first puts every coordinate on the same footing, so a few bits each is enough. It is randomness bought in exchange for never having to look at your data, the same bargain LSH makes for a different purpose.