Case Study 2.4: Compressing a Billion Embeddings

Concept. A float32 vector index stores more precision than search needs. Naive rounding fails because most variance lies along a few directions. TurboQuant rotates the vectors so their coordinates follow the same predictable distribution in high dimensions. One scalar quantizer can then compress every coordinate without a trained codebook, shrinking the index by 8 to 32 times.

Intuition. The Bay Area varies mostly north to south and much less east to west. Coarse axis-aligned rounding can collapse San Francisco and Berkeley into one cell. Rotating the coordinate system preserves every point and distance while spreading the east-west difference across multiple coordinates. A few bits per coordinate can then preserve their separation.

Case Study 2.4 Reading Time: 4 mins

A 100,000-document demo becomes a 60 GB index

A semantic-search demo runs instantly on 100,000 documents. In production, the corpus reaches 10 million documents and the raw float32 vector index no longer fits in memory. The memory requirement breaks the design.

Two RAM boxes with the same capacity line. Left: 10 million float32 embeddings stack to about 60 GB and overflow the box. Right: quantized 8 to 32 times, the same index drops under the line and fits.

Figure 1. A full-precision embedding is big. Most have about a thousand dimensions or more. OpenAI's smaller embedding model is 1,536-D and the large one is 3,072-D. At 1,536-D float32 that is about 6 KB per vector, so 10 million vectors take ~60 GB in RAM for fast search. The large model uses double. Quantizing each coordinate to a few bits shrinks the index 8 to 32 times and brings it under the RAM limit.

Most of the information hides in a few directions

Quantization rounds each coordinate to a few bits. Embedding variance is uneven: a few directions carry most of the signal.

Use two dimensions to see the problem. The Bay Area runs mainly north to south, so most location variance lies on that axis. Coarse rounding spends most of its resolution there and little from east to west. San Francisco and Berkeley can therefore land in the same cell even though they are not close in east-west.

Google's TurboQuant (2026) rotates each vector before rounding. Equivalently, rotate the grid while holding the points fixed. The points and distances stay unchanged, but the variance spreads across both axes. San Francisco and Berkeley now land in different cells.

One panel. San Francisco, Berkeley and Stanford stay at the same coordinates in every frame while the grid behind them rotates. SF and Berkeley begin in one cell and end in different cells after a spin of 58 degrees; Stanford, much further away, is unaffected.

Figure 2. Eight frames from 0° to 58°. The three cities sit at identical coordinates in every frame, so you can check for yourself that only the grid moved. San Francisco and Berkeley begin in one cell and end in different ones. Stanford, much further off, is untouched.

256 embedding coordinates drawn as bars beside a histogram of how many fall into each of the 16 levels a 4-bit quantiser offers. Before the rotation only 5 of the 16 levels are ever used and one holds most of the coordinates. After a random rotation the coordinates spread evenly and 15 levels are in use. Storage is 4 bits per coordinate either way.

Figure 2b. The same idea at 256 dimensions. A 4-bit budget is 16 levels. Before the rotation the coordinates are so uneven that only 5 levels are ever used, which is 2.3 bits of information in the 4 bits you are paying to store. After it, 15 levels are in play at 3.9 bits. Storage is 4 bits per coordinate either way; only the useful fraction changes.

The map supplies only a two-dimensional intuition. Real embeddings have hundreds or thousands of dimensions. A single rotation is ineffective in two dimensions. Across hundreds of dimensions, the rotations help.

Drop the precision

With the variation spread more evenly, we then quantize (round) each coordinate to fewer bits. Think of this step like how we can make a high-resolution image to a lower-res image and make it smaller.

One coordinate stored two ways: the exact value 0.4327 as a 32-bit float on the left, an arrow labelled keep fewer bits, and the same coordinate at 2 bits on the right, one of four buckets, reading 0.4. A strip below notes the distance barely moves and the index is sixteen times smaller with no codebook.

Figure 3. Scalar quantization of one coordinate. 0.4327 stored in 32 bits maps to a 2-bit bucket 10 and decodes to about 0.4. One coordinate is now rough, but a distance adds up hundreds of them, and the rounding errors mostly cancel out. The rotation first spreads the values evenly, so one simple quantizer fits every coordinate and every dataset, with no training. Two bits gives 16× compression, and one bit gives 32×.

Like LSH, the rotation is random and does not inspect the dataset. LSH uses randomness to bucket similar items. TurboQuant uses it to spread information before rounding. Google Research calls the method TurboQuant (ICLR 2026): spin, then round.

Try it yourself?

The cost in recall, the fraction of exact nearest neighbours returned, depends on the data. Measure it on your own corpus. FAISS implements this recipe, a random rotation followed by scalar quantization, in a few lines, making it a one-afternoon experiment. You oculd experiment with the same tradeoff in Project 2's NanoMem Systems track, where an AI agent's memory engine must fit within a tight memory budget.

Takeaway: Spin, then round. Rotation spreads the embedding's information across coordinates, so a shared low-bit quantizer can shrink the index by 8 to 32 times without training a codebook. Like LSH, TurboQuant uses randomness to avoid learning from the dataset, but for a different purpose.