Case Study 2.4: Compressing a Billion Embeddings
Concept. A float32 embedding index is mostly wasted precision, but you cannot just round it away: an embedding has hundreds of dimensions, and most of its information is packed into a few of them. TurboQuant spreads that information evenly with one random rotation, and only then drops precision from each number, shrinking the index 8 to 32 times with little loss, on any data, with no codebook to train.
Intuition. The Bay Area runs north-south, and most of the variation runs that way; the east-west cross axis barely moves. So two places that differ mainly east-west, San Francisco and Berkeley, blur together when you describe them coarsely. A random rotation turns the axes so that hidden difference spreads across them, without moving a point or changing a distance, and then a few bits per coordinate are enough.
The demo worked. Production did not.
A team ships semantic search. On a laptop with 100,000 documents it is instant, and the demo lands. Then it goes to production and the corpus is 10 million documents. The embeddings are OpenAI's 1,536-dimensional vectors: 1,536 × 4 bytes is about 6 KB per vector, so 10 million of them is roughly 60 GB of float32 that has to sit in RAM for search to be fast. The service that flew in the demo now will not fit on the box, and the memory bill is the whole infrastructure budget.
Every team that ships embedding search hits this the moment the corpus outgrows the demo. What breaks is not the algorithm, it is that the raw float32 index no longer fits in memory.
Most of the information hides in a few directions
To shrink the index you round each coordinate to a few bits. The catch is which coordinates, because an embedding's information is lopsided: most of its variation runs along a few directions and the rest of the coordinates barely move.
Picture the Bay Area. It runs north-south, the long way, and most of the variation in where things sit runs along that north-south axis; the east-west cross axis barely moves. Round each axis coarsely and you spend all your resolution on north-south and have almost none left for east-west. So San Francisco and Berkeley, which sit across the bay from each other and differ mainly east-west, round to the same description even though they are genuinely apart. Their difference fell in the direction the rounding threw away.
TurboQuant applies a random rotation to each vector before rounding. Picture it as turning the grid instead, which is the same thing: the points do not move and no distance changes, only the axes turn. But against the turned grid the variation no longer piles onto one axis, so San Francisco and Berkeley fall in different cells again. It needs no look at your data.
Figure 1. Why the spin helps, in 2-D, with the points held fixed. Most variation runs north-south, so the axis-aligned grid (left) resolves north-south but barely splits east-west, and San Francisco and Berkeley, which differ east-west, fall in one cell. Right: a random spin turns the grid, not the points. SF and Berkeley sit exactly where they were, the same distance apart, but the turned grid now falls between them, so they land in different cells. The spin re-aims the axes and changes no distance.
This map is only a 2-D picture, for intuition; it can make the trick look geographic. It is not. Real embeddings have hundreds of dimensions, and the rotation turns all of them at once. And the spreading is really a high-dimensional effect: in high dimensions a random rotation makes every coordinate fall into the same bell-shaped spread, whatever the data, so one fixed grid rounds them all well. A single spin in two dimensions is hit or miss; across hundreds of dimensions it is reliable. The picture hints at it; the math only works because the space is big.
Then just drop the precision
With the variation spread evenly, the compression itself is the easy part: round each coordinate to fewer bits. A value stored as 0.4327 becomes 0.43, then 0.4, then a 2-bit bucket. Each step roughly halves the bits and barely moves the distances, because no single coordinate is carrying the whole answer. That is the 8 to 32 times shrink, and because the rotation did the hard part with no training, the recipe works on any data with no codebook to build.
Like LSH, the rotation is random and needs no look at your data; but where LSH uses randomness to bucket similar items, TurboQuant uses it to spread the information so rounding is cheap. That is TurboQuant (Google Research, ICLR 2026): spin, then round.
Try it yourself
What the rounding costs in recall depends on your data, so the only way to know is to measure it on your own corpus. FAISS ships this exact recipe, a random rotation followed by a scalar quantizer, in a few lines, so it is a one-afternoon experiment. You will get a reason to: later in the course you build a memory engine for an AI agent (Project 2, NanoMem, Systems track) where a tight memory budget makes this trade a real lever.
Takeaway: A float32 index is mostly wasted precision, but you cannot round it away blindly, because an embedding's information hides in a few directions. TurboQuant's 2026 move is to spin each vector by a random angle first, spreading the information evenly, so rounding to a few bits shrinks the index 8 to 32 times on any data, with no codebook to train.