Google’s TurboQuant: Squeezing AI Models Without Breaking Them

Google Research just dropped something interesting at ICLR 2026: TurboQuant, a compression method that claims to squeeze large language models and vector search engines down to a fraction of their size without sacrificing accuracy. They’ve got the benchmarks to back it up, and honestly, this is one of those rare papers where the math actually delivers on the promise.

Let me break down why this matters. Anyone who’s worked with modern AI knows that high-dimensional vectors are the backbone of everything from image recognition to semantic search. The problem? They’re memory hogs. The key-value cache in transformer models can balloon to ridiculous sizes, and vector databases aren’t much better. Traditional quantization helps, but it comes with its own baggage: you end up storing quantization constants in full precision, which adds 1-2 bits per number. That overhead adds up fast, especially when you’re dealing with billions of parameters.

TurboQuant attacks this problem from two angles. First, it uses a method called PolarQuant, which rotates the data vectors randomly to simplify their geometry. Think of it like turning a messy pile of sticks into a neat bundle—you can then apply a standard quantizer to each part individually without losing the overall shape. This first pass does the heavy lifting, capturing the core essence of the vector with most of the compression bits.

But here’s the clever part: instead of stopping there, TurboQuant uses a second stage called Quantized Johnson-Lindenstrauss (QJL) to mop up the residual errors. QJL takes the tiny leftover noise from the first pass and compresses it down to a single bit using a mathematical trick called the Johnson-Lindenstrauss Transform. This preserves the distances between data points while requiring zero memory overhead for the quantization constants. The result is an attention score that’s practically as accurate as the original, but with way less memory.

I’ve seen a lot of compression schemes come and go, and they usually fall into two camps: aggressive but lossy, or conservative but memory-heavy. TurboQuant manages to thread the needle. In their tests, they showed that you can compress key-value caches down to 2 bits per number without any measurable drop in model performance. That’s roughly a 4x reduction compared to standard 8-bit quantization, and way better than the 16-bit float most models ship with.

PolarQuant deserves a bit more attention too. Instead of representing vectors in the usual Cartesian coordinates (x, y, z, etc.), it converts them into polar coordinates—angles and magnitudes. This shift in perspective naturally eliminates the need for those pesky quantization constants because the angular representation is inherently more stable under compression. It’s one of those ideas that seems obvious in hindsight but nobody had quite nailed down before.

The implications for vector search are huge. If you’re running a retrieval-augmented generation pipeline or a large-scale similarity search engine, memory is often the bottleneck, not compute. TurboQuant lets you pack more vectors into the same RAM, which means faster lookups and lower cloud bills. And since the accuracy loss is negligible, you don’t have to retrain your models or tune your search thresholds.

Of course, nothing’s perfect. The random rotation step in PolarQuant adds a small computational overhead upfront, though Google claims it’s amortized over repeated queries. And the QJL trick works best for attention scores that are already somewhat sparse—if your model has uniformly distributed attention, you might not see the same gains. But for the vast majority of transformer-based models, this should be a net win.

I’m also curious to see how well this generalizes to other architectures. Google tested it on their internal models, but the real test will be whether it works out of the box with open-source stuff like Llama or Mistral. The algorithms are theoretically sound, but engineering matters too.

Still, this is a solid step forward. If you’re building anything that relies on large-scale vector operations—search, recommendation, or generative AI—keep an eye on TurboQuant. The paper’s at ICLR 2026, and the code should follow soon. I’ll be playing with it as soon as it drops.