TurboQuant: An algorithm which broke the stock market
When TurboQuant first came out, it impacted stocks of companies like Micron, SanDisk, and Western Digital, dropping them by more than 7%. Let’s dive into this strange algorithm and understand it better.
First, let’s understand quantization. When it comes to LLMs, quantization usually means reducing the precision from a 32-bit representation of floating numbers to, let’s say, 16-bit precision or lesser. This reduces your GPU footprint and model size, meaning you can run a bigger model on a smaller GPU. Usually, this is done to the model’s weights. But we can actually do a similar thing to the KV Cache, which is the “context memory” of the model.
The Problem with Compression
Let’s say you have 2 files, each containing 10 million characters, and both are exactly 9.53MB. The first file is filled with just the letter ‘A’, while the second file is completely random characters. If I try to zip both of them, the first one compresses down to around 9 kilobytes — nearly a 1,000x compression rate. But the second one, because it’s so randomized, only compresses down to 7.91MB — barely a 1.2x compression rate.
There is a huge variance in compression depending on what’s actually contained in the file. But carrying this concept over to TurboQuant isn’t a perfect mental model. Standard zip files are lossless, meaning there’s a guaranteed preservation of information. In an LLM, the “data mess” is the KV Cache, and it is the absolute number one reason your GPU runs out of memory (OOM) when you try to have a long conversation with an AI.
To really understand TurboQuant, a better mental model is compressing a high-resolution photograph. If you take a vibrant landscape photo and heavily compress it by forcing the entire image to be represented by a “codebook” of only two colors, the file size drops dramatically. However, you lose crucial features — perhaps the sun in the background completely disappears, and the sky becomes a blocky mess. But, if you compress it using 64 colors instead, the file size still shrinks significantly, yet the reconstructed image looks practically identical to the original. You lost some raw data, but the core “features” remained intact.
TurboQuant falls under this exact category: lossy compression. It achieves near-minimum distortion while reducing the dimension of the KV cache to a much lower precision, cleverly discarding data without losing the overall “picture.”
Why Not Just Throw Data Away?
Many algorithms have been developed before to solve this KV Cache quantization problem, but most of them result in tossing out valuable information. Other methods (like Snap KV and Pyramid KV) use “pruning” or “selecting,” where you essentially kick irrelevant parts of the KV cache out. Given below are some examples of these algorithms and loss of data.
TurboQuant is fundamentally different. It doesn’t prune data; it preserves the entire attention but stores it at a much lower precision.
So, the million-dollar question is: how does the theory behind TurboQuant actually work?
Step 1: Random Rotation Matrix & Chaos
Context windows are unpredictable. One person might fill their prompt with perfectly structured HTML, while another pastes in dense, chaotic legal documents. Because of this, the data in the context window can be incredibly “spiky” and jumping all over the place, making it really hard to come up with a standard codebook to compress it.
To solve this, TurboQuant takes the input data and multiplies it by a random rotation matrix. Essentially, this dissipates the concentrated energy from those huge spikes and spreads it out evenly.
You might think it’s completely counterintuitive to jumble up known data into random noise. However, thanks to the Central Limit Theorem, this randomized data naturally converges into a highly predictable Gaussian distribution (a bell curve). By bringing order to chaos, TurboQuant ensures the data always looks relatively similar in distribution, regardless of what the original prompt was.
Step 2: Precomputing the Codebook
Because the data has now converged into this reliable Gaussian distribution, the researchers could mathematically precalculate the optimal “codebook” for it using the Max-Lloyd algorithm.
They do this for 1-bit, 2-bit, 3-bit widths, and so on, and store the results into a lookup table just once. Now, incoming KV cache data can be effortlessly and efficiently reduced to a lower precision using this exact table.
Step 3: Fixing the Error with QJL
Of course, snapping all of these values into a pre-computed codebook introduces a bit of bias, or “mean squared error”.
To deal with this residual error, TurboQuant uses the QJL algorithm (based on the Johnson-Lindenstrauss lemma). While step one relies entirely on snapping to the codebooks, step two takes a step back, takes the residual error left behind, and applies QJL to preserve the inner product error — which is critical for maintaining accurate attention scores in the LLM.
The Impact
The overall impact of this breakthrough is massive. Because you can reduce the memory requirement of the KV cache by up to 6x — compressing data from 16-bit down to just 3 bits — we can theoretically expand the context window drastically on the exact same graphics cards. Alternatively, we can keep the context window the same length but serve vastly more users concurrently.
This means tech companies might suddenly need two to three times fewer graphics cards for AI inference, which explains exactly why hardware providers took such a sudden stock hit when this paper dropped!