The rotation that rescues a 2-bit KV cache

TL;DR. A transformer’s key cache is dominated by a few outlier channels (30× the median in the layer I measured), and per-token quantization to 2 bits collapses everything else to noise: 0.95 relative reconstruction error. A Hadamard rotation before quantizing spreads those outliers across all channels and is invisible to attention, because it cancels inside the query-key dot product. That one step drops the error to 0.41, and it closes the gap between the key and value caches, which is what lets KVarN quantize both at a uniform 2 bits and still hold reasoning accuracy (AIME24 61.1 → 60.0 on Qwen3-4B). Every number here comes from a 40-line CPU demo on real Qwen2.5-0.5B KV.

Quantize a transformer’s key cache to two bits the obvious way and you don’t get a smaller cache, you get a broken one. On a real key tile from Qwen2.5-0.5B, naive 2-bit quantization lands at 0.95 relative reconstruction error. The model would be reading its own attention history as noise.

There is one trick, and it runs through a line of recent quantization work (QuaRot and SpinQuant for weights and activations, KVarN for the KV cache): rotate the data by a Hadamard matrix before you quantize. On the same tile, that one step takes the error from 0.95 to 0.41, and it does not change a single attention score. I wanted to understand why such a strange-sounding move works, so I measured it end to end on real KV tensors. Everything below comes from a forty-line script that runs on a laptop CPU.

The key cache is mostly a few loud channels

The KV cache is the memory that grows with your context. For every token you keep, you store a key vector and a value vector, in every layer, in every attention head. At long context or high batch it becomes the dominant consumer of GPU memory, which is why people want to quantize it: two bits instead of sixteen is an 8× reduction, and that 8× is often the difference between fitting a workload and not.

The problem is that the key cache is lopsided. A few channels carry far more magnitude than the rest, the “massive activations” that show up across modern LLMs. In the worst layer of the model I measured, the noisiest key channel swung about 30 times wider than the median channel (30× the per-channel standard deviation). The value cache was milder, about 3.9 times. That asymmetry turns out to explain why naive low-bit quantization breaks on the key cache.

Why naive 2-bit is fatal for keys

KV quantization runs per token. Each new token’s key vector is quantized on its own, with one scale computed across that vector’s channels, because tokens arrive one at a time during decoding and you cannot wait for the whole sequence. One scale per row, across all channels.

Now put a 30× channel into that row. The scale has to stretch to cover the outlier, and at two bits, which is four levels total, the rest of the channels collapse onto one or two of those levels. The information in the ordinary channels is gone. That is what 0.95 error means: not a slightly lossy cache but a destroyed one. Four-bit survives the same tile at 0.13, because sixteen levels leave enough resolution for the ordinary channels even with a stretched scale. Two bits do not.

Bar chart comparing relative reconstruction error of the key and value cache under naive 2-bit, rotated 2-bit, variance-normalized 2-bit, and naive 4-bit quantization. — Figure 1. Reconstruction error on a real key and value tile from Qwen2.5-0.5B. Naive per-token 2-bit destroys the key cache (0.95) because of 30× per-channel outliers; a Hadamard rotation cuts it to 0.41. The value cache, with 3.9× outliers, starts lower and gains less. Even rotated, 2-bit keys trail naive 4-bit (0.13).

The Hadamard rotation

The move: before quantizing, multiply each head’s key vectors by a Hadamard matrix $H$ , a square matrix of $\pm 1$ entries scaled by $1/ n$ so it is orthonormal. Quantize the rotated keys. At read time, dequantize and rotate back.

Two properties make this work. It is invisible to attention, and it flattens outliers.

Invisible to attention, because attention scores are dot products of queries and keys. Rotate both keys and queries by the same orthonormal $H$ and the rotation cancels inside the product: $(Q H) (K H)^{⊤} = Q H H^{⊤} K^{⊤} = Q K^{⊤}$ . The scores are identical to the bit; the quantization adds noise, the rotation itself adds none. In practice the rotation is cheap, a fast Hadamard transform that can be partly fused into the projection weights.

Flattens outliers, because a Hadamard transform mixes every input channel into every output channel with equal weight. The single 30× spike gets smeared across all 64 dimensions, no rotated channel dominates its row, the per-token scale becomes reasonable, and the four levels land on real structure instead of one screaming value. Reconstruction error falls from 0.95 to 0.41.

This is the same insight behind QuaRot and SpinQuant for weight and activation quantization. Outliers are concentrated in a few directions rather than spread randomly, and an orthonormal rotation redistributes them without changing the math the model computes.

The variance-normalization step, and its limit

KVarN, the recent paper this rotation-plus-normalization recipe comes from, adds a second step after the rotation: an iterative variance normalization. It alternately rescales the tile so every channel and every token has unit standard deviation, Sinkhorn-style, storing the per-channel and per-token scales as cheap side information. The goal is to hand the quantizer a tile that is as close to uniform as possible.

On my tile it moved the error from 0.41 to 0.40. Real, but small. Most of the win is the rotation; the normalization is a refinement that matters more on some distributions than others. Worth knowing when you decide how much machinery to bolt on.

What the rotation equalizes

Look at the two caches after the rotation. Naive 2-bit had them far apart: 0.95 error on the keys, 0.54 on the values, because the keys carried the bigger outliers (30× against 3.9×). After the rotation they land almost together, 0.41 on the keys and 0.44 on the values. The rotation does not just lower the key error, it erases the gap between keys and values.

That equalization is the point, because it is what makes a single uniform bit-width reasonable. KVarN’s paper runs keys and values both at 2 bits and reports that a Qwen3-4B model holds its reasoning accuracy: AIME24 goes from 61.1 to 60.0, about a point. Without the rotation, the 0.95 key error would make a uniform 2-bit config a non-starter; with it, keys and values sit in the same place and the model tolerates both.

One honest gap, because reconstruction error and task accuracy are not the same number. Even rotated, 2-bit sits at 0.41 against 0.13 for plain 4-bit, so the cache is visibly noisier than a 4-bit cache. The paper’s result says a reasoning model tolerates that noise; that is a fact about the task, not a sign that the cache is accurate. If you want a safety margin, the repo’s shipped vLLM backend defaults to something more conservative than the paper, four bits on keys and two on values (its k4v2 preset). My reconstruction numbers do not demand that split, but they do not argue against a margin either.

Run it yourself

The whole demonstration is one CPU script. It pulls a real KV tile out of a small model, quantizes it several ways, and prints the reconstruction errors plus the figure above. The core is short:

def quant_dequant(x, bits):                 # per-token (per-row) asymmetric RTN
    qmax = 2 ** bits - 1
    xmin, xmax = x.min(1, keepdims=True), x.max(1, keepdims=True)
    scale = np.where((xmax - xmin) == 0, 1.0, (xmax - xmin) / qmax)
    q = np.clip(np.round((x - xmin) / scale), 0, qmax)
    return q * scale + xmin

def hadamard(n):                            # orthonormal n x n, n a power of 2
    H = np.array([[1.0]])
    while H.shape[0] < n:
        H = np.block([[H, H], [H, -H]])
    return H / np.sqrt(n)

H = hadamard(K.shape[1])
naive = quant_dequant(K, 2)                 # 0.95 relative error
rotated = quant_dequant(K @ H, 2) @ H.T     # 0.41 relative error

No GPU, a few seconds to run. The full script adds the variance-normalization step and the real KV extraction with transformers.

Two honest notes

The mechanism is general and well established; the specific implementation I pulled the config from is early. KVarN’s vLLM backend only supports head_dim=128 right now, so it rejects head_dim=256 models like Google’s Gemma-4 at config validation, before any weights load. Its quantized server also has a heavy cold start today, with open throughput issues in its own tracker. The rotation-plus-normalization idea is solid and worth carrying around; treat that particular kernel as a moving target.

The number I could not produce myself is the end-to-end accuracy, whether a fully quantized model holds its reasoning scores. KVarN reports that it does (AIME24 61.1 → 60.0 at their 2-bit config), and the reconstruction numbers here are consistent with that, but I could not get the quantized server to finish an evaluation run to confirm it independently. So the reconstruction mechanism here is mine and measured; the downstream accuracy is theirs and cited.

Status and further reading

The rotation idea predates the KV-cache application and is worth reading at the source.

QuaRot (Ashkboos et al., 2024) — outlier-free 4-bit inference by rotating weights and activations with Hadamard matrices. The clearest statement of the “rotate to kill outliers” idea.
SpinQuant (Liu et al., 2024) — learns the rotation instead of fixing it to Hadamard, and quantifies how much the choice of rotation matters.
KVarN (Müller et al., Huawei, 2026) — applies rotation plus iterative variance normalization to the KV cache; the paper runs uniform 2-bit, the shipped vLLM backend defaults to the more conservative k4v2 (4-bit key, 2-bit value). Code.
KIVI (Liu et al., 2024) — the per-channel-keys, per-token-values asymmetry, derived from the same outlier observation, without rotation.