What attention stores, and what DeepSeek changed

TL;DR. When a transformer generates text, it stores a key and a value for every token it has already read, so it does not recompute them at each step. That store is the KV cache, and at long context it fills GPU memory before the model weights do. Multi-head Latent Attention (MLA), introduced in DeepSeek-V2 , stores one short latent vector per token instead of full per-head keys and values, and folds the reconstruction step into matrices that are applied once when the model loads. The cache shrinks by roughly an order of magnitude, and the attention scores come out identical to the uncompressed version. This post builds that claim on one small example you can check by hand.

The four numbers attention works with

Take three tokens. Give each one a vector of length four; call the three vectors $h_{1}, h_{2}, h_{3}$ . Everything below is built from those numbers.

A transformer layer first turns each token vector into three new vectors, each by multiplying it by a learned matrix:

q_{t} = W^{Q} h_{t}, k_{t} = W^{K} h_{t}, v_{t} = W^{V} h_{t}

These are the query, key, and value for token $t$ . The query is what the current token is looking for, the key is what each token offers when it is being looked at, and the value is what a token contributes once it has been selected. For this example each of $W^{Q}, W^{K}, W^{V}$ maps length-4 vectors to length-3 vectors, so $q_{t}, k_{t}, v_{t}$ each have three components.

With concrete numbers, the three keys and values come out as

K = 211012101, V = 011102110

one row per token. Now ask what token 3 attends to. Its query is $q_{3} = [2, 1, 0]$ . The attention score between token 3 and an earlier token $j$ is the dot product of $q_{3}$ with that token’s key, divided by $d_{h}$ where $d_{h} = 3$ is the per-vector length:

score (3, j) = \frac{q _{3} \cdot k _{j}}{d _{h}}

The division keeps the numbers from growing with vector length; it does not change which token wins. For token 3 the three scores are $[2.309, 1.732, 2.309]$ . Passing those through softmax Softmax turns a list of numbers into positive weights that sum to 1, with larger inputs getting exponentially larger weights. The mechanics are walked through in the streaming-softmax post. gives weights $[0.390, 0.219, 0.390]$ , and the output for token 3 is the weighted sum of the value rows:

out_{3} = 0.390 v_{1} + 0.219 v_{2} + 0.390 v_{3} = [0.610, 1.171, 0.610]

That is one attention head: scores from queries against keys, softmax, weighted sum of values.

What the cache holds, and why it grows

When the model generates the next token, it computes one new query and compares it against the keys of every token so far. Those earlier keys and values do not depend on the new token, so recomputing them every step would be wasted work. The model stores them. That store is the KV cache.

Real layers do not run one head. They run $n_{h}$ heads in parallel, each with its own $W^{Q}, W^{K}, W^{V}$ , and concatenate the results. So per token, per layer, the cache holds $n_{h}$ keys and $n_{h}$ values, each of length $d_{h}$ :

MHA cache per token per layer = 2 n_{h} d_{h}

This is standard Multi-Head Attention (MHA). The factor of two is keys plus values. Two earlier variants attack this number by sharing: Multi-Query Attention keeps one key and one value shared across all heads, and Grouped-Query Attention keeps one per group of heads. Both store fewer heads; both give up some of the per-head distinction that made multi-head attention work.

The number matters because it is multiplied by everything large. DeepSeek-V2 has $d_{h} = 128$ , $n_{h} = 128$ , and 60 layers. One token of MHA cache is $2 \times 128 \times 128 = 32, 768$ numbers per layer, so across all layers a single token costs about two million numbers. At a 128K-token context that is hundreds of gigabytes for one sequence in fp16, more than a single accelerator holds before the model weights are loaded at all. Drag the context length and watch where each variant crosses the memory budget:

Live · KV cache by attention variant (DeepSeek-V2 dimensions, fp16, one sequence)

context length 128K tokensGPU budget

At 128K tokens, one sequence of MHA cache needs 480 GB and overflows the 80 GB budget. MLA holds the same context in 8.44 GB, a 57× difference.

Per-token cache counts come straight from the formulas: MHA 2·n_h·d_h, GQA 2·n_g·d_h, MQA 2·d_h, MLA d_c + d_R. Bars are log-scaled because the spread is two orders of magnitude.

Cache a latent, not the keys

MLA changes one step. Before computing keys and values, it projects the token down into a short shared vector, the latent, and caches only that. Writing $c_{t}^{K V}$ for the latent of token $t$ :

c_{t}^{K V} = W^{D K V} h_{t}, k_{t}^{C} = W^{U K} c_{t}^{K V}, v_{t}^{C} = W^{U V} c_{t}^{K V}

The down-projection $W^{D K V}$ maps the token to a latent of length $d_{c}$ , much smaller than $n_{h} d_{h}$ . The up-projections $W^{U K}, W^{U V}$ rebuild full keys and values from that latent. The point is what gets stored: only $c_{t}^{K V}$ , of length $d_{c}$ , sits in the cache. The full keys and values are reconstructed when needed.

Stated plainly, that trade looks bad: the cache is smaller, but now every attention step has to rebuild the keys and values from the latent before it can do anything, which is extra compute. The next section is why that extra step never actually runs.

Two paths, one score

The score that attention needs is $q_{t} \cdot k_{j}$ , and for MLA the key is $k_{j}^{C} = W^{U K} c_{j}^{K V}$ . Substitute it in:

q_{t} \cdot k_{j}^{C} = q_{t}^{⊤} W^{U K} c_{j}^{K V} = ((W^{U K})^{⊤} q_{t}) \cdot c_{j}^{K V}

The middle and right expressions are the same product associated two ways. On the left, you build the full key $W^{U K} c_{j}^{K V}$ and dot the query against it. On the right, you fold $(W^{U K})^{⊤}$ into the query once, producing a short vector of length $d_{c}$ , then dot that straight against the cached latent. The key is never formed. Because $W^{U K}$ is fixed after training, the fold can be done a single time when the model loads, and absorbed into the query projection. DeepSeek-V2 says it directly: $W^{U K}$ can be absorbed into $W^{Q}$ , and $W^{U V}$ into the output projection $W^{O}$ , so keys and values are never computed out for attention.

Drag the components of the incoming query. The left box reconstructs the key and dots; the right box folds and dots the latent. The two scores track each other exactly, for every query:

Live · the same score, computed two ways

incoming query qₜ = [ 0.5, -0.4, 0.8, 0.2 ]

naive · reconstruct the key

k = W_UK · c (form the full key)

score = qₜ · k

0.239700

absorb · never form the key

q′ = W_UKᵀ · qₜ (fold once at load)

score = q′ · c (dot the cached latent)

0.239700

difference |naive − absorb| = 2.8e-17. Equal to floating-point precision, for any query you drag to.

The cached latent c is held fixed; only the query moves. The difference between the reconstruct-the-key path and the fold-into-the-query path stays at floating-point precision, because both compute the same product qᵀ W_UK c.

This is the heart of MLA. The compression does not cost compute at inference, because the decompression is algebraically absorbed elsewhere. What changes is what the cache stores, not what attention computes.

The rotary complication

One thing breaks the clean fold: position information. Modern transformers encode position with Rotary Position Embedding (RoPE), which rotates each query and key by an angle that depends on its position before the dot product. The rotation for token $j$ sits between $W^{Q}$ and $W^{U K}$ in the product. That rotation matrix does not commute with $W^{U K}$ , so you cannot slide $W^{U K}$ past the rotation to merge it with $W^{Q}$ into one constant. If MLA applied RoPE to the reconstructed keys, the absorb trick would stop working and the keys would have to be rebuilt for every token at every step.

DeepSeek-V2 resolves this by splitting the key in two. Most of each query and key carries no position information and stays absorbable exactly as above. A small extra slice of dimension $d_{R}$ per head carries RoPE and is computed separately, with a single shared rotary key cached for all heads:

q_{t} = [q_{t}^{C}; q_{t}^{R}], k_{t} = [k_{t}^{C}; k_{t}^{R}]

The content halves $q^{C}, k^{C}$ go through the absorb path; the rotary halves $q^{R}, k^{R}$ are dotted directly and added to the score. The cache now holds two things per token: the latent of length $d_{c}$ and one shared rotary key of length $d_{R}$ . The interactive above shows only the content term, which is where the absorb trick lives; the rotary term is added on top and is identical in both paths.

MLA cache per token per layer = d_{c} + d_{R}

The cache table

Put the four mechanisms side by side. The per-token-per-layer counts are the formulas from the DeepSeek-V2 comparison, with a small illustrative configuration ( $d_{h} = 4$ , $n_{h} = 4$ , $n_{g} = 2$ groups, $d_{c} = 3$ , $d_{R} = 2$ ) and DeepSeek-V2’s own dimensions ( $d_{h} = 128$ , $n_{h} = 128$ , $d_{c} = 512$ , $d_{R} = 64$ ):

Mechanism	cache per token per layer	small config	DeepSeek-V2
MHA	$2 n_{h} d_{h}$	32	32,768
GQA	$2 n_{g} d_{h}$	16	2,048 (8 groups)
MQA	$2 d_{h}$	8	256
MLA	$d_{c} + d_{R}$	5	576

DeepSeek-V2 sets $d_{c} = 4 d_{h}$ and $d_{R} = d_{h} /2$ , so MLA’s cost is $(4 d_{h} + d_{h} /2) = \frac{9}{2} d_{h}$ per token per layer. Against MHA’s $2 n_{h} d_{h}$ , the ratio is

\frac{d _{c} + d _{R}}{2 n _{h} d _{h}} = \frac{9/2}{2 n _{h}} = \frac{9}{4 n _{h}}

so at $n_{h} = 128$ heads, MLA’s cache is about $9/512$ of MHA’s, under two percent. The paper reports a 93.3% reduction against its own 67B MHA baseline ; the difference between that figure and the ratio here is the baseline, not the mechanism.

The reason this matters is that it does not cost quality. MQA and GQA buy their smaller caches by collapsing heads, and the DeepSeek-V2 ablations report MLA scoring above MHA on their benchmarks while caching about as little as GQA with 2.25 groups. That combination, small cache without the usual quality giveback, is why MLA spread from DeepSeek-V2 to V3, and to later models that adopted it. The cost is real but narrow: the projection matrices add parameters and the rotary split adds bookkeeping, and serving MLA is more involved than serving MHA. Some labs still choose GQA for that reason. At long context, where cache traffic dominates the bill, the trade has gone the other way.

If you want the full set of MLA equations, the DeepSeek-V2 paper lays them out in its Section 2.1, and Sebastian Raschka’s architecture gallery entry on MLA places it next to the other attention variants in current models.