Text is split into tokens (words or sub-words), each given a numeric ID.
Vectors are compared with each other to decide what each token should pay attention to.
Each token produces a Query (what it's looking for), a Key (what it offers) and a Value (what it passes on). Click a token to see what it attends to.
Attention and a feed-forward network, repeated many times, refine every token's meaning.
The model can only "see" a fixed number of recent tokens at once.
The decoder predicts the next token, samples one, appends it, and repeats.
The whole journey, start to finish. Press play to trace it — or click any stage to jump there.
LLMs convert text into numbers, understand relationships through attention, process information through many layers, and generate coherent text — one token at a time.
Everything above describes one attention mechanism: every token compares itself with every earlier token, scoring them with softmax(QKT / √dk)V. That is the version in the original Transformer paper, and almost no model serves it unchanged.
The reason is the cache rather than the mathematics. During generation the keys and values for every earlier token are kept so they are not recomputed each step. That cache grows with the number of heads, the number of layers and the length of the conversation, and on long contexts it becomes larger than the model weights. Each variant below trades some property of attention for a smaller cache or a cheaper comparison.
| Variant | What it changes | Cache cost | What you give up |
|---|---|---|---|
| Multi-head attention (MHA) | Nothing. Every head keeps its own keys and values. | Baseline, the highest | Nothing in quality. It is the memory cost that makes it impractical at long context. |
| Grouped-query attention (GQA) | Several query heads share one set of keys and values. | Divided by the group size | A small amount of head independence. The usual default in open-weight models, because the quality cost is slight. |
| Sliding window attention | Each token attends only to a fixed window of recent tokens. | Capped by the window, not the conversation | Direct links to distant tokens. Information from far back has to travel indirectly, layer by layer. |
| Multi-head latent attention (MLA) | Keys and values are compressed into a smaller latent vector, and that is cached instead. | A fraction of the baseline | Simplicity. The compression and expansion are extra work, and the latent size becomes another thing to tune. |
| Sparse attention | An index selects a subset of earlier tokens for each query rather than comparing against all of them. | Grows with the subset, not the context | Certainty about what was read. Quality now depends on the selector, and anything it misses is invisible to the model. |
| Hybrid attention and state space | Most layers become recurrent state-space layers, with attention kept in a few. | Constant per token in the recurrent layers | Exact recall of a specific distant token, which a fixed-size state holds less reliably. The retained attention layers exist to cover that. |
Read the column that matters for the decision in front of you. If the constraint is the cost of serving long conversations, the cache column is the one to look at. If the constraint is whether the model can reliably quote something said much earlier, read the last column instead, because that is where every one of these designs pays for the saving.