The Transformer Architecture: Self-Attention Explained

Self-attention from the inside: queries, keys and values, a worked example by hand, scaling, masks, multiple heads, and what it costs to run.

Generative AI

Self-attention is a short formula: softmax(QKᵀ / √d_k) · V. You can write it in five lines of PyTorch, and it sits at the centre of nearly every large language model in production. Short formulas hide a lot, though. Where do Q, K and V come from? Why the square root? What does a “head” actually split? And why does this one operation decide how much memory a model server needs?

The previous article in this series followed text generation from RNNs to the Transformer and stopped at the point where attention replaced recurrence. The Role of Transformers in Generative AI gave the overview. This article opens up the mechanism itself: the shapes, a calculation you can follow by hand, the masks, multi-head attention as real code, and the variants that today’s models use to keep the cost down.

The problem attention solves

A token embedding is a fixed vector. The word “bank” gets the same starting vector whether the sentence is about rivers or interest rates. To be useful, each token’s representation has to change based on the other tokens around it.

Self-attention is the step that does this. It takes a sequence of vectors, one per token, and returns a sequence of the same length. Each output vector is a weighted mix of information from every position in the input, with the weights computed from the content of the tokens themselves. After one attention layer, “bank” in “the river bank flooded” and “bank” in “the bank raised rates” are different vectors.

Two things make it different from earlier approaches:

  • The weights are computed per input, not fixed. A convolution always mixes the same neighbours with the same learned weights; attention decides which positions matter for this particular sentence.
  • Every position can reach every other position in one step. There is no chain of hidden states for information to survive.

A soft dictionary lookup

The clearest way to think about attention is as a lookup in a dictionary where every entry matches a little.

In a normal dictionary, you bring a key, find the one entry with that exact key, and get its value. In attention, each position brings a query. Every position also offers a key and a value. The query is compared with all the keys, each comparison produces a score, and the scores become weights that add up to one. The result is a blend of all the values, weighted by how well each key matched.

So a single position asks “what am I looking for?” (query), advertises “what do I contain?” (key), and hands over “here is my content” (value). Separating the key from the value matters: what makes a token relevant is not necessarily the same as what it should contribute once it is chosen.

flowchart LR
    X["Input vectors X"] --> WQ["x W_Q"]
    X --> WK["x W_K"]
    X --> WV["x W_V"]
    WQ --> Q["Queries Q"]
    WK --> K["Keys K"]
    WV --> V["Values V"]
    Q --> S["Scores: Q times K transposed"]
    K --> S
    S --> SC["Divide by sqrt of d_k"]
    SC --> M["Apply mask"]
    M --> SM["Softmax per row"]
    SM --> O["Weights times V"]
    V --> O
    O --> Y["Output vectors"]

Where queries, keys and values come from

Q, K and V are not three different inputs. In self-attention they all come from the same sequence X, through three learned weight matrices:

  • Q = X · W_Q
  • K = X · W_K
  • V = X · W_V

If the sequence has T tokens and the model width is d_model, then X has shape (T, d_model). Each projection maps d_model down to a smaller size, d_k for queries and keys and d_v for values. In the original Transformer, d_model was 512 and, with 8 heads, d_k = d_v = 64.

The score for position i attending to position j is the dot product of query i with key j. Computing all of them at once is one matrix multiplication, Q · Kᵀ, which gives a T by T score matrix. Row i holds the scores of position i against every position. A softmax across each row turns scores into weights, and multiplying the weight matrix by V gives the output: row i is the weighted sum of all value vectors, using row i’s weights.

The only learned parameters are the projection matrices (plus an output projection, covered below). The attention weights themselves are not parameters. They are recomputed for every input, which is the whole point.

A worked example by hand

Numbers make this concrete. Take three tokens and skip the projections by writing Q, K and V down directly, with d_k = 2:

TokenQueryKeyValue
1[1, 0][1, 0][1, 0]
2[0, 1][0, 1][0, 2]
3[1, 1][1, 1][1, 1]

Step 1: scores. Each entry is a query dotted with a key, divided by √2 ≈ 1.41. Token 1’s query [1, 0] against the three keys gives 1, 0 and 1, so its row is [0.71, 0, 0.71]. The full matrix:

Key 1Key 2Key 3
Query 10.7100.71
Query 200.710.71
Query 30.710.711.41

Step 2: softmax per row. For row 1, e^0.71 ≈ 2.03 and e^0 = 1, so the weights are 2.03, 1 and 2.03 divided by their sum of 5.06: about [0.40, 0.20, 0.40]. Row 2 comes out as [0.20, 0.40, 0.40] and row 3 as [0.25, 0.25, 0.50].

Step 3: weighted sum of values. Token 1’s output is 0.40 · [1, 0] + 0.20 · [0, 2] + 0.40 · [1, 1] = [0.80, 0.80]. Token 2 gets [0.60, 1.20] and token 3 gets [0.75, 1.00].

Notice what happened. Token 1’s query points in the first direction, so it drew most from tokens whose keys also point that way (1 and 3) and less from token 2. Token 3’s query matches every key and matches its own key best, so its output is a broad blend leaning on itself. Every output is a mixture, but a different mixture for each position.

Why divide by the square root of d_k

Drop the √d_k and the formula still runs. It just trains badly once d_k is realistic.

If the entries of a query and a key are independent with mean 0 and variance 1, their dot product is a sum of d_k such products, so its variance is d_k and its standard deviation is √d_k. A quick simulation with random vectors gives standard deviations of about 2 for d_k = 4, 8 for d_k = 64 and 22.6 for d_k = 512, right on √d_k.

Large score differences push the softmax into saturation. With scores spread over tens of units, one position gets a weight near 1 and the rest near 0, and the gradient of the softmax becomes tiny almost everywhere. Learning slows or stalls. Dividing by √d_k brings the scores back to unit variance regardless of head size, so the softmax starts out in a range where it can still learn which positions matter.

Why are attention scores divided by the square root of d_k?

  • To make the attention weights in each row add up to one
  • To keep dot products from growing with head size and saturating the softmax
  • To reduce the number of matrix multiplications

Correct! Dot product variance grows with d_k, and scaling by its square root keeps the softmax in a range with useful gradients.

Masks: controlling who sees whom

Before the softmax, the model can set some scores to minus infinity. Since e^(−∞) = 0, those positions get exactly zero weight. Two masks are common.

The causal mask

A text generator predicts each token from the tokens before it. During training, the whole sequence is fed in at once, so without a mask position 2 could simply look at position 3 and copy the answer. The causal mask blocks every score above the diagonal: position i can attend only to positions 1 through i.

Applied to the worked example, row 1 can see only itself, so its weights become [1, 0, 0] and its output is just its own value, [1, 0]. Row 2 sees tokens 1 and 2, with scores 0 and 0.71, giving weights of about [0.33, 0.67, 0] and an output of [0.33, 1.34]. Row 3 already sees everything, so it is unchanged.

flowchart LR
    subgraph Causal["Causal mask, 4 tokens"]
        R1["Token 1 sees: 1"]
        R2["Token 2 sees: 1, 2"]
        R3["Token 3 sees: 1, 2, 3"]
        R4["Token 4 sees: 1, 2, 3, 4"]
    end
    subgraph Full["No mask, encoder style"]
        F1["Every token sees: 1, 2, 3, 4"]
    end

The mask is what lets a decoder train on every position of a sequence in parallel while still behaving exactly as it will at generation time, when the future tokens do not exist yet. Encoder-only models such as BERT skip it, because they read the whole input and never generate left to right.

The padding mask

Batches hold sequences of different lengths, padded to the same size with filler tokens. Real tokens should not attend to padding, so a padding mask zeroes out those columns. In PyTorch you pass a boolean mask where True means “may attend”:

import torch
import torch.nn.functional as F

q = torch.randn(1, 4, 10, 16)                  # (batch, heads, tokens, d_k)
keep = torch.tensor([[True] * 7 + [False] * 3])  # last 3 tokens are padding
y = F.scaled_dot_product_attention(q, q, q, attn_mask=keep[:, None, None, :])
print(y.shape)                                  # torch.Size([1, 4, 10, 16])

Multi-head attention

One attention operation produces one set of weights per position. That is a single opinion about which tokens matter. A pronoun needs to find its noun, a verb needs its subject, a closing bracket needs its opening one. Asking one weight pattern to do all of that at once is a lot.

Multi-head attention runs several attention operations side by side. Each head has its own W_Q, W_K and W_V, projecting into a smaller space of size d_model / h. Each head computes its own weights and outputs. The head outputs are concatenated back to width d_model and passed through one more learned matrix, W_O, which mixes information across heads.

The total compute is about the same as one big head of full width, because each head works in a slice of the space. In the original paper, 8 heads of size 64 replaced one head of size 512.

flowchart TB
    X["Input: T x d_model"] --> P["Project to Q, K, V"]
    P --> H1["Head 1: attention in d_k dims"]
    P --> H2["Head 2"]
    P --> HD["..."]
    P --> HH["Head h"]
    H1 --> C["Concatenate: T x d_model"]
    H2 --> C
    HD --> C
    HH --> C
    C --> WO["Output projection W_O"]
    WO --> Y["Output: T x d_model"]

In code, the heads are not separate modules. You compute Q, K and V at full width with one matrix multiplication, then reshape so the head becomes its own dimension. Here is a complete multi-head self-attention layer:

import math
import torch
from torch import nn

class MultiHeadSelfAttention(nn.Module):
    def __init__(self, d_model=512, n_heads=8):
        super().__init__()
        assert d_model % n_heads == 0
        self.h, self.d_k = n_heads, d_model // n_heads
        self.qkv = nn.Linear(d_model, 3 * d_model, bias=False)  # W_Q, W_K, W_V together
        self.out = nn.Linear(d_model, d_model, bias=False)      # W_O

    def forward(self, x, causal=True):
        B, T, D = x.shape
        q, k, v = self.qkv(x).split(D, dim=-1)                  # each (B, T, D)
        # split the model width into heads: (B, h, T, d_k)
        q, k, v = (t.view(B, T, self.h, self.d_k).transpose(1, 2) for t in (q, k, v))
        scores = q @ k.transpose(-2, -1) / math.sqrt(self.d_k)  # (B, h, T, T)
        if causal:
            mask = torch.ones(T, T, dtype=torch.bool, device=x.device).triu(1)
            scores = scores.masked_fill(mask, float("-inf"))
        weights = scores.softmax(dim=-1)
        y = weights @ v                                         # (B, h, T, d_k)
        y = y.transpose(1, 2).reshape(B, T, D)                  # concatenate heads
        return self.out(y)

attn = MultiHeadSelfAttention()
x = torch.randn(2, 10, 512)          # 2 sequences, 10 tokens each
print(attn(x).shape)                 # torch.Size([2, 10, 512])

The view and transpose lines are where “multi-head” happens. Nothing is duplicated; the 512 columns are reinterpreted as 8 groups of 64, and the batched matrix multiplications treat each group independently.

In real code you would replace the four lines from scores to y with torch.nn.functional.scaled_dot_product_attention(q, k, v, is_causal=True). It computes the same result (checking it against the layer above gives matching outputs to within floating-point tolerance) and picks a fused kernel such as FlashAttention when the hardware supports it.

In a layer with d_model = 512 and 8 heads, how does each head relate to the full width?

  • Each head works on a full 512-dimensional copy, so compute grows eightfold
  • Each head attends to a different eighth of the tokens
  • Each head works in its own 64-dimensional projection, and outputs are concatenated back to 512

Correct! The width is split across heads, so total compute stays close to one full-width head while each head can learn its own attention pattern.

Self-attention and cross-attention

In self-attention, Q, K and V all come from the same sequence. Cross-attention takes queries from one sequence and keys and values from another. The original Transformer’s decoder had both: masked self-attention over the translation written so far, then cross-attention into the encoder’s representation of the source sentence. Speech models such as Whisper still work this way, with the decoder’s text queries attending to encoded audio, and many image generators use cross-attention to let image features attend to the text prompt.

Decoder-only language models have no second sequence, so they use only masked self-attention. The prompt and the answer are one sequence. The Role of Transformers article compares the three arrangements, and a later article in this series looks at BERT and GPT in detail.

What attention costs

The score matrix is T by T, per head, per layer. That is where both the power and the cost of attention come from.

With a context of 8,192 tokens, one head’s score matrix has about 67 million entries. Stored in 16-bit precision, that is about 134 MB, for one head in one layer of one sequence. Doubling the context to 16,384 tokens quadruples it. A naive implementation would write that matrix to GPU memory and read it back for the softmax and the multiplication by V, and that memory traffic, more than the arithmetic, is what made long contexts slow.

FlashAttention (Dao and colleagues, 2022) fixed the memory side without changing the result. It splits Q, K and V into tiles that fit in the GPU’s fast on-chip memory, computes the softmax incrementally tile by tile, and never writes the full score matrix out. The answer is exact attention, not an approximation, and memory use grows linearly with sequence length instead of quadratically. The arithmetic is still quadratic. Later versions improved GPU utilization further, and fused kernels like this are now the default path in PyTorch and in inference servers.

During generation, a different cost dominates. Each new token needs its query compared against the keys of every earlier token, and those keys and values are kept from step to step in the KV cache rather than recomputed. The cache size per token is:

2 (keys and values) × layers × key-value heads × head size × bytes per number

That formula explains the main family of attention variants in today’s models.

Sharing keys and values across heads

If every query head has its own keys and values, the KV cache is large. Llama 2 70B is a useful example: 80 layers, 64 query heads of size 128. With standard multi-head attention, the cache would hold 2 × 80 × 64 × 128 × 2 bytes, about 2.6 MB per token in 16-bit precision. A 4,096-token conversation would need around 10.7 GB of cache for a single user.

The fix is to let query heads share keys and values:

  • Multi-query attention (MQA), proposed by Noam Shazeer in 2019, uses a single key-value head shared by all query heads. The cache shrinks by the number of heads, at some cost in quality and training stability.
  • Grouped-query attention (GQA), from Ainslie and colleagues in 2023, sits in between: query heads are split into groups, and each group shares one key-value head. Llama 2 70B uses 8 key-value heads for its 64 query heads, so its cache is 8 times smaller than the multi-head version, about 330 KB per token.
  • Multi-head latent attention (MLA), introduced with DeepSeek-V2 in 2024, compresses keys and values into a small shared latent vector that is cached instead, and expands it back when attention is computed.
VariantKey-value headsKV cache sizeQualitySeen in
Multi-head (MHA)One per query headLargestBaselineOriginal Transformer, GPT-2, BERT
Multi-query (MQA)One for all headsSmallest of the three classic optionsSome lossPaLM, Falcon-7B
Grouped-query (GQA)One per groupIn between, often 4 to 8 times smaller than MHAClose to MHALlama 2 70B, Llama 3, Mistral 7B
Latent (MLA)Compressed latentSmallReported comparable to MHADeepSeek-V2 and V3

PyTorch’s scaled_dot_product_attention handles grouped-query attention directly when you pass fewer key-value heads and set enable_gqa=True:

q = torch.randn(1, 32, 16, 128)   # 32 query heads, 16 tokens, head size 128
k = torch.randn(1, 8, 16, 128)    # 8 shared key-value heads
v = torch.randn(1, 8, 16, 128)
y = F.scaled_dot_product_attention(q, k, v, is_causal=True, enable_gqa=True)
print(y.shape)                    # torch.Size([1, 32, 16, 128])

A model has 64 query heads and 8 key-value heads. What sets the size of its KV cache per token?

  • The 8 key-value heads, so the cache is 8 times smaller than with 64
  • The 64 query heads, because every query head is cached
  • Neither, because the cache size depends only on the vocabulary

Correct! Only keys and values are cached, so grouped-query attention shrinks the cache in proportion to the number of key-value heads.

Restricting the window

Another approach limits how far back each token looks. In sliding-window attention, each position attends only to the previous W tokens, so cost per token is fixed and the cache can be trimmed. Stacking layers still lets information travel further than one window, because each layer extends the reach by another W positions. Mistral 7B used a 4,096-token sliding window. Several recent models interleave sliding-window layers with a smaller number of full-attention layers, keeping exact long-range recall where it matters and saving memory everywhere else.

What attention heads learn

Because the weights are explicit, it is tempting to read them as explanations. Some heads do show clean patterns: attending to the previous token, to the matching bracket, or to the noun a pronoun refers to. Research at Anthropic in 2022 identified “induction heads”, pairs of heads in different layers that together find an earlier occurrence of the current token and copy what came after it, and linked them to in-context learning.

Treat attention maps with care, though. Jain and Wallace’s 2019 paper “Attention is not Explanation” showed that quite different attention distributions can produce the same prediction. A high weight says information flowed from one position to another in that layer. It does not say that token caused the output, because the value vectors, the feed-forward layers and every later layer also shape the result.

Using this in practice

You will rarely write attention from scratch for production. What this understanding buys you is the ability to reason about the models you deploy:

  • Context length drives cost. Long prompts mean more attention work during prefill and a larger KV cache during generation. That is why providers price input tokens and some charge more for very long contexts. Comparing Cloud Platforms for Generative AI covers how Amazon Bedrock, Microsoft Foundry and Google Cloud’s Gemini Enterprise Agent Platform present these models.
  • Head layout drives serving memory. If you self-host an open model on a cloud GPU, the number of key-value heads, not the number of query heads, decides how many concurrent conversations fit in memory. Check the model’s configuration file for num_key_value_heads before sizing instances.
  • Fused kernels matter. If you train or fine-tune with PyTorch, using scaled_dot_product_attention instead of a hand-written version gets you memory-efficient kernels with no code changes beyond that one call.

Further reading

Key takeaways

  • Self-attention turns fixed token embeddings into context-dependent ones by letting each position take a weighted mix of every other position, with weights computed from the input itself.
  • Queries, keys and values are three learned projections of the same input; the query-key dot product sets the weight, and the value is what gets passed along.
  • Scores are divided by √d_k because dot product variance grows with head size, and unscaled scores saturate the softmax.
  • Masks set scores to minus infinity before the softmax: the causal mask stops a decoder seeing future tokens, and the padding mask hides filler tokens.
  • Multi-head attention splits the model width into several smaller heads that learn different patterns, at about the cost of one full-width head.
  • Attention cost grows with the square of the context; FlashAttention removes the quadratic memory traffic, and GQA, MQA and MLA shrink the KV cache that dominates serving memory.

Part 8 of 8 · Deep Dive into Generative Models

  1. 01Mathematical Foundations of VAEs
  2. 02Implementing VAEs: A Step-by-Step Guide
  3. 03Training GANs: Tips, Tricks, and Common Pitfalls
  4. 04Conditional GANs: Controlling the Output
  5. 05StyleGAN: Generating High-Resolution Images
  6. 06CycleGAN: Image-to-Image Translation
  7. 07Text Generation Models: From RNNs to Transformers
  8. 08The Transformer Architecture: Self-Attention Explained
Ready to check your understanding?Take the Word Up! Bot quiz
esc

Type to search all articles · ↑↓ to move · ↵ to open