Implementing Transformers for Text Generation

Build a small GPT-style Transformer in PyTorch: data and batches, the decoder block, a training loop, sampling with temperature, top-k and top-p, and a KV cache.

Generative AI

A text-generating Transformer is smaller than its reputation. The model in this article fits in about 80 lines of PyTorch, and a scaled-down version of it trains on a laptop CPU. It has the same structure as GPT-2: token and position embeddings, a stack of decoder blocks, and a final layer that scores every token in the vocabulary. Large production models differ mostly in size and in a handful of component choices, which this article covers at the end.

The previous article took self-attention apart: queries, keys and values, the causal mask, and multi-head attention. Here you put it to work. You will prepare data, assemble a decoder-only model, train it, and write the generation loop, including temperature, top-k and top-p sampling and a KV cache. The PyTorch code in this article was run with PyTorch 2, and the outputs shown come from those runs.

What you are building

The model is a decoder-only Transformer that works at the character level. Characters keep the tokenizer trivial, so every line of code is about the model. The task is the one described in Text Generation Models: From RNNs to Transformers: given a sequence of tokens, predict the next one at every position.

flowchart TB
    I["Token ids: B x T"] --> TE["Token embedding"]
    I --> PE["Position embedding"]
    TE --> ADD["Add"]
    PE --> ADD
    ADD --> B1["Decoder block 1"]
    B1 --> BN["... block N"]
    BN --> LN["Final LayerNorm"]
    LN --> H["Linear head to vocab size"]
    H --> L["Logits: B x T x vocab"]

Each decoder block has two sub-layers, causal self-attention and a feed-forward network, and each is wrapped in a residual connection with a LayerNorm in front. The output is a set of logits, one score per vocabulary entry, at every position.

Preparing the data

Any plain-text file works. A common choice for this exercise is the Tiny Shakespeare file, about 1 MB of plays with 65 distinct characters. The tokenizer maps each distinct character to an integer:

import torch

text = open("input.txt", encoding="utf-8").read()
chars = sorted(set(text))
stoi = {ch: i for i, ch in enumerate(chars)}
itos = {i: ch for ch, i in stoi.items()}
encode = lambda s: [stoi[c] for c in s]
decode = lambda ids: "".join(itos[i] for i in ids)

data = torch.tensor(encode(text), dtype=torch.long)
n = int(0.9 * len(data))
train_data, val_data = data[:n], data[n:]

Training examples are random windows of the text. The target is the same window shifted one position to the right, so a single window of length T gives T training examples at once: predict character 2 from character 1, character 3 from characters 1 and 2, and so on. The causal mask is what makes this valid, because position t never sees the token it is asked to predict.

def get_batch(split, block_size, batch_size, device):
    src = train_data if split == "train" else val_data
    ix = torch.randint(len(src) - block_size - 1, (batch_size,))
    x = torch.stack([src[i : i + block_size] for i in ix])
    y = torch.stack([src[i + 1 : i + block_size + 1] for i in ix])
    return x.to(device), y.to(device)

block_size is the context length: the most tokens the model can see at once. The split is by position, not random, so the validation set is a stretch of text the model never trained on.

The model, one piece at a time

Configuration

Keep the hyperparameters in one place. These defaults give a model of about 10.8 million parameters with a 65-character vocabulary:

import math
import torch
from torch import nn
import torch.nn.functional as F

class Config:
    def __init__(self, vocab_size, block_size=256, n_layer=6, n_head=6, d_model=384, dropout=0.1):
        self.vocab_size = vocab_size
        self.block_size = block_size
        self.n_layer = n_layer
        self.n_head = n_head
        self.d_model = d_model
        self.dropout = dropout

Causal self-attention

This is the layer from the previous article, with the four lines of hand-written attention replaced by PyTorch’s fused scaled_dot_product_attention. With is_causal=True it applies the causal mask for you and uses a memory-efficient kernel when the hardware supports one.

class CausalSelfAttention(nn.Module):
    def __init__(self, cfg):
        super().__init__()
        self.n_head = cfg.n_head
        self.qkv = nn.Linear(cfg.d_model, 3 * cfg.d_model)
        self.proj = nn.Linear(cfg.d_model, cfg.d_model)
        self.dropout = cfg.dropout

    def forward(self, x):
        B, T, D = x.shape
        q, k, v = self.qkv(x).split(D, dim=-1)
        q, k, v = (t.view(B, T, self.n_head, D // self.n_head).transpose(1, 2) for t in (q, k, v))
        y = F.scaled_dot_product_attention(
            q, k, v, is_causal=True, dropout_p=self.dropout if self.training else 0.0)
        y = y.transpose(1, 2).reshape(B, T, D)
        return self.proj(y)

The feed-forward network

Attention moves information between positions. The feed-forward network then processes each position on its own: expand to four times the width, apply a non-linearity, project back. Most of a Transformer’s parameters live here.

class MLP(nn.Module):
    def __init__(self, cfg):
        super().__init__()
        self.fc = nn.Linear(cfg.d_model, 4 * cfg.d_model)
        self.proj = nn.Linear(4 * cfg.d_model, cfg.d_model)

    def forward(self, x):
        return self.proj(F.gelu(self.fc(x)))

The decoder block

The block is where the residual connections come in. Each sub-layer reads a normalized copy of the input and adds its result back to the unnormalized stream:

class Block(nn.Module):
    def __init__(self, cfg):
        super().__init__()
        self.ln1 = nn.LayerNorm(cfg.d_model)
        self.attn = CausalSelfAttention(cfg)
        self.ln2 = nn.LayerNorm(cfg.d_model)
        self.mlp = MLP(cfg)
        self.drop = nn.Dropout(cfg.dropout)

    def forward(self, x):
        x = x + self.drop(self.attn(self.ln1(x)))   # pre-norm residual
        x = x + self.drop(self.mlp(self.ln2(x)))
        return x

The original 2017 Transformer put the LayerNorm after the residual addition (post-norm). GPT-2 moved it to the input of each sub-layer (pre-norm), and nearly every model since has followed. With pre-norm there is a clean path from the output back to the embeddings through the additions alone, so gradients reach the early layers without passing through a normalization at every block. Deep pre-norm stacks are much less sensitive to the learning rate and warmup schedule than post-norm ones.

flowchart LR
    X["x"] --> N1["LayerNorm"]
    N1 --> A["Causal attention"]
    A --> P1["Add"]
    X --> P1
    P1 --> N2["LayerNorm"]
    N2 --> M["MLP"]
    M --> P2["Add"]
    P1 --> P2
    P2 --> Y["x out"]

Putting it together

The full model adds embeddings in front and a prediction head behind:

class TinyGPT(nn.Module):
    def __init__(self, cfg):
        super().__init__()
        self.cfg = cfg
        self.tok_emb = nn.Embedding(cfg.vocab_size, cfg.d_model)
        self.pos_emb = nn.Embedding(cfg.block_size, cfg.d_model)
        self.drop = nn.Dropout(cfg.dropout)
        self.blocks = nn.ModuleList(Block(cfg) for _ in range(cfg.n_layer))
        self.ln_f = nn.LayerNorm(cfg.d_model)
        self.head = nn.Linear(cfg.d_model, cfg.vocab_size, bias=False)
        self.head.weight = self.tok_emb.weight      # weight tying
        self.apply(self._init)

    def _init(self, m):
        if isinstance(m, (nn.Linear, nn.Embedding)):
            nn.init.normal_(m.weight, mean=0.0, std=0.02)
        if isinstance(m, nn.Linear) and m.bias is not None:
            nn.init.zeros_(m.bias)

    def forward(self, idx, targets=None):
        B, T = idx.shape
        pos = torch.arange(T, device=idx.device)
        x = self.drop(self.tok_emb(idx) + self.pos_emb(pos))
        for block in self.blocks:
            x = block(x)
        logits = self.head(self.ln_f(x))            # (B, T, vocab_size)
        if targets is None:
            return logits, None
        loss = F.cross_entropy(logits.view(-1, logits.size(-1)), targets.view(-1))
        return logits, loss

Three details are worth understanding:

  • Learned position embeddings. Attention on its own has no sense of order; shuffle the input and each output is the same mix. Adding a learned vector per position fixes that, and it also fixes the maximum context at block_size.
  • Weight tying. The head maps a vector to a score for each token, and the embedding maps each token to a vector. Sharing one matrix between them saves vocab_size × d_model parameters, which matters a great deal with a 50,000-token vocabulary, and usually helps quality slightly. GPT-2 does this.
  • Small initial weights. A normal distribution with standard deviation 0.02 keeps the initial logits close to zero, so the model starts out predicting a near-uniform distribution.

That last point gives you a free sanity check. A model that predicts uniformly over V tokens has a cross-entropy loss of ln(V). With 65 characters, ln(65) ≈ 4.17. If your first measured loss is far above that, something is wrong with the initialization or the targets.

A character-level model has a vocabulary of 65. Roughly what loss should it report before any training?

  • 0, because nothing has been learned yet
  • About 4.17, the natural log of 65
  • About 65, one unit per character

Correct! Small initial weights give a near-uniform prediction, and cross-entropy for a uniform guess over V tokens is ln(V).

Where the parameters go

Counting parameters by hand is a good way to check that you understand the shapes. For the default configuration (d_model 384, 6 layers, vocabulary 65, context 256):

ComponentFormulaParameters
Token embedding (shared with head)vocab × d24,960
Position embeddingblock_size × d98,304
Attention per block (QKV and output, with biases)4d² + 4d591,360
MLP per block (with biases)8d² + 5d1,181,568
LayerNorms per block4d1,536
Six blocks in total6 × above10,646,784
Final LayerNorm2d768
Total10,770,816

sum(p.numel() for p in model.parameters()) gives the same 10,770,816. Two thirds of each block is the MLP, and with a character vocabulary the embeddings barely register. In GPT-2 small, with its 50,257-token vocabulary and width of 768, the token embedding alone is about 38.6 million of its roughly 124 million parameters.

The training loop

Training is ordinary supervised learning: sample a batch, compute the loss, backpropagate, step the optimizer. The parts specific to Transformers are the choice of optimizer settings and the learning-rate schedule.

device = "cuda" if torch.cuda.is_available() else "cpu"
cfg = Config(vocab_size=len(chars))
model = TinyGPT(cfg).to(device)

max_steps, warmup, lr_max = 5000, 200, 1e-3
opt = torch.optim.AdamW(model.parameters(), lr=lr_max, betas=(0.9, 0.95), weight_decay=0.1)

def lr_at(step):
    if step < warmup:                                   # linear warmup
        return lr_max * (step + 1) / warmup
    progress = (step - warmup) / (max_steps - warmup)   # cosine decay to 10%
    return 0.1 * lr_max + 0.9 * lr_max * 0.5 * (1 + math.cos(math.pi * progress))

@torch.no_grad()
def estimate_loss(iters=50):
    model.eval()
    out = {}
    for split in ("train", "val"):
        losses = [model(*get_batch(split, cfg.block_size, 32, device))[1].item()
                  for _ in range(iters)]
        out[split] = sum(losses) / len(losses)
    model.train()
    return out

for step in range(max_steps):
    for g in opt.param_groups:
        g["lr"] = lr_at(step)
    x, y = get_batch("train", cfg.block_size, 32, device)
    _, loss = model(x, y)
    opt.zero_grad(set_to_none=True)
    loss.backward()
    torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
    opt.step()
    if step % 500 == 0:
        print(step, estimate_loss())
flowchart LR
    S["Sample batch x, y"] --> F["Forward: logits, loss"]
    F --> BW["Backward"]
    BW --> C["Clip gradient norm"]
    C --> O["AdamW step at scheduled LR"]
    O -->|"every N steps"| E["Estimate train and val loss"]
    O --> S
    E --> S

Each choice has a reason:

  • AdamW with weight decay is the standard optimizer for Transformers. A beta2 of 0.95 rather than the default 0.999 makes the second-moment estimate react faster, which tends to help stability in language model training.
  • Warmup starts the learning rate low while Adam’s moment estimates are still noisy, then ramps up. Cosine decay lowers it smoothly for the rest of training. The deep learning basics article covers learning rates in general.
  • Gradient clipping caps the size of any single update. An occasional bad batch produces a gradient spike, and without clipping one spike can undo hours of progress.
  • Separate train and validation loss. When validation loss stops falling while training loss keeps dropping, the model is memorizing the text. More dropout, a smaller model or more data are the usual fixes.

On a scaled-down configuration (2 layers, width 128, context 64) trained for only 300 steps on CPU, the loss fell from 5.0 to about 2.64 on a 149-character vocabulary, and samples moved from random characters to word-shaped strings with spaces in the right places. The full configuration needs a GPU and a few thousand steps to produce text with recognizable words and line structure. Don’t expect more from 10 million parameters and 1 MB of data; the point is that the mechanism works and scales.

Generating text

Generation is a loop around the model. Feed the sequence so far, take the logits at the last position, turn them into a probability distribution, sample a token, append it and repeat:

@torch.no_grad()
def generate(model, idx, max_new_tokens, temperature=1.0, top_k=None):
    model.eval()
    for _ in range(max_new_tokens):
        idx_cond = idx[:, -model.cfg.block_size:]        # crop to the context window
        logits, _ = model(idx_cond)
        logits = logits[:, -1, :] / temperature           # last position only
        if top_k is not None:
            kth = torch.topk(logits, top_k).values[:, [-1]]
            logits = logits.masked_fill(logits < kth, float("-inf"))
        probs = F.softmax(logits, dim=-1)
        next_id = torch.multinomial(probs, num_samples=1)
        idx = torch.cat([idx, next_id], dim=1)
    return idx

start = torch.tensor([encode("ROMEO:")], dtype=torch.long, device=device)
print(decode(generate(model, start, 300, temperature=0.8, top_k=40)[0].tolist()))

The crop to block_size matters because the position embedding table has only that many rows. Once the text grows longer than the context, the model sees only the most recent window.

Temperature, top-k and top-p

The raw distribution is rarely what you want to sample from. Three controls shape it:

  • Temperature divides the logits before the softmax. Below 1 it sharpens the distribution toward the most likely tokens; above 1 it flattens it. Temperature near 0 approaches greedy decoding, which always takes the top token and tends to loop.
  • Top-k keeps only the k highest-scoring tokens and sets the rest to minus infinity, so the long tail of unlikely tokens can never be picked.
  • Top-p (nucleus sampling) keeps the smallest set of tokens whose probabilities add up to at least p. Unlike top-k, the size of the set adapts: one token when the model is confident, many when it is not.

Top-p in code needs a sort and a cumulative sum:

def top_p_filter(logits, p=0.9):
    sorted_logits, sorted_idx = torch.sort(logits, descending=True)
    probs = F.softmax(sorted_logits, dim=-1)
    remove = probs.cumsum(dim=-1) - probs > p     # keep the token that crosses p
    sorted_logits[remove] = float("-inf")
    return logits.scatter(-1, sorted_idx, sorted_logits)

logits = torch.tensor([[2.0, 1.0, 0.5, -1.0]])
print(F.softmax(logits, -1))                       # [0.6095, 0.2242, 0.1360, 0.0303]
print(F.softmax(top_p_filter(logits, 0.8), -1))    # [0.7311, 0.2689, 0.0000, 0.0000]

With p = 0.8, the first token alone covers 61 percent, the first two cover 83 percent, so the filter keeps two tokens and renormalizes them. Call it in generate after the temperature division, in place of or after the top-k filter.

SettingWhat it changesLow valueHigh valueTypical starting point
TemperatureSharpness of the whole distributionRepetitive, safeVaried, more errors0.7 to 1.0
Top-kFixed number of candidatesNarrow, predictableClose to no filter40 to 50
Top-pCandidates by cumulative probabilityNarrow when uncertain tooClose to no filter0.9 to 0.95
GreedyAlways the top tokenDeterministicNot applicableShort factual answers, tests

The text generation overview shows how these settings appear in hosted model APIs, where they are request parameters rather than code.

How does top-p sampling differ from top-k sampling?

  • Top-p keeps a variable number of tokens based on cumulative probability; top-k always keeps a fixed number
  • Top-p changes the temperature; top-k does not
  • Top-p always picks the single most likely token

Correct! Top-p adapts the candidate set to the model's confidence, while top-k keeps exactly k tokens every time.

Making generation fast with a KV cache

The generate loop above has an obvious waste. To produce token 200, it runs the whole model over all 199 previous tokens, even though their keys and values were already computed on the previous step. Only the newest token is new.

A KV cache stores each layer’s keys and values and, at every step, runs the model on the newest token alone. Its query attends to the cached keys plus its own, and its key and value are appended to the cache.

sequenceDiagram
    participant G as Generate loop
    participant M as Model
    participant C as KV cache
    G->>M: Prefill: all prompt tokens
    M->>C: Store K and V for every layer
    M-->>G: Logits for last position
    loop Each new token
        G->>M: Newest token only
        M->>C: Read past K and V, append new K and V
        M-->>G: Logits for next token
    end

Here is the attention step written that way. It reuses the CausalSelfAttention layer’s weights:

def attn_step(layer, x_new, cache):
    """x_new: (B, 1, D), the newest token only. cache: dict holding past keys and values."""
    B, T, D = x_new.shape
    q, k, v = layer.qkv(x_new).split(D, dim=-1)
    q, k, v = (t.view(B, T, layer.n_head, D // layer.n_head).transpose(1, 2) for t in (q, k, v))
    if "k" in cache:
        k = torch.cat([cache["k"], k], dim=2)     # append along the time axis
        v = torch.cat([cache["v"], v], dim=2)
    cache["k"], cache["v"] = k, v
    y = F.scaled_dot_product_attention(q, k, v)   # one query, no mask needed
    return layer.proj(y.transpose(1, 2).reshape(B, T, D))

No mask is needed because the single query is the latest position, and everything in the cache is in its past. Feeding five tokens through attn_step one at a time gives the same output as one full causal forward pass over all five, to within floating-point tolerance.

This is how every serving system generates text. It splits work into two phases: prefill, which processes the prompt in one parallel pass and fills the cache, and decode, which produces one token per step. Prefill is limited by compute; decode is mostly limited by how fast the GPU can read the weights and the cache from memory. The cache itself grows with layers, key-value heads, head size and context length, which is why the grouped-query attention described in the previous article matters so much for serving.

What does a KV cache avoid recomputing during generation?

  • The softmax over the vocabulary for every past token
  • The model weights, which are reloaded from disk each step
  • The keys and values of earlier tokens, which do not change when new tokens are added

Correct! Under a causal mask, earlier tokens never see later ones, so their keys and values are fixed and can be stored and reused.

From this model to a modern LLM

The model above is structurally GPT-2. Current open models such as Llama, Mistral and Qwen keep the decoder-only design and change a few components:

ComponentThis article and GPT-2Common in recent modelsWhy
NormalizationLayerNorm, pre-normRMSNorm, pre-normSimpler and slightly cheaper, with similar results
PositionsLearned absolute embeddingsRotary position embeddings (RoPE)Encodes relative position inside attention; extends to longer contexts more gracefully
Feed-forwardGELU, 4x widthSwiGLU, a gated variantBetter quality at the same parameter count
AttentionMulti-headGrouped-query, sometimes sliding windowSmaller KV cache
TokenizerCharactersByte-level BPE or similar, vocabularies of tens of thousands or moreFar fewer tokens per sentence
BiasesIn linear layersOften removedFewer parameters, little or no quality cost reported

None of these changes the loop you wrote: embed, run the blocks, project to logits, sample, append.

Training at real scale adds engineering rather than new ideas. Mixed precision (torch.autocast with bfloat16) halves memory and speeds up matrix multiplications on modern GPUs. torch.compile(model) fuses operations. Gradient accumulation simulates large batches on small GPUs, and data, tensor and pipeline parallelism spread a model over many accelerators.

For anything beyond learning, you would start from a pretrained checkpoint. The Hugging Face transformers library wraps the same generation loop, KV cache included:

from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("gpt2")
model = AutoModelForCausalLM.from_pretrained("gpt2")
inputs = tok("The cloud is", return_tensors="pt")
out = model.generate(**inputs, max_new_tokens=40, do_sample=True, temperature=0.8, top_p=0.9)
print(tok.decode(out[0], skip_special_tokens=True))

If you want to train the model in this article on a cloud GPU rather than a laptop, Building Your First Generative AI Model walks through setting up the environment, and Comparing Cloud Platforms for Generative AI covers where managed notebooks and GPU instances sit on each provider. A later article in this series covers fine-tuning pretrained models, which is how most teams adapt these models in practice.

Further reading

Key takeaways

  • A GPT-style text generator is token and position embeddings, a stack of pre-norm decoder blocks (causal attention plus an MLP, each with a residual connection), and a linear head that scores the vocabulary.
  • Training uses windows of text with targets shifted by one position, so each window yields a prediction at every position; the causal mask keeps this honest.
  • A freshly initialized model should report a loss near ln(vocabulary size), which makes a quick check on your setup.
  • AdamW, warmup with cosine decay and gradient clipping are the standard training recipe; watch validation loss to catch memorization.
  • Generation samples one token at a time from the last position’s logits, shaped by temperature, top-k and top-p.
  • A KV cache stores past keys and values so each step processes only the newest token; serving systems split work into prefill and decode around it.
  • Modern LLMs keep this structure and swap in RMSNorm, RoPE, SwiGLU, grouped-query attention and subword tokenizers.

Part 9 of 9 · Deep Dive into Generative Models

  1. 01Mathematical Foundations of VAEs
  2. 02Implementing VAEs: A Step-by-Step Guide
  3. 03Training GANs: Tips, Tricks, and Common Pitfalls
  4. 04Conditional GANs: Controlling the Output
  5. 05StyleGAN: Generating High-Resolution Images
  6. 06CycleGAN: Image-to-Image Translation
  7. 07Text Generation Models: From RNNs to Transformers
  8. 08The Transformer Architecture: Self-Attention Explained
  9. 09Implementing Transformers for Text Generation
Ready to check your understanding?Take the Word Up! Bot quiz
esc

Type to search all articles · ↑↓ to move · ↵ to open