Implementing Transformers for Text Generation
Build a small GPT-style Transformer in PyTorch: data and batches, the decoder block, a training loop, sampling with temperature, top-k and top-p, and a KV cache.
A text-generating Transformer is smaller than its reputation. The model in this article fits in about 80 lines of PyTorch, and a scaled-down version of it trains on a laptop CPU. It has the same structure as GPT-2: token and position embeddings, a stack of decoder blocks, and a final layer that scores every token in the vocabulary. Large production models differ mostly in size and in a handful of component choices, which this article covers at the end.
The previous article took self-attention apart: queries, keys and values, the causal mask, and multi-head attention. Here you put it to work. You will prepare data, assemble a decoder-only model, train it, and write the generation loop, including temperature, top-k and top-p sampling and a KV cache. The PyTorch code in this article was run with PyTorch 2, and the outputs shown come from those runs.
What you are building
The model is a decoder-only Transformer that works at the character level. Characters keep the tokenizer trivial, so every line of code is about the model. The task is the one described in Text Generation Models: From RNNs to Transformers: given a sequence of tokens, predict the next one at every position.
flowchart TB
I["Token ids: B x T"] --> TE["Token embedding"]
I --> PE["Position embedding"]
TE --> ADD["Add"]
PE --> ADD
ADD --> B1["Decoder block 1"]
B1 --> BN["... block N"]
BN --> LN["Final LayerNorm"]
LN --> H["Linear head to vocab size"]
H --> L["Logits: B x T x vocab"]
Each decoder block has two sub-layers, causal self-attention and a feed-forward network, and each is wrapped in a residual connection with a LayerNorm in front. The output is a set of logits, one score per vocabulary entry, at every position.
Preparing the data
Any plain-text file works. A common choice for this exercise is the Tiny Shakespeare file, about 1 MB of plays with 65 distinct characters. The tokenizer maps each distinct character to an integer:
import torch
text = open("input.txt", encoding="utf-8").read()
chars = sorted(set(text))
stoi = {ch: i for i, ch in enumerate(chars)}
itos = {i: ch for ch, i in stoi.items()}
encode = lambda s: [stoi[c] for c in s]
decode = lambda ids: "".join(itos[i] for i in ids)
data = torch.tensor(encode(text), dtype=torch.long)
n = int(0.9 * len(data))
train_data, val_data = data[:n], data[n:]
Training examples are random windows of the text. The target is the same window shifted one position to the right, so a single window of length T gives T training examples at once: predict character 2 from character 1, character 3 from characters 1 and 2, and so on. The causal mask is what makes this valid, because position t never sees the token it is asked to predict.
def get_batch(split, block_size, batch_size, device):
src = train_data if split == "train" else val_data
ix = torch.randint(len(src) - block_size - 1, (batch_size,))
x = torch.stack([src[i : i + block_size] for i in ix])
y = torch.stack([src[i + 1 : i + block_size + 1] for i in ix])
return x.to(device), y.to(device)
block_size is the context length: the most tokens the model can see at once. The split is by position, not random, so the validation set is a stretch of text the model never trained on.
The model, one piece at a time
Configuration
Keep the hyperparameters in one place. These defaults give a model of about 10.8 million parameters with a 65-character vocabulary:
import math
import torch
from torch import nn
import torch.nn.functional as F
class Config:
def __init__(self, vocab_size, block_size=256, n_layer=6, n_head=6, d_model=384, dropout=0.1):
self.vocab_size = vocab_size
self.block_size = block_size
self.n_layer = n_layer
self.n_head = n_head
self.d_model = d_model
self.dropout = dropout
Causal self-attention
This is the layer from the previous article, with the four lines of hand-written attention replaced by PyTorch’s fused scaled_dot_product_attention. With is_causal=True it applies the causal mask for you and uses a memory-efficient kernel when the hardware supports one.
class CausalSelfAttention(nn.Module):
def __init__(self, cfg):
super().__init__()
self.n_head = cfg.n_head
self.qkv = nn.Linear(cfg.d_model, 3 * cfg.d_model)
self.proj = nn.Linear(cfg.d_model, cfg.d_model)
self.dropout = cfg.dropout
def forward(self, x):
B, T, D = x.shape
q, k, v = self.qkv(x).split(D, dim=-1)
q, k, v = (t.view(B, T, self.n_head, D // self.n_head).transpose(1, 2) for t in (q, k, v))
y = F.scaled_dot_product_attention(
q, k, v, is_causal=True, dropout_p=self.dropout if self.training else 0.0)
y = y.transpose(1, 2).reshape(B, T, D)
return self.proj(y)
The feed-forward network
Attention moves information between positions. The feed-forward network then processes each position on its own: expand to four times the width, apply a non-linearity, project back. Most of a Transformer’s parameters live here.
class MLP(nn.Module):
def __init__(self, cfg):
super().__init__()
self.fc = nn.Linear(cfg.d_model, 4 * cfg.d_model)
self.proj = nn.Linear(4 * cfg.d_model, cfg.d_model)
def forward(self, x):
return self.proj(F.gelu(self.fc(x)))
The decoder block
The block is where the residual connections come in. Each sub-layer reads a normalized copy of the input and adds its result back to the unnormalized stream:
class Block(nn.Module):
def __init__(self, cfg):
super().__init__()
self.ln1 = nn.LayerNorm(cfg.d_model)
self.attn = CausalSelfAttention(cfg)
self.ln2 = nn.LayerNorm(cfg.d_model)
self.mlp = MLP(cfg)
self.drop = nn.Dropout(cfg.dropout)
def forward(self, x):
x = x + self.drop(self.attn(self.ln1(x))) # pre-norm residual
x = x + self.drop(self.mlp(self.ln2(x)))
return x
The original 2017 Transformer put the LayerNorm after the residual addition (post-norm). GPT-2 moved it to the input of each sub-layer (pre-norm), and nearly every model since has followed. With pre-norm there is a clean path from the output back to the embeddings through the additions alone, so gradients reach the early layers without passing through a normalization at every block. Deep pre-norm stacks are much less sensitive to the learning rate and warmup schedule than post-norm ones.
flowchart LR
X["x"] --> N1["LayerNorm"]
N1 --> A["Causal attention"]
A --> P1["Add"]
X --> P1
P1 --> N2["LayerNorm"]
N2 --> M["MLP"]
M --> P2["Add"]
P1 --> P2
P2 --> Y["x out"]
Putting it together
The full model adds embeddings in front and a prediction head behind:
class TinyGPT(nn.Module):
def __init__(self, cfg):
super().__init__()
self.cfg = cfg
self.tok_emb = nn.Embedding(cfg.vocab_size, cfg.d_model)
self.pos_emb = nn.Embedding(cfg.block_size, cfg.d_model)
self.drop = nn.Dropout(cfg.dropout)
self.blocks = nn.ModuleList(Block(cfg) for _ in range(cfg.n_layer))
self.ln_f = nn.LayerNorm(cfg.d_model)
self.head = nn.Linear(cfg.d_model, cfg.vocab_size, bias=False)
self.head.weight = self.tok_emb.weight # weight tying
self.apply(self._init)
def _init(self, m):
if isinstance(m, (nn.Linear, nn.Embedding)):
nn.init.normal_(m.weight, mean=0.0, std=0.02)
if isinstance(m, nn.Linear) and m.bias is not None:
nn.init.zeros_(m.bias)
def forward(self, idx, targets=None):
B, T = idx.shape
pos = torch.arange(T, device=idx.device)
x = self.drop(self.tok_emb(idx) + self.pos_emb(pos))
for block in self.blocks:
x = block(x)
logits = self.head(self.ln_f(x)) # (B, T, vocab_size)
if targets is None:
return logits, None
loss = F.cross_entropy(logits.view(-1, logits.size(-1)), targets.view(-1))
return logits, loss
Three details are worth understanding:
- Learned position embeddings. Attention on its own has no sense of order; shuffle the input and each output is the same mix. Adding a learned vector per position fixes that, and it also fixes the maximum context at
block_size. - Weight tying. The head maps a vector to a score for each token, and the embedding maps each token to a vector. Sharing one matrix between them saves
vocab_size × d_modelparameters, which matters a great deal with a 50,000-token vocabulary, and usually helps quality slightly. GPT-2 does this. - Small initial weights. A normal distribution with standard deviation 0.02 keeps the initial logits close to zero, so the model starts out predicting a near-uniform distribution.
That last point gives you a free sanity check. A model that predicts uniformly over V tokens has a cross-entropy loss of ln(V). With 65 characters, ln(65) ≈ 4.17. If your first measured loss is far above that, something is wrong with the initialization or the targets.
A character-level model has a vocabulary of 65. Roughly what loss should it report before any training?
Correct! Small initial weights give a near-uniform prediction, and cross-entropy for a uniform guess over V tokens is ln(V).
Where the parameters go
Counting parameters by hand is a good way to check that you understand the shapes. For the default configuration (d_model 384, 6 layers, vocabulary 65, context 256):
| Component | Formula | Parameters |
|---|---|---|
| Token embedding (shared with head) | vocab × d | 24,960 |
| Position embedding | block_size × d | 98,304 |
| Attention per block (QKV and output, with biases) | 4d² + 4d | 591,360 |
| MLP per block (with biases) | 8d² + 5d | 1,181,568 |
| LayerNorms per block | 4d | 1,536 |
| Six blocks in total | 6 × above | 10,646,784 |
| Final LayerNorm | 2d | 768 |
| Total | 10,770,816 |
sum(p.numel() for p in model.parameters()) gives the same 10,770,816. Two thirds of each block is the MLP, and with a character vocabulary the embeddings barely register. In GPT-2 small, with its 50,257-token vocabulary and width of 768, the token embedding alone is about 38.6 million of its roughly 124 million parameters.
The training loop
Training is ordinary supervised learning: sample a batch, compute the loss, backpropagate, step the optimizer. The parts specific to Transformers are the choice of optimizer settings and the learning-rate schedule.
device = "cuda" if torch.cuda.is_available() else "cpu"
cfg = Config(vocab_size=len(chars))
model = TinyGPT(cfg).to(device)
max_steps, warmup, lr_max = 5000, 200, 1e-3
opt = torch.optim.AdamW(model.parameters(), lr=lr_max, betas=(0.9, 0.95), weight_decay=0.1)
def lr_at(step):
if step < warmup: # linear warmup
return lr_max * (step + 1) / warmup
progress = (step - warmup) / (max_steps - warmup) # cosine decay to 10%
return 0.1 * lr_max + 0.9 * lr_max * 0.5 * (1 + math.cos(math.pi * progress))
@torch.no_grad()
def estimate_loss(iters=50):
model.eval()
out = {}
for split in ("train", "val"):
losses = [model(*get_batch(split, cfg.block_size, 32, device))[1].item()
for _ in range(iters)]
out[split] = sum(losses) / len(losses)
model.train()
return out
for step in range(max_steps):
for g in opt.param_groups:
g["lr"] = lr_at(step)
x, y = get_batch("train", cfg.block_size, 32, device)
_, loss = model(x, y)
opt.zero_grad(set_to_none=True)
loss.backward()
torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
opt.step()
if step % 500 == 0:
print(step, estimate_loss())
flowchart LR
S["Sample batch x, y"] --> F["Forward: logits, loss"]
F --> BW["Backward"]
BW --> C["Clip gradient norm"]
C --> O["AdamW step at scheduled LR"]
O -->|"every N steps"| E["Estimate train and val loss"]
O --> S
E --> S
Each choice has a reason:
- AdamW with weight decay is the standard optimizer for Transformers. A beta2 of 0.95 rather than the default 0.999 makes the second-moment estimate react faster, which tends to help stability in language model training.
- Warmup starts the learning rate low while Adam’s moment estimates are still noisy, then ramps up. Cosine decay lowers it smoothly for the rest of training. The deep learning basics article covers learning rates in general.
- Gradient clipping caps the size of any single update. An occasional bad batch produces a gradient spike, and without clipping one spike can undo hours of progress.
- Separate train and validation loss. When validation loss stops falling while training loss keeps dropping, the model is memorizing the text. More dropout, a smaller model or more data are the usual fixes.
On a scaled-down configuration (2 layers, width 128, context 64) trained for only 300 steps on CPU, the loss fell from 5.0 to about 2.64 on a 149-character vocabulary, and samples moved from random characters to word-shaped strings with spaces in the right places. The full configuration needs a GPU and a few thousand steps to produce text with recognizable words and line structure. Don’t expect more from 10 million parameters and 1 MB of data; the point is that the mechanism works and scales.
Generating text
Generation is a loop around the model. Feed the sequence so far, take the logits at the last position, turn them into a probability distribution, sample a token, append it and repeat:
@torch.no_grad()
def generate(model, idx, max_new_tokens, temperature=1.0, top_k=None):
model.eval()
for _ in range(max_new_tokens):
idx_cond = idx[:, -model.cfg.block_size:] # crop to the context window
logits, _ = model(idx_cond)
logits = logits[:, -1, :] / temperature # last position only
if top_k is not None:
kth = torch.topk(logits, top_k).values[:, [-1]]
logits = logits.masked_fill(logits < kth, float("-inf"))
probs = F.softmax(logits, dim=-1)
next_id = torch.multinomial(probs, num_samples=1)
idx = torch.cat([idx, next_id], dim=1)
return idx
start = torch.tensor([encode("ROMEO:")], dtype=torch.long, device=device)
print(decode(generate(model, start, 300, temperature=0.8, top_k=40)[0].tolist()))
The crop to block_size matters because the position embedding table has only that many rows. Once the text grows longer than the context, the model sees only the most recent window.
Temperature, top-k and top-p
The raw distribution is rarely what you want to sample from. Three controls shape it:
- Temperature divides the logits before the softmax. Below 1 it sharpens the distribution toward the most likely tokens; above 1 it flattens it. Temperature near 0 approaches greedy decoding, which always takes the top token and tends to loop.
- Top-k keeps only the k highest-scoring tokens and sets the rest to minus infinity, so the long tail of unlikely tokens can never be picked.
- Top-p (nucleus sampling) keeps the smallest set of tokens whose probabilities add up to at least p. Unlike top-k, the size of the set adapts: one token when the model is confident, many when it is not.
Top-p in code needs a sort and a cumulative sum:
def top_p_filter(logits, p=0.9):
sorted_logits, sorted_idx = torch.sort(logits, descending=True)
probs = F.softmax(sorted_logits, dim=-1)
remove = probs.cumsum(dim=-1) - probs > p # keep the token that crosses p
sorted_logits[remove] = float("-inf")
return logits.scatter(-1, sorted_idx, sorted_logits)
logits = torch.tensor([[2.0, 1.0, 0.5, -1.0]])
print(F.softmax(logits, -1)) # [0.6095, 0.2242, 0.1360, 0.0303]
print(F.softmax(top_p_filter(logits, 0.8), -1)) # [0.7311, 0.2689, 0.0000, 0.0000]
With p = 0.8, the first token alone covers 61 percent, the first two cover 83 percent, so the filter keeps two tokens and renormalizes them. Call it in generate after the temperature division, in place of or after the top-k filter.
| Setting | What it changes | Low value | High value | Typical starting point |
|---|---|---|---|---|
| Temperature | Sharpness of the whole distribution | Repetitive, safe | Varied, more errors | 0.7 to 1.0 |
| Top-k | Fixed number of candidates | Narrow, predictable | Close to no filter | 40 to 50 |
| Top-p | Candidates by cumulative probability | Narrow when uncertain too | Close to no filter | 0.9 to 0.95 |
| Greedy | Always the top token | Deterministic | Not applicable | Short factual answers, tests |
The text generation overview shows how these settings appear in hosted model APIs, where they are request parameters rather than code.
How does top-p sampling differ from top-k sampling?
Correct! Top-p adapts the candidate set to the model's confidence, while top-k keeps exactly k tokens every time.
Making generation fast with a KV cache
The generate loop above has an obvious waste. To produce token 200, it runs the whole model over all 199 previous tokens, even though their keys and values were already computed on the previous step. Only the newest token is new.
A KV cache stores each layer’s keys and values and, at every step, runs the model on the newest token alone. Its query attends to the cached keys plus its own, and its key and value are appended to the cache.
sequenceDiagram
participant G as Generate loop
participant M as Model
participant C as KV cache
G->>M: Prefill: all prompt tokens
M->>C: Store K and V for every layer
M-->>G: Logits for last position
loop Each new token
G->>M: Newest token only
M->>C: Read past K and V, append new K and V
M-->>G: Logits for next token
end
Here is the attention step written that way. It reuses the CausalSelfAttention layer’s weights:
def attn_step(layer, x_new, cache):
"""x_new: (B, 1, D), the newest token only. cache: dict holding past keys and values."""
B, T, D = x_new.shape
q, k, v = layer.qkv(x_new).split(D, dim=-1)
q, k, v = (t.view(B, T, layer.n_head, D // layer.n_head).transpose(1, 2) for t in (q, k, v))
if "k" in cache:
k = torch.cat([cache["k"], k], dim=2) # append along the time axis
v = torch.cat([cache["v"], v], dim=2)
cache["k"], cache["v"] = k, v
y = F.scaled_dot_product_attention(q, k, v) # one query, no mask needed
return layer.proj(y.transpose(1, 2).reshape(B, T, D))
No mask is needed because the single query is the latest position, and everything in the cache is in its past. Feeding five tokens through attn_step one at a time gives the same output as one full causal forward pass over all five, to within floating-point tolerance.
This is how every serving system generates text. It splits work into two phases: prefill, which processes the prompt in one parallel pass and fills the cache, and decode, which produces one token per step. Prefill is limited by compute; decode is mostly limited by how fast the GPU can read the weights and the cache from memory. The cache itself grows with layers, key-value heads, head size and context length, which is why the grouped-query attention described in the previous article matters so much for serving.
What does a KV cache avoid recomputing during generation?
Correct! Under a causal mask, earlier tokens never see later ones, so their keys and values are fixed and can be stored and reused.
From this model to a modern LLM
The model above is structurally GPT-2. Current open models such as Llama, Mistral and Qwen keep the decoder-only design and change a few components:
| Component | This article and GPT-2 | Common in recent models | Why |
|---|---|---|---|
| Normalization | LayerNorm, pre-norm | RMSNorm, pre-norm | Simpler and slightly cheaper, with similar results |
| Positions | Learned absolute embeddings | Rotary position embeddings (RoPE) | Encodes relative position inside attention; extends to longer contexts more gracefully |
| Feed-forward | GELU, 4x width | SwiGLU, a gated variant | Better quality at the same parameter count |
| Attention | Multi-head | Grouped-query, sometimes sliding window | Smaller KV cache |
| Tokenizer | Characters | Byte-level BPE or similar, vocabularies of tens of thousands or more | Far fewer tokens per sentence |
| Biases | In linear layers | Often removed | Fewer parameters, little or no quality cost reported |
None of these changes the loop you wrote: embed, run the blocks, project to logits, sample, append.
Training at real scale adds engineering rather than new ideas. Mixed precision (torch.autocast with bfloat16) halves memory and speeds up matrix multiplications on modern GPUs. torch.compile(model) fuses operations. Gradient accumulation simulates large batches on small GPUs, and data, tensor and pipeline parallelism spread a model over many accelerators.
For anything beyond learning, you would start from a pretrained checkpoint. The Hugging Face transformers library wraps the same generation loop, KV cache included:
from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("gpt2")
model = AutoModelForCausalLM.from_pretrained("gpt2")
inputs = tok("The cloud is", return_tensors="pt")
out = model.generate(**inputs, max_new_tokens=40, do_sample=True, temperature=0.8, top_p=0.9)
print(tok.decode(out[0], skip_special_tokens=True))
If you want to train the model in this article on a cloud GPU rather than a laptop, Building Your First Generative AI Model walks through setting up the environment, and Comparing Cloud Platforms for Generative AI covers where managed notebooks and GPU instances sit on each provider. A later article in this series covers fine-tuning pretrained models, which is how most teams adapt these models in practice.
Further reading
- Attention Is All You Need, Vaswani and colleagues, 2017
- Language Models are Unsupervised Multitask Learners, the GPT-2 paper, Radford and colleagues, 2019
- The Curious Case of Neural Text Degeneration, Holtzman and colleagues, 2019, which introduced nucleus sampling
- RoFormer: Enhanced Transformer with Rotary Position Embedding, Su and colleagues, 2021
- GLU Variants Improve Transformer, Shazeer, 2020
- PyTorch scaled_dot_product_attention and Hugging Face text generation documentation
Key takeaways
- A GPT-style text generator is token and position embeddings, a stack of pre-norm decoder blocks (causal attention plus an MLP, each with a residual connection), and a linear head that scores the vocabulary.
- Training uses windows of text with targets shifted by one position, so each window yields a prediction at every position; the causal mask keeps this honest.
- A freshly initialized model should report a loss near ln(vocabulary size), which makes a quick check on your setup.
- AdamW, warmup with cosine decay and gradient clipping are the standard training recipe; watch validation loss to catch memorization.
- Generation samples one token at a time from the last position’s logits, shaped by temperature, top-k and top-p.
- A KV cache stores past keys and values so each step processes only the newest token; serving systems split work into prefill and decode around it.
- Modern LLMs keep this structure and swap in RMSNorm, RoPE, SwiGLU, grouped-query attention and subword tokenizers.
Part 9 of 9 · Deep Dive into Generative Models
- 01Mathematical Foundations of VAEs
- 02Implementing VAEs: A Step-by-Step Guide
- 03Training GANs: Tips, Tricks, and Common Pitfalls
- 04Conditional GANs: Controlling the Output
- 05StyleGAN: Generating High-Resolution Images
- 06CycleGAN: Image-to-Image Translation
- 07Text Generation Models: From RNNs to Transformers
- 08The Transformer Architecture: Self-Attention Explained
- 09Implementing Transformers for Text Generation