Text Generation Models: From RNNs to Transformers

Every text generator, from a 1990s toy that writes fake Shakespeare to a model answering questions in a chat window, does the same basic job: given the text so far, predict the next token, pick one, append it and repeat. What changed over three decades is how the model remembers “the text so far”. That one design decision decides how far back a model can look, how fast it trains and how much memory it needs to answer you.

This article follows that decision from recurrent neural networks (RNNs) through LSTMs, sequence-to-sequence models and attention, to the Transformer. It ends with the recent return of recurrence in state space models. The series has so far been about images, from VAEs to CycleGAN; this is where it turns to text. The next article, on self-attention, goes deeper into the mechanism the Transformer is built on.

The task every text model solves

A language model assigns a probability to a sequence of tokens. Tokens can be characters, words or, as in most modern models, subword pieces. Using the chain rule of probability, the probability of a whole sequence breaks down into a product of next-token predictions:

P(x₁, …, x_T) = P(x₁) · P(x₂ | x₁) · P(x₃ | x₁, x₂) · … · P(x_T | x₁, …, x_{T−1})

So a model that can estimate P(next token | everything before it) can both score text and generate it. Generation is a loop: sample a token from the predicted distribution, feed it back in as input, predict again. This is called autoregressive generation, and every architecture in this article uses it at inference time.

flowchart LR
    P["Prompt tokens"] --> M["Model"]
    M --> D["Next-token distribution"]
    D --> S["Sample or pick top token"]
    S --> A["Append to sequence"]
    A -->|"repeat until stop"| M

Training is simpler than generation. You take real text, feed the model each prefix and ask it to predict the token that actually came next, using cross-entropy loss. Feeding the true previous tokens rather than the model’s own guesses is called teacher forcing. The question each architecture answers differently is how to summarize a prefix of any length into something a fixed-size network can work with. The text generation overview covers the sampling side (temperature, top-p and so on); here the focus is the model.

Before neural networks: counting n-grams

The classic approach was the n-gram model. A trigram model assumes the next word depends only on the previous two, and estimates P(word | previous two words) by counting how often each triple appears in a corpus.

N-gram models are fast and easy to build, and they powered speech recognition and machine translation for years. Their limits are structural:

• Fixed, short context. A 5-gram model cannot connect a pronoun to a noun six words back.
• Sparsity. Most long word sequences never appear in the training data, so counts are zero and need smoothing tricks.
• No notion of similarity. “cat sat” and “dog sat” are unrelated counts; seeing one teaches nothing about the other.

Bengio and colleagues addressed the third problem in 2003 with a neural probabilistic language model. It learned a vector, an embedding, for each word and predicted the next word from the embeddings of the previous few. Similar words ended up with similar vectors, so knowledge transferred between them. The context was still a fixed window, though. Removing that window was the job of recurrent networks.

Recurrent neural networks

An RNN reads a sequence one token at a time and keeps a hidden state, a vector that is updated at every step. Jeffrey Elman’s 1990 “simple recurrent network” is the usual starting point. At step t:

h_t = tanh(W_h · h_{t−1} + W_x · x_t + b)

y_t = softmax(W_y · h_t)

The same weights are used at every step, so the network handles sequences of any length, and in principle h_t carries information from every earlier token. The output y_t is the distribution over the next token.

flowchart LR
    X1["x1"] --> H1["h1"]
    X2["x2"] --> H2["h2"]
    X3["x3"] --> H3["h3"]
    H0["h0"] --> H1
    H1 --> H2
    H2 --> H3
    H1 --> Y1["next token"]
    H2 --> Y2["next token"]
    H3 --> Y3["next token"]

Tomas Mikolov’s RNN language models (2010) beat strong n-gram baselines on speech recognition tasks, and in 2015 Andrej Karpathy’s char-rnn post showed character-level RNNs producing convincing fake Shakespeare, Wikipedia markup and Linux kernel C code. For many people that was the first sign that neural networks could generate text.

Why plain RNNs forget

Training an RNN uses backpropagation through time: unroll the network across the sequence and backpropagate through every step. The gradient that reaches an early step has been multiplied by the recurrent weight matrix and the tanh derivative once for each step in between. If those factors are mostly below 1, the gradient shrinks exponentially and the early tokens stop influencing learning. If they are above 1, it explodes.

Bengio, Simard and Frasconi analysed this in 1994. Exploding gradients have a simple fix, gradient clipping. Vanishing gradients do not, and in practice a plain RNN struggles to use information more than a few dozen steps back. That is a problem for text, where a sentence’s verb may depend on a subject many words earlier and a story’s ending on its first paragraph.

LSTMs and GRUs

The Long Short-Term Memory network, from Hochreiter and Schmidhuber (1997), adds a separate cell state c_t that flows through time with mostly additive updates, controlled by gates. The forget gate, added by Gers, Schmidhuber and Cummins in 2000, completed the version used today.

• The forget gate decides how much of the old cell state to keep.
• The input gate decides how much new information to write.
• The output gate decides how much of the cell state to expose as the hidden state.

Each gate is a sigmoid layer producing values between 0 and 1, computed from the current input and the previous hidden state. Because the cell state is updated by adding and scaling rather than by repeated matrix multiplication and squashing, gradients can travel along it for much longer before vanishing. LSTMs reliably handle dependencies over hundreds of steps where plain RNNs fail at tens.

The Gated Recurrent Unit (GRU), introduced by Cho and colleagues in 2014, merges the cell and hidden state and uses two gates (update and reset) instead of three. It has fewer parameters and often performs about as well as an LSTM; which one wins depends on the task.

Here is a character-level LSTM language model in PyTorch, with a sampling loop. It is close to what char-rnn did:

import torch
from torch import nn

class CharLSTM(nn.Module):
    def __init__(self, vocab_size, emb=64, hidden=256, layers=2):
        super().__init__()
        self.embed = nn.Embedding(vocab_size, emb)
        self.lstm = nn.LSTM(emb, hidden, num_layers=layers, batch_first=True)
        self.head = nn.Linear(hidden, vocab_size)

    def forward(self, x, state=None):
        out, state = self.lstm(self.embed(x), state)   # out: (batch, seq, hidden)
        return self.head(out), state                    # logits for every position

@torch.no_grad()
def generate(model, start_ids, steps, temperature=1.0):
    model.eval()
    x = torch.tensor([start_ids])
    logits, state = model(x)                  # read the whole prompt once
    out = list(start_ids)
    for _ in range(steps):
        probs = torch.softmax(logits[:, -1] / temperature, dim=-1)
        nxt = torch.multinomial(probs, 1)     # shape (1, 1)
        out.append(nxt.item())
        logits, state = model(nxt, state)     # one token in, state carried forward
    return out

Training uses teacher forcing: the input is a chunk of text and the target is the same chunk shifted one character left.

loss_fn = nn.CrossEntropyLoss()
logits, _ = model(batch[:, :-1])                            # (B, T, V)
loss = loss_fn(logits.reshape(-1, logits.size(-1)),
               batch[:, 1:].reshape(-1))

Notice the generation loop. After the prompt, each step feeds in one token and the fixed-size state. Memory use during generation does not grow with the length of the text. Keep that in mind; it comes back at the end.

Sequence to sequence and the bottleneck

Generating text from scratch is one task. Many useful tasks generate text from other text: translation, summarization, question answering. In 2014 Sutskever, Vinyals and Le, and separately Cho and colleagues, proposed the encoder-decoder (sequence-to-sequence) design:

1. An encoder RNN reads the source sentence and ends with a final hidden state.
2. That single vector initializes a decoder RNN, which generates the target sentence token by token.

Sutskever’s system used deep LSTMs and was competitive with phrase-based statistical translation on English to French. One of its practical tricks was to reverse the order of the source sentence, which put the first source words close to the first target words and made optimization easier.

The weakness is the hand-off. The entire source sentence, however long, has to fit into one fixed-size vector. Translation quality dropped as sentences grew longer, which is exactly what you would expect from a bottleneck.

Attention: let the decoder look back

Bahdanau, Cho and Bengio (2014) removed the bottleneck. Instead of passing only the final encoder state, keep every encoder hidden state. At each decoding step, the decoder scores each encoder state for relevance to what it is about to produce, turns the scores into weights with a softmax, and takes the weighted sum as a context vector. When producing the French word for “cat”, the weights concentrate on the encoder state for “cat”.

flowchart TD
    subgraph Encoder["Encoder RNN"]
        E1["h1"] --> E2["h2"] --> E3["h3"] --> E4["h4"]
    end
    S["Decoder state s_t"] --> SC["Score each h_i against s_t"]
    E1 --> SC
    E2 --> SC
    E3 --> SC
    E4 --> SC
    SC --> W["Softmax weights"]
    W --> C["Context: weighted sum of h_i"]
    C --> D["Decoder predicts next token"]
    S --> D

Luong, Pham and Manning (2015) simplified the scoring functions, including a plain dot product between decoder and encoder states. Attention-based LSTM encoder-decoders became the standard for neural machine translation; Google’s production translation system in 2016 (GNMT) was a deep LSTM encoder-decoder with attention.

Attention also gave a side benefit: you could plot the weights and see which source words the model used for each output word. But the recurrence was still there. The encoder and decoder both processed tokens one step at a time, and attention had been added on top of them. The next step was to ask whether the recurrence was needed at all.

The Transformer: attention without recurrence

“Attention Is All You Need” (Vaswani and colleagues, 2017) answered that it was not. The Transformer keeps the encoder-decoder shape but builds both halves only from attention and feed-forward layers:

• Self-attention lets every token in a sequence attend to every other token in the same sequence, so each position builds its representation directly from all the others, one hop away, rather than through a chain of hidden states.
• Multi-head attention runs several attention operations in parallel so different heads can track different relationships.
• Positional encodings are added to the token embeddings, because attention by itself has no idea of word order.
• Residual connections and layer normalization around each sub-layer keep deep stacks trainable.
• In the decoder, a causal mask stops each position from attending to later positions, so the model cannot see the token it is meant to predict.

The headline result was on translation. The large Transformer reached 28.4 BLEU on the WMT 2014 English to German benchmark, better than earlier models including ensembles, after 3.5 days of training on eight P100 GPUs. The base model trained in about 12 hours on the same hardware. The speed was as important as the score.

Why it trained so much faster

In an RNN, step t needs h_{t−1}, so the steps of a sequence must be computed in order even during training. The Transformer has no such chain. With teacher forcing, the whole target sequence is known in advance, so the model computes predictions for every position at once, as a few large matrix multiplications. GPUs are built for exactly that.

The path between any two tokens is also short. In an RNN, information from token 1 reaches token 500 through 499 state updates. In a Transformer layer it is a single attention step. Long-range dependencies stop being an optimization problem and become a question of whether the context window is long enough.

The cost is that self-attention compares every token with every other token, so compute and memory for attention grow with the square of the sequence length. Doubling the context roughly quadruples the attention work.

Here is a minimal decoder-only Transformer language model in PyTorch, using the built-in layers. The causal mask is the important line:

import torch
from torch import nn

class TinyGPT(nn.Module):
    def __init__(self, vocab_size, d_model=256, heads=4, layers=4, max_len=512):
        super().__init__()
        self.tok = nn.Embedding(vocab_size, d_model)
        self.pos = nn.Embedding(max_len, d_model)       # learned positions
        block = nn.TransformerEncoderLayer(d_model, heads, 4 * d_model,
                                           batch_first=True, norm_first=True)
        self.blocks = nn.TransformerEncoder(block, layers)
        self.head = nn.Linear(d_model, vocab_size)

    def forward(self, x):                                # x: (batch, seq)
        T = x.size(1)
        h = self.tok(x) + self.pos(torch.arange(T, device=x.device))
        mask = nn.Transformer.generate_square_subsequent_mask(T, device=x.device)
        h = self.blocks(h, mask=mask, is_causal=True)    # no peeking ahead
        return self.head(h)                              # logits for every position

It uses nn.TransformerEncoderLayer because a decoder-only language model has no encoder to attend to; with a causal mask, an “encoder” layer is exactly a GPT-style block. The training step is the same as the LSTM’s: shift by one and apply cross-entropy. What differs is inside the model, where all positions are processed together.

Encoders, decoders and the GPT line

The original Transformer was an encoder-decoder for translation. Researchers soon split it:

• Decoder-only models keep the causally masked stack and train on next-token prediction over large amounts of unlabelled text. OpenAI’s GPT (2018) was a 12-layer decoder-only Transformer; GPT-2 (2019) scaled it to 1.5 billion parameters and GPT-3 (2020) to 175 billion. Most text generators today follow this pattern, as covered in Exploring OpenAI’s GPT Models.
• Encoder-only models such as BERT (2018) drop the causal mask and train by filling in masked words. They read text very well but are not built to generate it.
• Encoder-decoder models such as T5 keep both halves and suit tasks with a clear input and output.

A later article in this series compares BERT and GPT in detail. For generation, the decoder-only design won mainly because it is simple and scales: one stack, one objective, and any text on the internet is training data.

The three generations side by side

The cost of generation, and recurrence returning

Look at the last two rows of the table again. The Transformer won on training, which is where the biggest cost used to be. Generation is a different story. Producing each new token still requires attending over every previous token. Implementations avoid recomputing old work by storing each layer’s attention keys and values for all earlier tokens, the key-value (KV) cache. That cache grows linearly with the length of the conversation, and it is a large share of the memory a model server needs, which is part of why long-context requests cost more.

An RNN’s state, by contrast, stays the same size no matter how much text it has read. That property has brought recurrence back in new forms that try to keep the Transformer’s parallel training while running like an RNN at inference:

• State space models. Mamba (Gu and Dao, 2023) uses a selective state space layer whose parameters depend on the input, so the model chooses what to keep or forget at each token. It trains with a hardware-aware parallel scan and generates with a fixed-size state, scaling linearly with sequence length.
• Modernized RNNs. RWKV and xLSTM revisit recurrent and LSTM-style designs with changes that make training parallel or efficient at scale.
• Hybrids. Several recent models interleave attention layers with state space or recurrent layers, keeping some attention for precise recall over the context while cutting the KV cache.

None of these has displaced attention in the largest models; hybrids are where most of the practical activity is. The broader point is that the trade-off this article traced, between compressing the past into a state and keeping all of it around, has not been settled. It has just moved from training cost to serving cost.

flowchart LR
    NG["N-grams: count"] --> NN["Neural LM: embeddings"]
    NN --> RNN["RNN: hidden state"]
    RNN --> LSTM["LSTM / GRU: gated memory"]
    LSTM --> S2S["Seq2seq + attention"]
    S2S --> TF["Transformer: attention only"]
    TF --> HY["Hybrids: attention + state space"]

Running these models in the cloud

You will rarely train a large text model from scratch. The practical choices are:

• Call a hosted model. Amazon Bedrock, Microsoft Foundry and Google Cloud’s Gemini Enterprise Agent Platform (formerly Vertex AI) all serve Transformer-based text models through an API. Comparing Cloud Platforms for Generative AI walks through the differences.
• Fine-tune an existing model on your own data, covered later in this series.
• Train a small model yourself to learn how it works. The two PyTorch models above train on a single GPU in a notebook on any of the three clouds, and a character-level model on a few megabytes of text produces recognisable output within an hour or so of training.

If you train both on the same corpus with similar parameter counts, you will see the trade-offs directly: the Transformer reaches a lower loss in less wall-clock time, while the LSTM’s generation loop uses a constant amount of memory.

Further reading

• Attention Is All You Need, Vaswani and colleagues, 2017
• Neural Machine Translation by Jointly Learning to Align and Translate, Bahdanau, Cho and Bengio
• Sequence to Sequence Learning with Neural Networks, Sutskever, Vinyals and Le
• Mamba: Linear-Time Sequence Modeling with Selective State Spaces, Gu and Dao
• PyTorch nn.LSTM and nn.TransformerEncoderLayer documentation
• On this site: The Role of Transformers in Generative AI and Understanding Neural Networks

Key takeaways

• Every model here generates text the same way, by predicting the next token and feeding it back in; they differ in how they represent the text so far.
• N-gram models count short windows; neural language models added embeddings so similar words share knowledge.
• RNNs carry a hidden state across any length of text but suffer from vanishing gradients; LSTMs and GRUs use gates to keep information over hundreds of steps.
• Encoder-decoder models squeezed the source into one vector; attention removed that bottleneck by letting the decoder look at every encoder state.
• The Transformer dropped recurrence entirely, which made training parallel and put every pair of tokens one step apart, at the price of attention cost that grows with the square of the length.
• At generation time the Transformer’s key-value cache grows with the text, which is why fixed-state designs such as Mamba and hybrid models are getting renewed attention.