Training GANs: Tips, Tricks, and Common Pitfalls

A VAE that trains badly still gives you a loss curve that goes down. A GAN gives you two curves that bounce around each other, and they can look the same whether the model is learning, stuck or collapsed onto a single output. Most of the difficulty in training GANs comes from that: the numbers you would normally watch do not tell you much, so you have to know which other signals to look at and which fixes address which failure.

This article covers the practical side. It starts with a baseline that usually trains, explains the handful of bugs that break most first attempts, then works through the common failure modes and the stabilization techniques that address them. If the generator and discriminator are new to you, read the introduction to GANs first. The code uses PyTorch and follows the same style as implementing VAEs, the previous article in this series.

Why GAN training is hard

A normal network minimizes one loss over fixed data. A GAN trains two networks against each other. The discriminator D learns to tell real samples from generated ones, and the generator G learns to produce samples D accepts as real. Each network’s loss depends on the other’s current weights, so the target each one aims at keeps moving.

Three consequences follow:

• There is no single number to minimize. The generator’s loss going up can mean it got worse or that the discriminator got better. Neither curve on its own measures sample quality.
• The game can cycle instead of converging. Gradient descent on a two-player game does not always settle at an equilibrium. The networks can chase each other indefinitely, with sample quality rising and falling.
• One side can win. If the discriminator becomes too confident, its feedback to the generator carries almost no gradient. If the generator finds a single output that fools the current discriminator, it can stop exploring.

One training step alternates the two updates:

flowchart TD
    A["Sample real batch x"] --> C["D scores real and fake"]
    B["Sample noise z, fake = G(z)"] --> C
    C --> D["D loss: real as 1, fake as 0"]
    D --> E["Update D only"]
    E --> F["D scores fake again"]
    F --> G["G loss: fake as 1"]
    G --> H["Update G only"]
    H --> A

The discriminator step uses the fake batch with its link to the generator cut, so only D’s weights change. The generator step passes gradients back through D into G, but only G’s optimizer applies them.

Start from a baseline that works

The guidelines from the DCGAN paper by Radford, Metz and Chintala are still the right starting point for small image GANs. They were found by trial and error, but they remove most of the instability you hit with an arbitrary architecture:

• Use strided convolutions for downsampling in D and transposed convolutions for upsampling in G. No pooling layers.
• Use batch normalization in G and in D, except on G’s output layer and D’s input layer.
• Use ReLU in G with tanh on the output, and LeakyReLU with slope 0.2 in D.
• Initialize weights from a normal distribution with mean 0 and standard deviation 0.02.
• Train both networks with Adam at a learning rate of 0.0002 and beta1 of 0.5 instead of the default 0.9.

Here is a small DCGAN for 28 by 28 grayscale images such as MNIST:

import torch
from torch import nn
from torch.utils.data import DataLoader
from torchvision import datasets, transforms

device = "cuda" if torch.cuda.is_available() else "cpu"
z_dim = 100

transform = transforms.Compose([
    transforms.ToTensor(),                 # [0, 1]
    transforms.Normalize((0.5,), (0.5,)),  # [-1, 1], to match tanh
])
loader = DataLoader(
    datasets.MNIST("data", train=True, download=True, transform=transform),
    batch_size=128, shuffle=True, drop_last=True,
)

G = nn.Sequential(
    nn.ConvTranspose2d(z_dim, 128, 7, 1, 0, bias=False),  # 1x1 -> 7x7
    nn.BatchNorm2d(128), nn.ReLU(),
    nn.ConvTranspose2d(128, 64, 4, 2, 1, bias=False),     # 7 -> 14
    nn.BatchNorm2d(64), nn.ReLU(),
    nn.ConvTranspose2d(64, 1, 4, 2, 1),                   # 14 -> 28
    nn.Tanh(),
).to(device)

D = nn.Sequential(
    nn.Conv2d(1, 64, 4, 2, 1),                            # 28 -> 14
    nn.LeakyReLU(0.2),
    nn.Conv2d(64, 128, 4, 2, 1, bias=False),              # 14 -> 7
    nn.BatchNorm2d(128), nn.LeakyReLU(0.2),
    nn.Conv2d(128, 1, 7, 1, 0),                           # 7 -> 1, a single logit
    nn.Flatten(),
).to(device)

def weights_init(m):
    if isinstance(m, (nn.Conv2d, nn.ConvTranspose2d)):
        nn.init.normal_(m.weight, 0.0, 0.02)
    elif isinstance(m, nn.BatchNorm2d):
        nn.init.normal_(m.weight, 1.0, 0.02)
        nn.init.zeros_(m.bias)

G.apply(weights_init)
D.apply(weights_init)

opt_g = torch.optim.Adam(G.parameters(), lr=2e-4, betas=(0.5, 0.999))
opt_d = torch.optim.Adam(D.parameters(), lr=2e-4, betas=(0.5, 0.999))

Two details matter more than they look. The data is scaled to [-1, 1] because the generator ends in tanh. If you feed D real images in [0, 1] and fakes in [-1, 1], it can separate them by their range alone, and the generator learns nothing useful. And the discriminator ends in a raw logit, not a sigmoid, so the loss can apply the sigmoid in a numerically stable way.

The training step

criterion = nn.BCEWithLogitsLoss()
fixed_z = torch.randn(64, z_dim, 1, 1, device=device)   # reuse for progress grids

for epoch in range(25):
    for real, _ in loader:
        real = real.to(device)
        z = torch.randn(real.size(0), z_dim, 1, 1, device=device)
        fake = G(z)

        # Discriminator: real -> 1, fake -> 0
        opt_d.zero_grad(set_to_none=True)
        logits_real = D(real)
        logits_fake = D(fake.detach())                  # no gradient into G
        loss_d = (criterion(logits_real, torch.ones_like(logits_real))
                  + criterion(logits_fake, torch.zeros_like(logits_fake)))
        loss_d.backward()
        opt_d.step()

        # Generator: non-saturating loss, wants D to say 1 on fakes
        opt_g.zero_grad(set_to_none=True)
        logits_fake = D(fake)
        loss_g = criterion(logits_fake, torch.ones_like(logits_fake))
        loss_g.backward()
        opt_g.step()

The generator’s backward pass also writes gradients into D’s parameters. That is harmless here because opt_d.zero_grad clears them before the next discriminator update, but it is one reason to keep the two zero_grad calls exactly where they are.

The non-saturating loss

The original GAN objective has the generator minimize log(1 − D(G(z))). Early in training the discriminator rejects fakes easily, D(G(z)) is close to 0, and that function is almost flat there, so the generator gets tiny gradients exactly when it needs large ones. Goodfellow and colleagues pointed this out in the original paper and suggested the fix used above: have the generator maximize log D(G(z)) instead, which in code means labelling fakes as real in the generator’s loss. The fixed point is the same, but the gradient is strong when the generator is losing.

Use the non-saturating form by default. Writing the generator loss as the negative of the discriminator’s loss on fakes is a common mistake that brings the saturation problem back.

Bugs that break most first attempts

Before looking at instability, rule out plain bugs. These account for a large share of GANs that “don’t train”:

• Data range mismatch. Real images in [0, 1] or [0, 255] against a tanh generator. Check real.min() and real.max() once.
• Sigmoid twice. A sigmoid at the end of D combined with BCEWithLogitsLoss squashes the logits again. Either drop the sigmoid or use BCELoss, and prefer the first.
• Missing detach in the discriminator step, or a missing zero_grad for one of the optimizers.
• One optimizer for both networks. Passing all parameters to a single Adam means each step moves both networks in the direction of whichever loss you just computed.
• Generating in training mode. Batch normalization uses batch statistics in training mode. Switch G to eval() (and back) when you produce sample grids, or compare like with like.
• Mixed real and fake batches through batch norm. Concatenating real and fake into one batch for D is a reasonable choice, but if you do it, do it consistently. Switching between mixed and separate batches changes what the normalization statistics mean.

A quick sanity check catches several of these at once. At initialization, D outputs logits near 0, so D(x) is about 0.5 for everything. The discriminator loss above should start near 2 ln 2, about 1.386, and the generator loss near ln 2, about 0.693. Values far from these at step zero point to a scaling or labelling problem.

Reading the training signals

Log more than the two losses. The averages of D’s probability on real and on fake samples, sigmoid of the logits, tell you who is winning:

with torch.no_grad():
    d_real = torch.sigmoid(logits_real).mean().item()
    d_fake = torch.sigmoid(D(fake.detach())).mean().item()

A healthy run has the discriminator ahead but not dominant: D(x) above D(G(z)), with neither pinned at 1 or 0, and losses that wobble around a level instead of trending. The most useful signal of all is a grid of images generated from the same fixed_z every few hundred steps. Because the noise is fixed, you see how the mapping changes over time, and mode collapse shows up as rows of near-identical images.

flowchart TD
    S["Samples look wrong"] --> Q1{"D loss near 0?"}
    Q1 -->|Yes| A1["D wins: weaken D, add noise or regularize"]
    Q1 -->|No| Q2{"Samples all alike?"}
    Q2 -->|Yes| A2["Mode collapse: R1, spectral norm, lower G lr"]
    Q2 -->|No| Q3{"Quality rises then falls?"}
    Q3 -->|Yes| A3["Oscillation: lower lr, EMA, checkpoints"]
    Q3 -->|No| Q4{"Grid pattern artifacts?"}
    Q4 -->|Yes| A4["Upsampling: kernel divisible by stride or resize-conv"]
    Q4 -->|No| A5["Train longer, check capacity and data"]

Common failure modes

The discriminator wins

The discriminator loss drops toward zero, D(x) sits near 1 and D(G(z)) near 0. Samples stop improving. This happens when D is much stronger than G, learns faster, or has an easy cue such as the data range mismatch above.

Fixes, roughly in order of how often they help: check the data pipeline, add a gradient penalty or spectral normalization to D, reduce D’s capacity or learning rate, and add instance noise (small Gaussian noise on both real and fake inputs to D, annealed toward zero). Training D fewer times per step is a blunt tool and rarely needed with a regularized discriminator.

Mode collapse

The generator produces a small set of outputs regardless of z: the same digit, the same face with minor changes. Partial collapse, where some classes never appear, is more common than total collapse and easier to miss. It happens because the generator only has to fool the current discriminator, and producing one convincing output does that until D catches on, at which point G can jump to another one.

The losses often look fine during collapse, which is why the fixed-noise grid matters. Remedies include regularizing D (R1, spectral norm), a lower generator learning rate, and the minibatch standard deviation layer from Progressive GAN, which gives D a feature describing how varied a batch is, so a collapsed batch becomes easy to spot.

Oscillation and divergence

Sample quality improves, degrades, then improves again, or the run is fine for hours and then falls apart. This is the cycling behaviour of the two-player game. Lower learning rates, a regularized discriminator and an exponential moving average of the generator’s weights all help. So does saving checkpoints regularly: with GANs, the last checkpoint is often not the best.

Checkerboard artifacts

A faint grid pattern across generated images usually comes from transposed convolutions whose kernel size is not divisible by the stride, which makes some output pixels receive more contributions than others. Odena, Dumoulin and Olah describe this in detail. Use a kernel size divisible by the stride, as in the baseline above (4 with stride 2), or replace transposed convolutions with nearest-neighbour upsampling followed by a regular convolution.

Memorization

With small datasets, the discriminator can memorize the training images, and the generator then learns to reproduce near-copies of them. Samples look excellent and are not new. Check by finding the nearest training image to a sample of generated ones, in pixel space or a feature space. Data augmentation designed for GANs, covered below, is the usual fix.

Stabilization techniques

Most of the progress in GAN stability since DCGAN has come from controlling the discriminator: keeping its gradients well behaved so that the generator always gets useful feedback.

Alternative losses

The Wasserstein GAN (Arjovsky, Chintala and Bottou) replaces the classifier with a critic that outputs an unbounded score and trains it to estimate the Wasserstein distance between real and generated distributions. This requires the critic to be 1-Lipschitz. The original paper enforced that by clipping weights, which works poorly; WGAN-GP (Gulrajani and colleagues) replaced clipping with a gradient penalty on points interpolated between real and fake samples:

def gradient_penalty(critic, real, fake):
    eps = torch.rand(real.size(0), 1, 1, 1, device=real.device)
    mixed = (eps * real + (1 - eps) * fake.detach()).requires_grad_(True)
    scores = critic(mixed)
    grads, = torch.autograd.grad(scores.sum(), mixed, create_graph=True)
    return ((grads.flatten(1).norm(2, dim=1) - 1) ** 2).mean()

# critic step
loss_c = critic(fake.detach()).mean() - critic(real).mean() \
         + 10.0 * gradient_penalty(critic, real, fake)
# generator step
loss_g = -critic(fake).mean()

The paper uses a penalty weight of 10, five critic steps per generator step, and Adam with beta1 of 0 and beta2 of 0.9. It also advises against batch normalization in the critic, because the penalty is computed per sample and batch norm mixes samples together; layer normalization or no normalization works instead.

The hinge loss, used in SAGAN and BigGAN, is another common choice, usually paired with spectral normalization. In practice the regularizer matters more than which of these losses you pick.

Spectral normalization

Spectral normalization (Miyato and colleagues) divides each layer’s weight by an estimate of its largest singular value, which bounds how sharply the discriminator can change with its input. It costs one power iteration per layer per step and needs almost no tuning. In PyTorch, wrap each layer of D:

from torch.nn.utils.parametrizations import spectral_norm

D = nn.Sequential(
    spectral_norm(nn.Conv2d(1, 64, 4, 2, 1)), nn.LeakyReLU(0.2),
    spectral_norm(nn.Conv2d(64, 128, 4, 2, 1)), nn.LeakyReLU(0.2),
    spectral_norm(nn.Conv2d(128, 1, 7, 1, 0)), nn.Flatten(),
).to(device)

When you use spectral normalization, drop batch normalization from D. The older torch.nn.utils.spectral_norm still exists, but the PyTorch documentation marks the parametrizations version as its replacement.

R1 gradient penalty

R1 (Mescheder, Geiger and Nowozin) penalizes the squared gradient of D’s output with respect to real inputs only. It keeps the discriminator flat around the real data, which the authors show helps training converge instead of circling. It is the regularizer used in the StyleGAN family.

real.requires_grad_(True)
logits_real = D(real)
grads, = torch.autograd.grad(logits_real.sum(), real, create_graph=True)
r1 = grads.pow(2).flatten(1).sum(1).mean()
loss_d = loss_d + (gamma / 2) * r1

The weight gamma depends on the dataset and resolution, so treat it as a hyperparameter and try a few values an order of magnitude apart. StyleGAN2 also showed that applying R1 only every 16 discriminator steps, with the weight scaled up to match, saves compute with little effect on results.

Learning rates and update ratios

The two time-scale update rule (Heusel and colleagues, the same paper that introduced FID) gives D and G different learning rates, typically a higher one for D, instead of training D several times per generator step. SAGAN used 0.0004 for D and 0.0001 for G. With a regularized discriminator this is usually simpler and faster than tuning an update ratio.

Generator weight averaging

Keep an exponential moving average of the generator’s weights and generate from the average, not the live network. The live generator jitters from step to step as it chases the discriminator; the average smooths that out, and it was standard from Progressive GAN onward.

import copy

g_ema = copy.deepcopy(G).eval().requires_grad_(False)

@torch.no_grad()
def update_ema(decay=0.999):
    for p_ema, p in zip(g_ema.parameters(), G.parameters()):
        p_ema.lerp_(p, 1 - decay)
    for b_ema, b in zip(g_ema.buffers(), G.buffers()):
        b_ema.copy_(b)

Call update_ema() after each generator step. The EMA model is not trained and adds no cost beyond the extra copy of the weights.

Augmentation for small datasets

Normal augmentation applied only to real images teaches the generator to produce augmented-looking images. Adaptive discriminator augmentation (ADA, Karras and colleagues) and differentiable augmentation (DiffAugment, Zhao and colleagues) instead apply the same augmentations to both real and fake images before D sees them, in a way gradients can pass through. ADA also adjusts the augmentation strength during training based on how much D is overfitting. Both make GANs trainable on a few thousand images, where they would otherwise memorize.

Older tricks

Salimans and colleagues collected several techniques in “Improved Techniques for Training GANs”: feature matching, minibatch discrimination, historical averaging and one-sided label smoothing (training D toward 0.9 instead of 1 on real images, while keeping fakes at 0). Label smoothing is still a cheap thing to try on a plain DCGAN. Most of the others have been superseded by the regularizers above.

Evaluating progress

Image grids are essential but subjective. For a number to compare runs, the Fréchet Inception Distance (FID) compares statistics of real and generated images in the feature space of a pretrained Inception network; lower is better. It is sensitive to sample count and implementation details, so only compare FIDs computed the same way, with the same number of samples, and on the same reference set. A later article in this series covers evaluation metrics in depth.

Practical habits that save time:

• Compute FID on a schedule, using the EMA generator, and keep the checkpoint with the best score rather than the last one.
• Save both networks and both optimizers’ states in each checkpoint. Resuming with fresh Adam statistics can destabilize a run that was going well.
• Fix the random seed for the evaluation noise so that numbers from different checkpoints are comparable.
• Change one thing at a time. GAN runs are noisy enough that two simultaneous changes are hard to attribute, and a single run can mislead; if a change matters, it should hold across a couple of seeds.

If you train larger models on cloud GPUs, the same habits apply, with the added reason that preemptible or spot instances can stop at any time. The AWS and GCP articles in this track cover the managed training options, and deep learning basics covers the learning rate behaviour that shows up here in exaggerated form.

A debugging order

When a GAN does not train, work through these in order and stop when samples improve:

1. Check the data range and the generator’s output activation agree.
2. Check the step-zero losses are near 1.386 for D and 0.693 for G with the BCE setup above.
3. Confirm the detach, both zero_grad calls and two separate optimizers.
4. Train on a small subset and confirm the samples start to resemble the data at all.
5. Return to the DCGAN defaults: architecture rules, initialization, Adam with beta1 of 0.5.
6. Add spectral normalization or R1 to the discriminator.
7. Add generator EMA and pick checkpoints by FID.
8. For small datasets, add ADA or DiffAugment.

Steps 1 to 5 fix most first attempts. Steps 6 to 8 are what modern GANs use to train reliably at scale.

Further reading

• Generative Adversarial Nets, Goodfellow and colleagues: the original paper and the non-saturating loss.
• Unsupervised Representation Learning with Deep Convolutional GANs, Radford, Metz and Chintala: the DCGAN guidelines.
• Improved Training of Wasserstein GANs, Gulrajani and colleagues: WGAN-GP.
• Spectral Normalization for GANs, Miyato and colleagues.
• Which Training Methods for GANs do actually Converge?, Mescheder, Geiger and Nowozin: R1.
• GANs Trained by a Two Time-Scale Update Rule, Heusel and colleagues: TTUR and FID.
• Training GANs with Limited Data, Karras and colleagues: ADA.
• Deconvolution and Checkerboard Artifacts, Odena, Dumoulin and Olah.
• PyTorch DCGAN tutorial and the spectral_norm documentation.

Key takeaways

• GAN losses do not measure sample quality. Log D(x) and D(G(z)) and look at a fixed-noise image grid throughout training.
• Rule out bugs first: data range against tanh, sigmoid applied twice, missing detach or zero_grad, and a shared optimizer.
• Start from the DCGAN defaults and the non-saturating generator loss; with BCE, expect losses near 1.386 for D and 0.693 for G at step zero.
• Most failures come from the discriminator: too strong, too sharp or overfitting. Spectral normalization, WGAN-GP or R1 address this directly.
• Mode collapse can hide behind normal-looking losses, so check diversity directly.
• Generate from an exponential moving average of the generator, and pick checkpoints by FID rather than taking the last one.
• On small datasets, augment real and fake images alike with ADA or DiffAugment.