Conditional GANs: Controlling the Output

A GAN trained on handwritten digits will give you a convincing digit for every noise vector you feed it, but you have no say in which digit. To get a 7, you sample until one appears. That is fine for a demo and useless for most real work, where you want a particular class, a particular attribute, or an output that corresponds to a specific input image.

A conditional GAN fixes this by giving both networks an extra input: the thing you want the output to match. This article covers how that conditioning works, the main ways of wiring it into the generator and the discriminator, and how the same idea scales from digit labels to image-to-image translation. It builds on the training setup from Training GANs: Tips, Tricks, and Common Pitfalls, and the code reuses that article’s MNIST baseline. If the generator and discriminator are new to you, start with the introduction to GANs.

From an unconditional to a conditional GAN

An unconditional generator learns a mapping from noise to data, G(z), so that its outputs look like samples from the data distribution p(x). It models the whole distribution at once and gives you no handle on which part of it you get.

A conditional generator takes a second input y and learns G(z, y), so that its outputs look like samples from p(x | y): the distribution of images given a label, a text description or another image. The noise z still supplies the variation. For the label 7, different z values give different styles of 7.

The discriminator is conditioned too. It sees the pair (x, y) and has to decide whether x is a real example of y. This is the part people tend to miss. If only the generator received the label, the discriminator would judge realism alone, and the generator could ignore y completely and still win. Because the discriminator rejects a realistic 3 offered as a 7, the generator is pushed to respect the condition.

flowchart LR
    Z["Noise z"] --> G["Generator"]
    Y["Condition y"] --> G
    G --> F["Fake x"]
    F --> D["Discriminator"]
    Y --> D
    R["Real x with its y"] --> D
    D --> O["Real and matching, or not"]

Mirza and Osindero introduced this setup in “Conditional Generative Adversarial Nets” (2014). The objective is the original GAN objective with y added to both networks:

min over G, max over D of E[log D(x, y)] + E[log(1 − D(G(z, y), y))]

The expectations run over real pairs (x, y) from the data, and over noise z with labels y drawn from the label distribution. In practice you train with the non-saturating generator loss, exactly as in the unconditional case.

What counts as a condition

Anything you can turn into a tensor can serve as y. The architecture for feeding it in depends on its shape.

Feeding the condition to the generator

Concatenation with the noise

The simplest method, used in the original cGAN paper, is to join the condition to z before the first layer. For a class label you can concatenate a one-hot vector, but a learned embedding works better: it lets the network decide how classes relate to each other, and it costs nothing extra.

Here is the generator from the previous article’s DCGAN, with a class embedding concatenated to the noise:

import torch
from torch import nn

device = "cuda" if torch.cuda.is_available() else "cpu"
z_dim, n_classes, emb_dim = 100, 10, 50

class CondGenerator(nn.Module):
    def __init__(self):
        super().__init__()
        self.embed = nn.Embedding(n_classes, emb_dim)
        self.net = nn.Sequential(
            nn.ConvTranspose2d(z_dim + emb_dim, 128, 7, 1, 0, bias=False),  # 1 -> 7
            nn.BatchNorm2d(128), nn.ReLU(),
            nn.ConvTranspose2d(128, 64, 4, 2, 1, bias=False),               # 7 -> 14
            nn.BatchNorm2d(64), nn.ReLU(),
            nn.ConvTranspose2d(64, 1, 4, 2, 1),                             # 14 -> 28
            nn.Tanh(),
        )

    def forward(self, z, y):
        h = torch.cat([z, self.embed(y)], dim=1)   # (B, z_dim + emb_dim)
        return self.net(h[:, :, None, None])        # reshape to (B, C, 1, 1)

Concatenation works well for small models. Its weakness is that the condition enters once, at the bottom of the network, and the deeper layers have to carry it forward on their own.

Conditional normalization

Larger class-conditional GANs feed the label into every block instead, through the normalization layers. A batch normalization layer normalizes its input and then applies a learned scale and shift. Conditional batch normalization makes that scale and shift depend on the class, so each class gets its own per-channel modulation of the features at every depth.

class CondBatchNorm2d(nn.Module):
    def __init__(self, num_features, n_classes):
        super().__init__()
        self.bn = nn.BatchNorm2d(num_features, affine=False)   # normalize only
        self.gamma = nn.Embedding(n_classes, num_features)
        self.beta = nn.Embedding(n_classes, num_features)
        nn.init.ones_(self.gamma.weight)                        # start as identity
        nn.init.zeros_(self.beta.weight)

    def forward(self, x, y):
        out = self.bn(x)
        gamma = self.gamma(y)[:, :, None, None]
        beta = self.beta(y)[:, :, None, None]
        return gamma * out + beta

The idea comes from conditional instance normalization for style transfer (Dumoulin, Shlens and Kudlur) and conditional batch normalization for vision and language (de Vries and colleagues). SAGAN and BigGAN use it in their generators. BigGAN goes a step further: one shared class embedding, combined with a slice of the noise vector, is projected by a small linear layer into the gains and biases of each block, instead of every layer keeping its own embedding table. That saves parameters and lets the noise influence every resolution too.

SPADE (Park and colleagues) applies the same principle to segmentation maps: the scale and shift are predicted per pixel from the map by a small convolutional network, so the layout is reinjected at each resolution rather than washed out by repeated normalization.

Feeding the condition to the discriminator

The discriminator side has seen more variation, and the choice matters more for quality. There are three main designs.

flowchart TD
    subgraph A["Input concatenation"]
        A1["x and y"] --> A2["Shared network"] --> A3["Real or fake"]
    end
    subgraph B["Auxiliary classifier"]
        B1["x only"] --> B2["Shared network"]
        B2 --> B3["Real or fake"]
        B2 --> B4["Predicted class"]
    end
    subgraph C["Projection"]
        C1["x"] --> C2["Features phi(x)"]
        C2 --> C3["Unconditional score"]
        C2 --> C4["Inner product with embed(y)"]
        C3 --> C5["Sum"]
        C4 --> C5
    end

Input concatenation

The original cGAN concatenates y with x, or with an early hidden layer. For images, a class label is usually broadcast to a set of constant feature maps, one per class or embedding dimension, and stacked with the image channels. It is easy to write and works on MNIST-scale problems, but the network has to learn how to use the condition from scratch, and on harder datasets it tends to be the weakest option.

Auxiliary classifier (AC-GAN)

AC-GAN (Odena, Olah and Shlens, 2017) does not give the label to the discriminator at all. Instead, the discriminator has two heads on a shared body: one predicts real or fake, the other predicts the class. Both real and generated images are trained to be classified correctly, so the generator is rewarded for making images whose class is obvious.

bce = nn.BCEWithLogitsLoss()
ce = nn.CrossEntropyLoss()

# Discriminator step: D returns (source_logit, class_logits)
src_real, cls_real = D(real)
src_fake, cls_fake = D(fake.detach())
loss_d = (bce(src_real, torch.ones_like(src_real))
          + bce(src_fake, torch.zeros_like(src_fake))
          + ce(cls_real, y_real) + ce(cls_fake, y_fake))

# Generator step: look real, and be classified as the requested label
src_fake, cls_fake = D(fake)
loss_g = bce(src_fake, torch.ones_like(src_fake)) + ce(cls_fake, y_fake)

The classifier gives the generator a strong, direct signal about the class, which helped AC-GAN produce recognizable ImageNet samples at 128 by 128 when that was hard. The cost is diversity. A generator rewarded for easy-to-classify images drifts toward the most typical examples of each class and away from the ambiguous ones near class boundaries. Later work noted this tendency, and several follow-ups modified the AC-GAN loss to counter it.

Projection discriminator

The projection discriminator (Miyato and Koyama, 2018) is the design most large class-conditional GANs settled on, including BigGAN. The discriminator computes a feature vector φ(x) from the image. Its output is an unconditional score plus the inner product between those features and a learned embedding of the label:

D(x, y) = ψ(φ(x)) + embed(y) · φ(x)

The authors derived this form from the structure of the optimal discriminator, under the assumption that the label, given the image features, follows a simple log-linear model. In plain terms, the network first decides how realistic the image is, then adds a term measuring how well its features line up with the requested class. Here is a projection discriminator for MNIST, with spectral normalization as recommended in the previous article:

from torch.nn.utils.parametrizations import spectral_norm

class ProjDiscriminator(nn.Module):
    def __init__(self):
        super().__init__()
        self.features = nn.Sequential(
            spectral_norm(nn.Conv2d(1, 64, 4, 2, 1)), nn.LeakyReLU(0.2),    # 28 -> 14
            spectral_norm(nn.Conv2d(64, 128, 4, 2, 1)), nn.LeakyReLU(0.2),  # 14 -> 7
        )
        self.linear = spectral_norm(nn.Linear(128, 1))
        self.embed = spectral_norm(nn.Embedding(n_classes, 128))

    def forward(self, x, y):
        h = self.features(x).sum(dim=(2, 3))                    # global sum pool -> (B, 128)
        out = self.linear(h).squeeze(1)                         # unconditional term
        return out + (self.embed(y) * h).sum(dim=1)             # projection term

Compared with AC-GAN, the projection term is part of the real-or-fake decision rather than a separate classification task, so the generator is not pushed toward stereotypical samples in the same way.

Training a class-conditional GAN

The training loop barely changes from the unconditional one. Real images come with their labels from the data loader. For fakes, you draw labels at random, generate with them, and pass the same labels to the discriminator. Using the projection discriminator and the generator above:

from torch.utils.data import DataLoader
from torchvision import datasets, transforms

transform = transforms.Compose([transforms.ToTensor(), transforms.Normalize((0.5,), (0.5,))])
loader = DataLoader(datasets.MNIST("data", train=True, download=True, transform=transform),
                    batch_size=128, shuffle=True, drop_last=True)

G, D = CondGenerator().to(device), ProjDiscriminator().to(device)
opt_g = torch.optim.Adam(G.parameters(), lr=2e-4, betas=(0.5, 0.999))
opt_d = torch.optim.Adam(D.parameters(), lr=2e-4, betas=(0.5, 0.999))
criterion = nn.BCEWithLogitsLoss()

for epoch in range(25):
    for real, y_real in loader:
        real, y_real = real.to(device), y_real.to(device)
        b = real.size(0)
        z = torch.randn(b, z_dim, device=device)
        y_fake = torch.randint(0, n_classes, (b,), device=device)
        fake = G(z, y_fake)

        opt_d.zero_grad(set_to_none=True)
        logits_real = D(real, y_real)
        logits_fake = D(fake.detach(), y_fake)
        loss_d = (criterion(logits_real, torch.ones_like(logits_real))
                  + criterion(logits_fake, torch.zeros_like(logits_fake)))
        loss_d.backward()
        opt_d.step()

        opt_g.zero_grad(set_to_none=True)
        logits_fake = D(fake, y_fake)
        loss_g = criterion(logits_fake, torch.ones_like(logits_fake))
        loss_g.backward()
        opt_g.step()

Draw fake labels from the same distribution as the real ones. MNIST is close to balanced, so uniform sampling works. On an imbalanced dataset, uniform labels ask the generator for rare classes far more often than the discriminator sees real examples of them, and quality on those classes suffers.

Everything from the previous article still applies: weights initialized as in DCGAN if you keep batch norm, the step-zero loss check, a fixed-noise grid, generator EMA and checkpoints chosen by FID. The one change to monitoring is that diversity now has to be checked per class. A model can cover all ten digits and still collapse within one of them.

Checking that it follows the label

The most useful grid for a conditional model holds the noise fixed along each row and varies the label along each column:

from torchvision.utils import save_image

G.eval()
with torch.no_grad():
    z = torch.randn(10, z_dim, device=device).repeat_interleave(10, dim=0)  # each z ten times
    y = torch.arange(10, device=device).repeat(10)                          # 0..9 in every row
    samples = G(z, y)
save_image(samples * 0.5 + 0.5, "grid.png", nrow=10)
G.train()

Each column should be one digit throughout. Each row should share a style, such as stroke thickness or slant, because the same z produced it. That second property is the payoff of conditioning: the label controls content and the noise controls everything else, so you can change one without disturbing the other. If rows look unrelated, the generator is using the label for style as well; if columns contain the wrong digits, the discriminator is not enforcing the condition.

For a number, train a separate classifier on the real data and measure how often it assigns generated samples to the label they were generated with. High accuracy alone is not enough, since AC-GAN-style collapse also scores well, so pair it with FID computed per class or over a class-balanced sample.

Image-to-image translation with pix2pix

When the condition is itself an image, the generator becomes a translator. pix2pix (Isola, Zhu, Zhou and Efros, 2017) learns mappings such as edges to photos, labels to street scenes, and day to night, from paired examples.

flowchart LR
    X["Input image x"] --> G["U-Net generator"]
    G --> F["Output G(x)"]
    X --> P1["Stack x with G(x)"]
    F --> P1
    X --> P2["Stack x with real y"]
    Y["Target image y"] --> P2
    P1 --> D["PatchGAN discriminator"]
    P2 --> D
    F --> L["L1 loss to y"]
    Y --> L

Three choices define it:

• The generator is a U-Net. An encoder-decoder with skip connections between mirrored layers, so low-level structure such as edge positions passes straight from input to output instead of being squeezed through a bottleneck.
• The discriminator sees the input and the output together, stacked along the channel dimension. It is the same conditioning principle as before: it judges whether this output is a plausible translation of this input.
• The discriminator is a PatchGAN. It outputs a grid of real-or-fake scores, each covering a patch of the image (70 by 70 pixels in the paper’s default), and the loss averages them. It enforces realistic local texture and leaves global structure to the L1 term.

The generator loss adds an L1 distance to the ground truth, weighted at 100 in the paper:

l1 = nn.L1Loss()

pred_fake = D(torch.cat([x, fake], dim=1))           # (B, 1, H', W') patch logits
loss_g = (criterion(pred_fake, torch.ones_like(pred_fake))
          + 100.0 * l1(fake, y_target))

The two terms divide the work. L1 alone produces blurry outputs because averaging over plausible answers minimizes it. The adversarial term alone produces sharp outputs that may not match the target. Together they give sharp images aligned with the input.

The authors also reported that the generator learned to ignore an explicit noise input, so pix2pix gets its limited randomness from dropout, which stays on at test time. The model is close to deterministic as a result: one input gives essentially one output. When you need several plausible outputs for the same input, or do not have paired data, other designs fit better. CycleGAN, the subject of a later article in this series, removes the need for pairs.

Text and other rich conditions

Reed and colleagues’ “Generative Adversarial Text to Image Synthesis” (2016) conditioned a GAN on sentence embeddings of captions. Its most reusable idea is the matching-aware discriminator. Besides real images with their captions and fakes with theirs, the discriminator also sees real images paired with the wrong caption, labelled as fake. Without that third case, the discriminator can learn to spot fakes by image quality alone and never check the text. The same trick works for any rich condition where “realistic but mismatched” is a failure you care about.

Text-to-image work has since moved largely to diffusion models and transformers, described in Generative AI in Image and Video Synthesis. The conditioning ideas carried over: a learned embedding of the condition, injected through normalization or attention at many depths, is how modern image generators follow a prompt too.

Conditioning without labels

Sometimes you want control without labelled data. InfoGAN (Chen and colleagues, 2016) splits the generator’s input into noise and a small latent code c, and adds a term that maximizes the mutual information between c and the generated image, estimated by an auxiliary network that predicts c from the output. On MNIST, a 10-way categorical code learned to correspond largely to digit identity and continuous codes to rotation and width, without labels. The result is not guaranteed: which factors the codes capture depends on the data and the run, so treat it as discovery rather than control.

The other route is to train unconditionally and find directions in the latent space afterwards, which works especially well with the StyleGAN family covered in the next article. Conditional VAEs, a close relative, add y to both the encoder and decoder of a variational autoencoder and are often easier to train when sharpness matters less.

Choosing an approach

• Class labels, small model: concatenate an embedding with z in the generator and use a projection discriminator. It is barely more code than input concatenation and works better.
• Class labels, large model: conditional batch norm in the generator, projection discriminator, spectral normalization, as in BigGAN.
• Paired image translation: pix2pix, with a U-Net generator, a PatchGAN discriminator and an L1 term.
• Dense layouts: SPADE-style spatially varying normalization.
• Captions or other rich inputs: embed the condition, and train the discriminator on mismatched pairs as well.
• No labels: InfoGAN-style codes or latent-space editing on an unconditional model.

For where GANs sit among other generative model families, and why the generator and discriminator split matters, see Generative vs. Discriminative Models and the overview of generative models.

Further reading

• Conditional Generative Adversarial Nets, Mirza and Osindero.
• Conditional Image Synthesis with Auxiliary Classifier GANs, Odena, Olah and Shlens.
• cGANs with Projection Discriminator, Miyato and Koyama.
• Large Scale GAN Training for High Fidelity Natural Image Synthesis, Brock, Donahue and Simonyan: BigGAN.
• Image-to-Image Translation with Conditional Adversarial Networks, Isola and colleagues: pix2pix.
• Generative Adversarial Text to Image Synthesis, Reed and colleagues.
• Semantic Image Synthesis with Spatially-Adaptive Normalization, Park and colleagues: SPADE.
• InfoGAN, Chen and colleagues.
• PyTorch nn.Embedding documentation.

Key takeaways

• A conditional GAN models p(x | y) by giving the condition to both networks. The discriminator must see it, or the generator has no reason to follow it.
• In the generator, concatenate a learned embedding with the noise for small models, or inject it at every block through conditional normalization for larger ones.
• In the discriminator, the projection design is the strong default for class labels. AC-GAN gives a strong class signal but tends to reduce diversity.
• Train with real labels for real images and sampled labels for fakes, drawn from the real label distribution.
• Check control with a grid that fixes noise along rows and varies the label along columns, and measure diversity per class.
• pix2pix applies the same idea to images: a U-Net generator, a PatchGAN discriminator on input-output pairs, and an L1 term for alignment.
• For rich conditions, train the discriminator on mismatched pairs so it checks that the output fits the condition, not only that it looks real.