CycleGAN: Image-to-Image Translation

pix2pix can turn a sketch into a photo or a map into an aerial view, but only if you hand it thousands of matched pairs: this exact sketch, and this exact photo. For many problems those pairs do not exist. Nobody has a photo of the same horse standing in the same field as a zebra, or a photograph of the scene Monet was looking at when he painted it.

CycleGAN, from Zhu, Park, Isola and Efros (2017), learns translations like these from two unrelated piles of images, one per domain. The trick is a constraint called cycle consistency: if you translate an image to the other domain and back, you should get the original. This article explains why that constraint is needed, how the losses fit together, what the networks look like, and where the method breaks. It follows Conditional GANs: Controlling the Output, which covered pix2pix, and uses the training ideas from Training GANs: Tips, Tricks, and Common Pitfalls.

Paired and unpaired translation

Image-to-image translation means learning a mapping from one image domain X to another domain Y while keeping the content and changing the appearance. Domains are just collections: summer photos and winter photos, horses and zebras, street photos and semantic label maps.

With paired data, every training input x has a matching target y. pix2pix uses that directly: an L1 loss compares the generator’s output with y pixel by pixel, and a conditional discriminator checks that the pair looks real.

With unpaired data you have a set of images from X and a separate set from Y, with no correspondence between them. The horse photos and the zebra photos were taken in different places, of different animals. There is no target to compare against, so the L1 term is gone.

Why an adversarial loss alone is not enough

The obvious first attempt is a single GAN: a generator G that takes a horse photo and produces an image, and a discriminator D_Y that judges whether the result looks like a real zebra photo. Train it and G will learn to produce convincing zebras.

The problem is that nothing ties the zebra to the horse. The adversarial loss only asks that the distribution of outputs matches the distribution of zebra photos. G could map every horse to the same good-looking zebra, or to zebras in different poses and backgrounds that have nothing to do with the input, and the discriminator would have no complaint. In practice this shows up as mode collapse or as outputs that ignore the input’s layout. The task is under-constrained: there are infinitely many mappings from X to Y that produce the right output distribution.

What you want is a mapping that changes the texture and keeps everything else, which needs another constraint.

Cycle consistency

CycleGAN trains two generators at once. G maps X to Y, and F maps Y back to X. Each has its own discriminator: D_Y judges G’s outputs against real Y images, and D_X judges F’s outputs against real X images.

The key added requirement is that the two mappings undo each other. Translate a horse to a zebra with G, translate the result back with F, and you should get the original horse: F(G(x)) ≈ x. The same holds the other way round: G(F(y)) ≈ y. The paper’s analogy is language translation: translate a sentence from English to French and back, and you should end up with the sentence you started with.

flowchart LR
    X["Real horse x"] --> G["G: X to Y"]
    G --> FY["Fake zebra G(x)"]
    FY --> DY["D_Y: real zebra?"]
    FY --> F["F: Y to X"]
    F --> RX["Reconstructed F(G(x))"]
    RX --> L["L1 vs x: cycle loss"]
    X --> L

The reverse cycle runs the same loop starting from a real zebra, through F, D_X and back through G.

Cycle consistency rules out most of the degenerate solutions. If G turned every horse into the same zebra, F could not recover which horse it started from, and the reconstruction loss would be large. To pass the round trip, G has to keep enough information about the input, such as pose, outline and background, for F to rebuild it. The easiest way to do that is to change only what distinguishes the domains.

flowchart TD
    subgraph Forward["Forward cycle"]
        X1["x"] --> G1["G"] --> Y1["G(x)"] --> F1["F"] --> X2["F(G(x)) close to x"]
    end
    subgraph Backward["Backward cycle"]
        Y2["y"] --> F2["F"] --> X3["F(y)"] --> G2["G"] --> Y3["G(F(y)) close to y"]
    end

The full objective

The CycleGAN objective has three parts.

Adversarial losses. One for each direction. G tries to make D_Y classify G(x) as real; D_Y tries to tell G(x) from real y. F and D_X play the same game in the other direction. The paper used a least-squares loss (from LSGAN) rather than the original log loss, because it trained more stably and gave better results. The discriminator pushes real outputs toward 1 and fakes toward 0; the generator pushes its fakes toward 1.

Cycle consistency loss. The L1 distance between each image and its reconstruction, summed over both directions:

L_cyc = E[ ||F(G(x)) − x||₁ ] + E[ ||G(F(y)) − y||₁ ]

The total. The generators minimize

L = L_GAN(G, D_Y) + L_GAN(F, D_X) + λ · L_cyc

with λ = 10 in the paper. That weight matters: cycle consistency is ten times as important as fooling either discriminator, which keeps the outputs anchored to their inputs.

The identity loss

For photo-to-painting tasks the paper added a fourth term. Without it, the generators sometimes changed the colours of the whole image, for example turning a daytime painting into a sunset photo, because nothing in the objective says colour composition should be preserved. The identity loss feeds each generator an image that is already in its target domain and asks it to leave it alone:

L_identity = E[ ||G(y) − y||₁ ] + E[ ||F(x) − x||₁ ]

The paper weighted it at 0.5λ. In the reference implementation this is the --lambda_identity 0.5 option, a fraction of the cycle weight. It helps when colour should be kept and is unnecessary for tasks where the colour must change, so treat it as task-dependent.

Network architecture

Generators

CycleGAN does not use the U-Net from pix2pix. Its generators follow the architecture from Johnson and colleagues’ work on fast neural style transfer: a few strided convolutions that downsample, a stack of residual blocks, then fractionally strided convolutions that upsample back to full size. The paper uses 6 residual blocks for 128 by 128 images and 9 for 256 by 256 and above.

Residual blocks suit the task. Each block computes x + f(x), so the easiest thing for the network to do is pass its input through nearly unchanged and learn a correction. Translation between related domains is mostly that: keep the structure, adjust the texture.

flowchart LR
    I["Input 256x256x3"] --> C1["7x7 conv, 64"]
    C1 --> D1["Down: 128"]
    D1 --> D2["Down: 256"]
    D2 --> R["9 residual blocks"]
    R --> U1["Up: 128"]
    U1 --> U2["Up: 64"]
    U2 --> C2["7x7 conv, 3, tanh"]
    C2 --> O["Output 256x256x3"]

All layers use instance normalization instead of batch normalization. Training runs with a batch size of 1, which makes batch statistics meaningless, and instance normalization removes per-image contrast differences, which is helpful for style-like changes.

Here is the generator in PyTorch, close to the reference implementation:

import torch
from torch import nn

class ResBlock(nn.Module):
    def __init__(self, ch):
        super().__init__()
        self.block = nn.Sequential(
            nn.ReflectionPad2d(1), nn.Conv2d(ch, ch, 3),
            nn.InstanceNorm2d(ch), nn.ReLU(inplace=True),
            nn.ReflectionPad2d(1), nn.Conv2d(ch, ch, 3),
            nn.InstanceNorm2d(ch),
        )

    def forward(self, x):
        return x + self.block(x)

class ResnetGenerator(nn.Module):
    def __init__(self, in_ch=3, out_ch=3, ngf=64, n_blocks=9):
        super().__init__()
        layers = [nn.ReflectionPad2d(3), nn.Conv2d(in_ch, ngf, 7),
                  nn.InstanceNorm2d(ngf), nn.ReLU(inplace=True)]
        ch = ngf
        for _ in range(2):                                   # downsample
            layers += [nn.Conv2d(ch, ch * 2, 3, stride=2, padding=1),
                       nn.InstanceNorm2d(ch * 2), nn.ReLU(inplace=True)]
            ch *= 2
        layers += [ResBlock(ch) for _ in range(n_blocks)]
        for _ in range(2):                                   # upsample
            layers += [nn.ConvTranspose2d(ch, ch // 2, 3, stride=2,
                                          padding=1, output_padding=1),
                       nn.InstanceNorm2d(ch // 2), nn.ReLU(inplace=True)]
            ch //= 2
        layers += [nn.ReflectionPad2d(3), nn.Conv2d(ch, out_ch, 7), nn.Tanh()]
        self.model = nn.Sequential(*layers)

    def forward(self, x):
        return self.model(x)

Reflection padding, rather than zero padding, reduces artifacts at the image borders.

Discriminators

Both discriminators are the 70 by 70 PatchGAN from pix2pix. Instead of one real-or-fake score for the whole image, it outputs a grid of scores, each covering an overlapping 70 by 70 patch. That concentrates the discriminator on local texture and style, which is what changes between domains, and it has fewer parameters than a full-image discriminator. Unlike pix2pix, the discriminator sees only the image, not an input-output pair, because there is no paired input to show it.

Training details that matter

Several choices in the paper are there to keep training stable:

• Least-squares loss. As above, LSGAN in place of the log loss.
• An image history buffer. The discriminators are updated with a mix of the current generator’s outputs and older ones drawn from a buffer of the 50 most recently generated images. This stops the discriminator from overfitting to the generator’s latest output and reduces oscillation. The idea comes from Shrivastava and colleagues (2017).
• Optimizer and schedule. Adam with a learning rate of 0.0002 and batch size 1. The rate stays constant for the first 100 epochs and decays linearly to zero over the next 100.
• Discriminator loss halved. The discriminator’s loss is divided by 2 so it learns more slowly relative to the generator.

The history buffer is short to implement:

import random
import torch

class ImagePool:
    def __init__(self, size=50):
        self.size, self.images = size, []

    def query(self, batch):
        out = []
        for img in batch:
            img = img.unsqueeze(0)
            if len(self.images) < self.size:          # fill the buffer first
                self.images.append(img)
                out.append(img)
            elif random.random() < 0.5:               # swap with an old image
                i = random.randrange(self.size)
                out.append(self.images[i].clone())
                self.images[i] = img
            else:                                     # use the current image
                out.append(img)
        return torch.cat(out, 0)

And one training step, putting the losses together. G_XY and G_YX are generators, D_X and D_Y are PatchGAN discriminators, and both pools are ImagePool instances:

import itertools
import torch
from torch import nn

mse, l1 = nn.MSELoss(), nn.L1Loss()
lam_cyc, lam_id = 10.0, 0.5

opt_G = torch.optim.Adam(itertools.chain(G_XY.parameters(), G_YX.parameters()),
                         lr=2e-4, betas=(0.5, 0.999))
opt_D = torch.optim.Adam(itertools.chain(D_X.parameters(), D_Y.parameters()),
                         lr=2e-4, betas=(0.5, 0.999))

def gan_loss(pred, is_real):
    target = torch.ones_like(pred) if is_real else torch.zeros_like(pred)
    return mse(pred, target)

def train_step(x, y):
    # Generators
    for p in itertools.chain(D_X.parameters(), D_Y.parameters()):
        p.requires_grad_(False)
    fake_y, fake_x = G_XY(x), G_YX(y)
    loss_gan = gan_loss(D_Y(fake_y), True) + gan_loss(D_X(fake_x), True)
    loss_cyc = l1(G_YX(fake_y), x) + l1(G_XY(fake_x), y)
    loss_id = l1(G_XY(y), y) + l1(G_YX(x), x)
    loss_G = loss_gan + lam_cyc * loss_cyc + lam_cyc * lam_id * loss_id
    opt_G.zero_grad()
    loss_G.backward()
    opt_G.step()

    # Discriminators, fed from the history buffers
    for p in itertools.chain(D_X.parameters(), D_Y.parameters()):
        p.requires_grad_(True)
    fy = pool_y.query(fake_y.detach())
    fx = pool_x.query(fake_x.detach())
    loss_D = 0.5 * (gan_loss(D_Y(y), True) + gan_loss(D_Y(fy), False)) \
           + 0.5 * (gan_loss(D_X(x), True) + gan_loss(D_X(fx), False))
    opt_D.zero_grad()
    loss_D.backward()
    opt_D.step()
    return loss_G.item(), loss_D.item()

Freezing the discriminators during the generator step avoids computing gradients for them that would be thrown away. Detaching the fakes before the discriminator step keeps the discriminator loss from flowing back into the generators.

Running the reference implementation

The authors maintain pytorch-CycleGAN-and-pix2pix, which covers both models. Its README says the code was recently updated for Python 3.11 and PyTorch 2.4, and it now supports multi-GPU training on one machine. To translate horses into zebras with a pretrained model:

git clone https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix
cd pytorch-CycleGAN-and-pix2pix
conda env create -f environment.yml
conda activate pytorch-img2img

bash ./scripts/download_cyclegan_model.sh horse2zebra
bash ./datasets/download_cyclegan_dataset.sh horse2zebra
python test.py --dataroot datasets/horse2zebra/testA \
    --name horse2zebra_pretrained --model test --no_dropout

--model test loads a single generator and runs it one way only. Results are written as an HTML page under ./results/.

Training your own model needs a folder with trainA, trainB, testA and testB subfolders, one pair per domain. The README’s example trains on the maps dataset:

bash ./datasets/download_cyclegan_dataset.sh maps
python train.py --dataroot ./datasets/maps --name maps_cyclegan \
    --model cycle_gan --use_wandb
python test.py --dataroot ./datasets/maps --name maps_cyclegan --model cycle_gan

--use_wandb logs progress to Weights and Biases; without it, intermediate results still go to an HTML page under ./checkpoints/maps_cyclegan/web/. At batch size 1 with 256 by 256 crops, a single modern GPU is enough, but 200 epochs over a few thousand images still takes many hours. Renting a GPU instance is usually cheaper than waiting on a laptop; the cloud platform comparison covers the options.

What CycleGAN is good and bad at

The paper’s results are strongest for changes of colour and texture that keep geometry fixed: horse to zebra, apple to orange, summer to winter in Yosemite photos, photos to paintings in the styles of Monet, Van Gogh, Cézanne and Ukiyo-e, and the reverse, paintings to photo-like images. It also handles semantic labels to photos and maps to aerial images, though for those pix2pix does better when pairs are available.

The limits follow from the design:

• Geometry changes fail. Asked to turn dogs into cats, CycleGAN makes small changes to the input rather than reshaping it. The residual generator and the cycle loss both reward keeping the layout, which is exactly wrong when the layout has to change.
• It learns the dataset, not the concept. The paper shows a man riding a horse who gets zebra stripes too, because the horse training set had no riders. The model has learned “texture of horse regions in these photos”, not “horse”.
• It is one-to-one. Each input gives one output. A summer scene has many plausible winter versions, but CycleGAN cannot sample among them.
• It can hide information. Chu, Zhmoginov and Sandler (2017) showed that CycleGAN can satisfy the cycle loss by encoding details of the source image in a near-invisible, high-frequency signal in the translated image, which the reverse generator then decodes. The reconstruction looks perfect even when the translation threw content away. A low cycle loss is not proof that content was preserved.

One practical use worth noting is domain adaptation. Models trained on rendered images from a simulator or a game engine often fail on real photos. Translating the synthetic images to look real, while keeping their labels valid because the layout is unchanged, lets you train on cheap labelled data. CyCADA (Hoffman and colleagues, 2018) built on this idea for semantic segmentation. The broader approach is covered in the upcoming article on synthetic data generation.

What came after

CycleGAN set the pattern for unpaired translation, and later work relaxed its constraints.

CUT (contrastive unpaired translation), from Park, Efros, Zhang and Zhu, is the direct successor from the same group. It drops the second generator: instead of requiring a round trip, it asks that a patch in the output be more similar to the corresponding input patch than to other patches. That avoids the steganography problem and halves the generator count.

CycleGAN-Turbo, from the img2img-turbo project, reflects the shift described in Generative AI in Image and Video Synthesis: diffusion models now dominate image generation, and recent translation methods start from a pretrained diffusion backbone rather than training a GAN from scratch. CycleGAN still matters as a small, well-understood baseline that trains on a single GPU from two folders of images, and its cycle consistency idea appears in work far beyond images. For how its building blocks relate to other generators, see Overview of Generative Models: VAEs, GANs, and More.

Further reading

• Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks, Zhu, Park, Isola and Efros: the CycleGAN paper.
• CycleGAN project page, with results and failure cases.
• pytorch-CycleGAN-and-pix2pix, the reference implementation, and its tips document for training on new datasets.
• CycleGAN, a Master of Steganography, Chu, Zhmoginov and Sandler.
• Contrastive Learning for Unpaired Image-to-Image Translation, Park and colleagues: CUT.
• img2img-turbo, which includes CycleGAN-Turbo.

Key takeaways

• CycleGAN learns image-to-image translation from two unpaired sets of images, where pix2pix needs matched pairs.
• An adversarial loss alone only matches output distributions; it does not tie each output to its input.
• Cycle consistency trains two generators that must undo each other, so each translation keeps enough of the input to be reversed.
• The objective combines two least-squares adversarial losses with an L1 cycle loss weighted at 10, plus an optional identity loss that preserves colour.
• Generators are residual networks with instance normalization; discriminators are 70 by 70 PatchGANs; a 50-image history buffer steadies the discriminators.
• It works well for texture, colour and style changes and fails when the shape must change.
• A low cycle loss can hide a cheat: the generator may encode the source in invisible noise.
• Successors such as CUT and diffusion-based CycleGAN-Turbo remove the second generator or start from pretrained diffusion models.