StyleGAN: Generating High-Resolution Images

Most GANs feed a noise vector into the first layer and let the network do the rest. The vector controls everything at once: pose, identity, lighting, hair, the texture of skin. Change one number and several of those move together, because nothing in the architecture asks them to stay apart.

StyleGAN, from Karras, Laine and Aila at NVIDIA (2019), rearranged the generator so that the latent code enters at every resolution instead of only at the bottom. The result was 1024 by 1024 faces that were hard to tell from photographs, and a latent space where coarse attributes and fine detail could be controlled separately. This article walks through how the architecture works, why each piece is there, and what the three follow-ups (StyleGAN2, StyleGAN2-ADA and StyleGAN3) fixed. It builds on Training GANs: Tips, Tricks, and Common Pitfalls and Conditional GANs: Controlling the Output, which introduced conditional normalization, the idea StyleGAN takes furthest.

Where StyleGAN came from

StyleGAN grew out of the same group’s Progressive GAN (Karras, Aila, Laine and Lehtinen, 2018). High-resolution GAN training had been unstable because the discriminator could tell real from fake easily at full resolution, and its gradients gave the generator little useful direction. Progressive GAN started both networks at 4 by 4 pixels and added layers during training, doubling the resolution step by step up to 1024 by 1024. Each new layer was faded in smoothly so the existing layers were not disrupted. The networks learned large-scale structure first and detail later, which made training far more stable.

Progressive GAN also introduced CelebA-HQ, a 1024 by 1024 version of the CelebA face dataset, and techniques such as equalized learning rate and minibatch standard deviation that StyleGAN kept.

What it did not change was the generator’s input. The latent vector still went into the first layer, and the network had no structural reason to separate what happens at 8 by 8 from what happens at 512 by 512. StyleGAN kept the progressive training and replaced the generator’s design.

The style-based generator

The StyleGAN generator has two parts: a mapping network that turns the latent code into an intermediate code, and a synthesis network that builds the image from a learned constant, with the intermediate code injected at every layer.

flowchart LR
    Z["Latent z (512)"] --> N["Normalize"]
    N --> M["Mapping network, 8 FC layers"]
    M --> W["Intermediate w (512)"]
    W --> A1["Affine A"]
    W --> A2["Affine A"]
    W --> A3["Affine A"]
    C["Learned constant 4x4x512"] --> S1["4x4 block"]
    A1 --> S1
    S1 --> S2["8x8 block"]
    A2 --> S2
    S2 --> S3["... up to 1024x1024"]
    A3 --> S3
    NZ["Per-pixel noise"] --> S1
    NZ --> S2
    NZ --> S3
    S3 --> I["Image"]

The mapping network

The latent z is a 512-dimensional vector sampled from a standard normal distribution. The mapping network, eight fully connected layers, maps it to w, another 512-dimensional vector. The space of all possible w values is called W.

The reason for this step is disentanglement. The training data has a fixed distribution of attributes. In a face dataset, for example, there may be few images of long-haired men. A generator that reads z directly has to map the whole Gaussian onto that lopsided distribution, which forces it to warp the space: directions in z end up mixing attributes together so that every z lands on a plausible face. The mapping network can absorb that warping, which leaves W free to be closer to linear, with directions that correspond more cleanly to single attributes. The paper measured this with two metrics it introduced, perceptual path length and linear separability, and W scored better than Z on both.

Styles and adaptive instance normalization

In the original StyleGAN, w controls the synthesis network through adaptive instance normalization (AdaIN). Each convolution layer’s output is normalized per channel, per image, to zero mean and unit variance, then scaled and shifted:

AdaIN(x, y) = y_s · (x − μ(x)) / σ(x) + y_b

The scale y_s and bias y_b come from w through a learned affine layer (the “A” boxes in the diagram), a separate one for each layer. These per-layer values are the styles. This is the same idea as the conditional batch normalization in the previous article, except that the input is a learned code instead of a class label, and the normalization is per instance.

Because each layer renormalizes its input before applying its own style, a style only affects one convolution before the next normalization overrides it. That locality is what lets different layers control different things.

The synthesis network does not need z or w as an input image. It starts from a learned 4 by 4 by 512 constant tensor, the same for every image. All variation comes from the styles and the noise. The paper found that removing the traditional input did not hurt quality.

Noise inputs

Some details in a photograph are random: the exact placement of individual hairs, freckles, pores, the arrangement of stubble. A generator with only a global latent code has to invent a way to produce pseudo-random variation from that code, which wastes capacity and tends to produce repeated patterns.

StyleGAN adds a single-channel image of Gaussian noise after each convolution, scaled per channel by learned weights (the “B” inputs in the paper’s figure). Fresh noise for each image gives the network a ready source of randomness at every resolution. Resampling the noise while keeping w fixed changes hair strands and skin texture but leaves identity, pose and expression untouched. Setting it to zero produces a smooth, painted look.

Which layers control what

At 1024 by 1024, the synthesis network has nine resolutions from 4 by 4 to 1024 by 1024, two convolution layers each, so 18 layers take a style. Swapping styles between images showed what each range controls:

The boundaries are approximate and depend on the dataset, but the pattern is consistent: low resolutions decide structure, high resolutions decide appearance.

Style mixing

Since each layer gets its own copy of w, nothing forces them to be the same. During training, StyleGAN uses style mixing regularization: for a share of training images (90 percent in the paper), it samples two latents, maps both, and switches from w1 to w2 at a random layer. The generator then has to produce a plausible image from a mix of two codes.

This keeps adjacent layers from relying on each other, which strengthens the separation between coarse, middle and fine styles. It also gives you a direct way to combine images at inference time: take the coarse styles from one face and the fine styles from another, and you get the first face’s pose and shape with the second face’s colouring.

flowchart TD
    Z1["z1"] --> M1["Mapping"] --> W1["w1"]
    Z2["z2"] --> M2["Mapping"] --> W2["w2"]
    W1 --> L1["Layers 1 to 8: coarse and middle"]
    W2 --> L2["Layers 9 to 18: fine"]
    L1 --> O["Image: pose and shape from 1, colour and texture from 2"]
    L2 --> O

A set of 18 separate w vectors, one per layer, is often called W+. It is more expressive than W, which matters for editing real photos, covered below.

The truncation trick

Regions of the latent space that saw little training data produce worse images. A simple fix is to pull each w toward the average:

w’ = w_avg + ψ (w − w_avg)

where w_avg is the mean of w over many samples, tracked during training, and ψ is between 0 and 1. With ψ = 1 you get the full distribution. Lower values give cleaner, more typical images at the cost of variety. At ψ = 0 every latent produces the same average face.

Doing this in W rather than Z works better, because W’s average is a meaningful image. You can also apply truncation only to the coarse layers, which keeps typical structure while leaving fine detail varied. The official code exposes this as truncation_cutoff.

Remember that truncated samples are not samples from the learned distribution. Published FID scores are computed without truncation, and images shown with ψ around 0.5 to 0.7 look better than the model’s average output.

StyleGAN2: fixing the artifacts

Once StyleGAN was widely used, two flaws stood out. Karras and colleagues addressed both in “Analyzing and Improving the Image Quality of StyleGAN” (2020).

Water droplets and weight demodulation

Many StyleGAN images had a blob-shaped artifact, like a water droplet, that appeared at 64 by 64 and persisted to full resolution. The authors traced it to AdaIN. Because instance normalization discards the magnitude of each feature map, the generator learned to sneak information past it by creating a strong local spike that dominated the statistics, then scaling everything else around it.

StyleGAN2 removes the explicit normalization of activations and moves the style into the convolution weights. Modulation scales each input channel of the weights by the style. Demodulation then rescales each output channel so the expected output has unit variance, based on the weights rather than on the actual activations:

w’_ijk = s_i · w_ijk w”_ijk = w’_ijk / sqrt(Σ_i,k (w’_ijk)² + ε)

Here i is the input channel, j the output channel, k the spatial position in the kernel and s_i the style. Normalizing a statistical expectation instead of the measured activations removes the artifact while keeping the per-layer style control.

Here is a modulated convolution in PyTorch. Each sample has its own weights, so the batch is folded into the channel dimension and run as one grouped convolution:

import torch
from torch import nn
import torch.nn.functional as F

class ModulatedConv2d(nn.Module):
    def __init__(self, in_ch, out_ch, kernel, w_dim=512, demodulate=True):
        super().__init__()
        self.weight = nn.Parameter(torch.randn(out_ch, in_ch, kernel, kernel))
        self.affine = nn.Linear(w_dim, in_ch)          # w -> per-input-channel style
        nn.init.ones_(self.affine.bias)                # styles start near 1
        self.demodulate = demodulate
        self.scale = 1 / (in_ch * kernel * kernel) ** 0.5
        self.padding = kernel // 2

    def forward(self, x, w):
        b, in_ch, h, wd = x.shape
        s = self.affine(w)                                          # (B, in_ch)
        weight = self.scale * self.weight[None] * s[:, None, :, None, None]   # modulate
        if self.demodulate:
            d = torch.rsqrt(weight.pow(2).sum(dim=(2, 3, 4)) + 1e-8)          # (B, out_ch)
            weight = weight * d[:, :, None, None, None]                       # demodulate
        out_ch = weight.shape[1]
        weight = weight.reshape(b * out_ch, in_ch, *weight.shape[3:])
        x = x.reshape(1, b * in_ch, h, wd)
        out = F.conv2d(x, weight, padding=self.padding, groups=b)
        return out.reshape(b, out_ch, h, wd)

The final layer that converts features to RGB uses modulation without demodulation, since its output should not be forced to unit variance.

Dropping progressive growing

The second flaw was phase artifacts: some details, such as teeth or eyes, stayed fixed in screen position when the face turned, instead of moving with it. The authors linked this to progressive growing, where each resolution is at some point the output resolution and learns to produce maximum detail there.

StyleGAN2 trains the full network from the start. The generator uses skip connections, summing RGB outputs from each resolution after upsampling, and the discriminator uses residual connections. Analysis of the trained network showed it still focuses on low resolutions early in training and shifts to high ones later, so the coarse-to-fine benefit of progressive growing remained without the fade-in schedule.

Regularization

StyleGAN2 also adds path length regularization, which encourages a fixed-size step in W to produce a fixed-size change in the image, whatever the direction. It makes the generator smoother and easier to invert. It keeps the R1 gradient penalty on the discriminator described in the GAN training article, and applies both regularizers lazily, every 16 minibatches for R1, to save compute.

StyleGAN2-ADA: training on small datasets

A GAN discriminator overfits when data is scarce. It memorizes the training images, its feedback to the generator stops being useful, and training diverges. With a few thousand images, StyleGAN2 broke down.

Standard data augmentation does not help directly: if you flip, crop or recolour only the real images, the generator learns to produce flipped, cropped and recoloured images. “Training Generative Adversarial Networks with Limited Data” (Karras and colleagues, 2020) applies the augmentations to every image the discriminator sees, real and generated, with probability p. The paper shows that for a wide set of augmentations, as long as p stays below roughly 0.8, the augmentations do not “leak”: the generator is not pushed to reproduce them, because the only way to match the augmented real distribution is to match the clean one.

The adaptive part is choosing p automatically. A heuristic measures overfitting from the sign of the discriminator’s outputs on real training images: when the discriminator is confidently right about nearly all of them, it is memorizing. The training loop raises p when this measure exceeds a target (0.6 in the paper) and lowers it otherwise. The paper showed good results on datasets of a few thousand images, including MetFaces, a set of around 1,300 faces from museum artworks that the authors released.

ADA is usually the right starting point for custom datasets. It is enabled by default in the official code.

StyleGAN3: alias-free generation

Even StyleGAN2 has a quirk visible in video. When you interpolate through the latent space, fine details such as hair and skin texture appear glued to fixed pixel coordinates while the face moves under them. The authors called this texture sticking.

“Alias-Free Generative Adversarial Networks” (Karras and colleagues, 2021), released as StyleGAN3, traced the cause to aliasing. Operations such as nearest-neighbour upsampling, pointwise nonlinearities and padding at image borders leak information about absolute pixel positions into the feature maps, and the network learns to use it. The fix comes from signal processing: treat feature maps as continuous signals, use carefully designed low-pass filters around every upsampling step and nonlinearity, and replace the 4 by 4 learned constant input with Fourier features so the network has no fixed grid to anchor to.

There are two configurations. StyleGAN3-T is equivariant to translation: shift the input and the output shifts with it. StyleGAN3-R is equivariant to translation and rotation. FID is about the same as StyleGAN2, so the gain shows in motion, not in a single still image. If you only need stills, StyleGAN2 trains faster and is still a strong choice.

The family at a glance

Running a pretrained model

NVIDIA’s official StyleGAN3 repository covers StyleGAN2 and StyleGAN3 in PyTorch, with pretrained networks for faces (FFHQ), animal faces (AFHQv2) and MetFaces. The README lists 64-bit Python 3.8 and PyTorch 1.9.0 or later, and CUDA toolkit 11.1 or later, and the code is under the NVIDIA Source Code License, which limits use to non-commercial purposes. Read it before you build anything on the weights.

Generating an image from a seed:

git clone https://github.com/NVlabs/stylegan3.git
cd stylegan3
python gen_images.py --outdir=out --trunc=1 --seeds=2 \
    --network=https://api.ngc.nvidia.com/v2/models/nvidia/research/stylegan3/versions/1/files/stylegan3-r-afhqv2-512x512.pkl

From Python, you can call the mapping and synthesis networks separately, which is what you need for mixing, truncation and editing. The pickles reference modules in the repository, so run this from the repository root:

import pickle
import torch

with open("stylegan3-r-afhqv2-512x512.pkl", "rb") as f:
    G = pickle.load(f)["G_ema"].cuda()     # EMA weights of the generator

z = torch.randn([2, G.z_dim]).cuda()
c = None                                   # no class label for this model
ws = G.mapping(z, c, truncation_psi=0.7)   # (2, num_ws, 512): one w per layer
mixed = ws[:1].clone()
mixed[:, 8:] = ws[1:, 8:]                  # fine styles from the second latent
img = G.synthesis(mixed, noise_mode="const")
img = (img.clamp(-1, 1) + 1) * 127.5      # NCHW in [0, 255]

G.mapping returns w already repeated for each layer, so W+ editing is a matter of changing individual rows. noise_mode="const" uses fixed noise buffers so results are reproducible.

Training on your own data

Training needs a dataset packed with the repository’s tool, which resizes and stores images in a zip. All images should be square and the same size, and the subject should be aligned consistently; StyleGAN’s quality comes partly from FFHQ’s careful face alignment.

python dataset_tool.py --source=~/my-images --dest=~/datasets/my-512x512.zip \
    --resolution=512x512

python train.py --outdir=~/training-runs --cfg=stylegan3-t \
    --data=~/datasets/my-512x512.zip \
    --gpus=8 --batch=32 --gamma=8.2 --mirror=1

The README singles out --gpus, --batch and --gamma (the R1 weight) as the options you must choose with care, and the repository’s configuration guide gives starting values per resolution. --mirror=1 adds horizontal flips, which is fine for faces and wrong for anything with text. --resume with a pretrained pickle fine-tunes from an existing model, which is how most people get good results from small datasets, and pairs well with ADA.

StyleGAN training at high resolution takes days on several data-centre GPUs, so plan for cloud GPU capacity rather than a laptop. The AWS, GCP and Azure overviews cover the options.

Editing real images

A trained StyleGAN is useful well beyond random samples, because its latent space is organized enough to edit. The workflow has two steps.

flowchart LR
    P["Real photo"] --> INV["Invert: find w that reproduces it"]
    INV --> WV["w or W+ code"]
    WV --> E["Move along an edit direction"]
    E --> G["Generator"]
    G --> O["Edited photo"]

Inversion finds a latent that reproduces a given photo. Optimization methods such as Image2StyleGAN run gradient descent on w with a perceptual loss, usually in W+ for accuracy. Encoder methods such as pSp and e4e train a network to predict the code in one pass. Pivotal Tuning Inversion (PTI) inverts first, then fine-tunes the generator slightly around that code, which keeps identity much better on out-of-distribution photos. There is a trade-off throughout: codes that reconstruct the photo exactly tend to sit off the learned distribution and edit badly, while codes closer to W edit well and reconstruct less faithfully.

Editing moves the code along a direction. InterFaceGAN fits a linear classifier for an attribute, such as age or smile, on generated samples and uses its normal vector as the direction. GANSpace finds directions without labels by running PCA on sampled w vectors. StyleCLIP uses CLIP to steer edits from a text description.

These techniques are why StyleGAN remained a research tool after diffusion models took over text-to-image generation, a shift covered in Generative AI in Image and Video Synthesis. For a single domain such as faces, a StyleGAN generator produces an image in one forward pass, where a diffusion model needs many denoising steps, and its latent space supports smooth, controllable edits.

The same realism raises the misuse concerns described in Real-World Use Cases: Deepfakes and Their Impact. Synthetic StyleGAN faces have been used for fake profile photos, and the consistent eye placement learned from aligned training data is one of the tells investigators look for.

Further reading

• A Style-Based Generator Architecture for Generative Adversarial Networks, Karras, Laine and Aila: StyleGAN.
• Progressive Growing of GANs for Improved Quality, Stability, and Variation, Karras and colleagues.
• Analyzing and Improving the Image Quality of StyleGAN, Karras and colleagues: StyleGAN2.
• Training Generative Adversarial Networks with Limited Data, Karras and colleagues: StyleGAN2-ADA.
• Alias-Free Generative Adversarial Networks, Karras and colleagues: StyleGAN3.
• Official StyleGAN3 code and the FFHQ dataset.
• Interpreting the Latent Space of GANs for Semantic Face Editing, Shen and colleagues: InterFaceGAN.
• Pivotal Tuning for Latent-based Editing of Real Images, Roich and colleagues.

Key takeaways

• StyleGAN’s generator starts from a learned constant and injects the latent at every layer, so each resolution can be controlled separately.
• The mapping network turns Gaussian z into an intermediate w whose space is less entangled, which makes interpolation and editing better behaved.
• Low-resolution styles control pose and shape, high-resolution styles control colour and texture, and per-layer noise supplies random fine detail.
• Style mixing during training keeps layers independent; truncation in W trades variety for typical, clean samples.
• StyleGAN2 replaced AdaIN with weight demodulation and dropped progressive growing, removing droplet and phase artifacts.
• StyleGAN2-ADA augments everything the discriminator sees, with an adaptive probability, so training works on a few thousand images.
• StyleGAN3 removes aliasing so details move with the object in video. For still images, StyleGAN2 remains a good, faster choice.
• Inversion plus latent directions turns a trained StyleGAN into a photo editor, which is its main use today.