Skip to content
Back to search results

Diffusion Model

Introduced by
Ho, J., Jain, A. & Abbeel, P.
Year
2020
Type
Generative deep learning (denoising diffusion)
Font size

What does it do?

A diffusion model is a generative deep-learning method, introduced by Ho, Jain and Abbeel in 2020 (DDPM), that learns to produce high-quality images, audio and molecular structures by reversing a step-by-step noising process.

It has largely displaced GANs as the current state of the art in generative modelling.

The intuition behind the method

Imagine gradually blurring a photo into pure static by adding a little random noise at a time. A diffusion model learns the opposite journey: starting from random noise, it removes a little noise at each step until a clean, realistic sample emerges. Because it only ever has to undo one small noise step at a time, the task is far more stable to learn than generating a whole image in one shot.

History

Denoising Diffusion Probabilistic Models were introduced by Ho, Jain and Abbeel in 2020, building on earlier diffusion ideas and showing image quality competitive with GANs. In 2022 Rombach and colleagues introduced Latent Diffusion Models, running the diffusion process in a compressed latent space; this made high-resolution synthesis practical and underpins systems such as Stable Diffusion. Diffusion models have since become the dominant generative approach, largely replacing GANs.

When to use it

Use a diffusion model when you need to generate high-fidelity images, audio, or molecular structures and have a large dataset (around 1000 samples or more) or access to a pretrained model. It assumes substantial data or pretraining and requires GPU/TPU hardware; inference can be slow because sampling iterates over many denoising steps. With fewer than about 1000 samples the model cannot learn the denoising process well, and a simpler generator such as a variational autoencoder is safer.

Limitations and considerations

  • Requires a large dataset or a pretrained model — small data leads to poor generation quality.
  • GPU/TPU hardware is required for training and practical use.
  • Inference is slow because sampling iterates over many denoising steps.
  • Below about 1000 samples the denoising process cannot be learned reliably.

Source conditions do not replace independent scientific review or validation by an analysis engine.

Strengths

  • Produces higher-quality and more diverse samples than GANs across images, audio and molecules.
  • Training is stable, avoiding the mode collapse and adversarial instability that plague GANs.
  • Latent-diffusion variants make high-resolution generation computationally tractable.
  • Naturally supports conditional generation, such as text-to-image prompting.

Applications

  • High-resolution image synthesis and text-to-image generation.
  • Audio and speech generation.
  • Molecular and material structure generation in chemistry and the natural sciences.
  • Data augmentation and conditional generation from pretrained backbones.

How it works

How it works

1

A fixed forward process corrupts each training example by adding small amounts of Gaussian noise over many steps, until the data is turned into pure noise. This process needs no learning — it simply defines the trajectory from clean data to noise that the model will later reverse.

Formula
q(x_t \mid x_{t-1}) = \mathcal{N}(x_t; \sqrt{1-\beta_t}\,x_{t-1}, \beta_t I)
2

A neural network (typically a U-Net) is trained to predict the noise that was added at a given step. By minimising the squared error between the true and predicted noise across many time steps and examples, the network learns how to take a noisy input one step closer to clean data.

Formula
\mathcal{L} = \mathbb{E}_{t, x_0, \epsilon}\big[\,\lVert \epsilon - \epsilon_\theta(x_t, t) \rVert^2\,\big]
3

To generate a new sample, the model starts from random noise and repeatedly applies the learned denoising step, subtracting predicted noise and adding a small stochastic term until a clean output appears. Latent-diffusion variants (Stable Diffusion) run this loop in a compressed latent space to make high-resolution generation tractable.

Formula
x_{t-1} = \frac{1}{\sqrt{\alpha_t}}\Big(x_t - \frac{\beta_t}{\sqrt{1-\bar{\alpha}_t}}\,\epsilon_\theta(x_t, t)\Big) + \sigma_t z

Common pitfalls

  • Attempting to train from scratch on a small dataset instead of fine-tuning a pretrained model.
  • Underestimating compute: training and even inference demand GPU/TPU resources.
  • Expecting real-time output when multi-step sampling makes inference inherently slow.
  • Using a diffusion model where a much simpler generator (e.g. a VAE) would suffice for the data on hand.

Frequently asked questions

How is a diffusion model different from a GAN?

Both are generative, but a GAN trains a generator against a discriminator, which can be unstable and prone to mode collapse. A diffusion model instead learns to reverse a noising process step by step, giving more stable training and typically higher-quality, more diverse samples.

How much data do I need?

Training from scratch generally needs around 1000 samples or more; below that the denoising process cannot be learned well. With small data, fine-tune a pretrained model or use a simpler generator such as a variational autoencoder.

Why is generation slow?

Sampling starts from noise and applies the learned denoising step many times in sequence, so producing one sample requires many network passes. Latent-diffusion variants reduce the cost by working in a compressed latent space.

Do I need special hardware?

Yes. Diffusion models require GPU or TPU hardware for training and for practical inference; CPU-only use is generally impractical.

Related methods

Sources

  1. Ho, J., Jain, A. & Abbeel, P. (2020). Denoising Diffusion Probabilistic Models. NeurIPS.

    Open publication reference
    Copied.
  2. Rombach, R., Blattmann, A., Lorenz, D., Esser, P. & Ommer, B. (2022). High-Resolution Image Synthesis with Latent Diffusion Models. CVPR.

    Open publication reference
    Copied.