Skip to content
Back to search results

Score-Based Generative Model

Introduced by
Song, Y. & Ermon, S.
Year
2019
Type
Score-based generative model (SDE framework)
Font size

What does it do?

A score-based generative model, introduced by Yang Song and Stefano Ermon in 2019 and generalized to the stochastic differential equation (SDE) framework in 2021, learns the gradient of the data density — the score — rather than predicting noise directly, and uses it to generate new samples.

It is the mathematical generalization that unifies diffusion models under a continuous-time formulation.

The intuition behind the method

Instead of memorizing what the data looks like, the model learns which direction to nudge any point so that it moves toward regions where real data is dense. That direction is the score: the gradient of the log-density. Once you can point toward higher density from anywhere, you can start from pure noise and follow these directions step by step until you arrive at a realistic sample.

History

Score-based generative modeling was introduced by Song and Ermon in 2019 as generative modeling by estimating gradients of the data distribution. In 2021 Song and colleagues recast the idea in a stochastic differential equation framework, showing that score matching across continuous noise levels generalizes diffusion models and unifies several earlier generative formulations under a single continuous-time view.

When to use it

Suitable for high-fidelity generation and density estimation of continuous data when a large training set (around 1000 observations or more) or a pretrained model is available and GPU/TPU resources can be used. It makes no normality assumption, but the sampler must be matched to the chosen SDE. Below about 1000 observations a variational autoencoder is a safer choice, and below 500 a simpler method such as PCA is preferable.

Limitations and considerations

  • Requires a large dataset (around 1000+ observations) or a pretrained model to estimate the density gradient reliably.
  • Needs GPU/TPU hardware to train and sample.
  • The sampler must be chosen to match the SDE type, adding configuration burden.
  • On small samples the score estimate is poor and generation becomes meaningless.

Source conditions do not replace independent scientific review or validation by an analysis engine.

Strengths

  • Learns the data-density gradient directly, giving a principled mathematical generalization of diffusion models.
  • Continuous-time SDE framework unifies multiple diffusion formulations under one objective.
  • No normality assumption on the data.
  • Produces high-fidelity samples of continuous data and supports density estimation.

Applications

  • High-fidelity generation of continuous data such as images.
  • Density estimation and modeling of complex data distributions.
  • Continuous-time generative modeling that subsumes discrete diffusion formulations.
  • Prediction and explanatory generative tasks where large data or pretrained models are available.

How it works

How it works

1

A neural network is trained to estimate the score of the data distribution — the gradient of the log-density with respect to the input. Rather than predicting added noise, it learns the field of directions that point toward where data is concentrated, which is what makes the approach a mathematical generalization of diffusion models.

Formula
s_\theta(x) \approx \nabla_x \log p(x)
2

A forward stochastic differential equation gradually corrupts data into noise over continuous time, with a drift term and a diffusion term injecting Gaussian noise. This defines a smooth family of increasingly noisy distributions whose scores the network learns across all noise levels.

Formula
dx = f(x,t)\,dt + g(t)\,dw
3

Generation reverses the process by solving the corresponding reverse-time SDE, which uses the learned score to steer samples from noise back toward the data distribution. The choice of SDE type determines which sampler is appropriate, and the integration is run on GPU or TPU hardware.

Formula
dx = [f(x,t) - g(t)^2 \nabla_x \log p_t(x)]\,dt + g(t)\,d\bar{w}

Common pitfalls

  • Applying it below the ~1000-observation threshold, where the score cannot be estimated accurately.
  • Using a sampler that does not match the chosen SDE formulation.
  • Attempting to train without GPU/TPU resources.
  • Falling below ~500 observations, where SDE-based generation is meaningless and a simpler model should be used.

Frequently asked questions

What is the score that the model learns?

The score is the gradient of the log probability density with respect to the input. Knowing it at every point and noise level lets the model push samples toward where real data is concentrated.

How is this related to diffusion models?

It is the mathematical generalization of diffusion models: by formulating the forward corruption and reverse generation as continuous-time stochastic differential equations, score-based modeling unifies diffusion approaches under one framework.

How much data do I need?

Around 1000 observations or more, or a pretrained model. Below 1000 a variational autoencoder is safer, and below 500 SDE-based generation is meaningless and a simpler method such as PCA is preferable.

What hardware does it require?

GPU or TPU resources are needed for both training and sampling, and the sampler must be chosen to match the type of SDE used.

Related methods

Sources

  1. Song, Y. & Ermon, S. (2019). Generative Modeling by Estimating Gradients of the Data Distribution. NeurIPS 32, 11895–11907.

    Open publication reference
    Copied.
  2. Song, Y. et al. (2021). Score-Based Generative Modeling through Stochastic Differential Equations. ICLR.

    Open publication reference
    Copied.