Skip to content
Back to search results

Bayesian Gaussian Process

Introduced by
O'Hagan, A.; Neal, R. M.; Rasmussen, C. E. & Williams, C. K. I.
Year
1978–2006
Type
Probabilistic kernel model
Family
Machine learning
Font size

What does it do?

A Bayesian Gaussian Process (GP) places a probability distribution directly over functions, using a kernel to encode similarity between inputs.

After observing data, Bayes' rule converts this prior into a posterior that yields not just point predictions but calibrated uncertainty estimates at every new input — making it one of the most principled probabilistic models in machine learning.

The intuition behind the method

Imagine drawing infinitely many plausible curves through your data, each consistent with what you know. A GP quantifies how likely each curve is before and after seeing observations. The kernel determines which curves look smooth or wiggly, near or far-reaching. Once data arrive, the GP narrows the family of plausible curves around the observations while retaining honest uncertainty where data are sparse. This gives both a prediction and a confidence band that genuinely reflects how much evidence supports the forecast.

History

The use of Gaussian processes in statistics traces back to O'Hagan (1978) and Kriging in geostatistics (Matheron, 1963). Their adoption in machine learning was catalysed by Neal (1996), who showed deep networks converge to GPs in the limit of infinite width, and by Rasmussen and Williams's 2006 book, which became the standard reference. Today GPs underpin Bayesian optimisation pipelines in hyperparameter tuning and experimental design across engineering and science.

When to use it

Use a Bayesian GP when you need calibrated uncertainty alongside predictions — for example in active learning, Bayesian optimisation, safety-critical applications, or scientific inference where knowing what the model does not know matters as much as the prediction itself. It excels on small-to-medium datasets (up to a few thousand observations) with continuous or structured inputs and smooth underlying functions. Avoid GPs when the dataset exceeds roughly 10,000 rows without sparse or approximate GP methods, when the input space is high-dimensional (above ~20 features) without strong prior structure, or when a fast, opaque point predictor is all that is needed.

Limitations and considerations

  • Cubic computational cost O(n^3) in exact inference limits use to datasets of a few thousand observations without sparse approximations.
  • Kernel choice is influential and requires domain expertise; a misspecified kernel degrades both predictions and uncertainty estimates.
  • Closed-form inference breaks down for non-Gaussian likelihoods (classification, counts), requiring approximations that add complexity.
  • Scales poorly to high-dimensional input spaces without dimensionality reduction or structured kernels.

Source conditions do not replace independent scientific review or validation by an analysis engine.

Strengths

  • Produces fully calibrated predictive distributions, not just point estimates.
  • Closed-form posterior for regression — exact, no sampling required for moderate data.
  • Kernel selection encodes rich domain knowledge about smoothness, periodicity, and structure.
  • Log marginal likelihood provides automatic hyperparameter learning and model comparison.
  • Naturally supports active learning and Bayesian optimisation by quantifying where uncertainty is highest.
  • Sample-efficient: performs well on small datasets where deep learning cannot generalise.

Applications

  • Bayesian optimisation for hyperparameter tuning, experimental design, and AutoML.
  • Surrogate modelling in engineering simulations and computer experiments.
  • Spatial interpolation and geostatistics (Kriging).
  • Time-series forecasting with principled uncertainty quantification.
  • Active learning: selecting the most informative next observation based on posterior variance.
  • Scientific inference where uncertainty propagation is required alongside the prediction.

How it works

How it works

1

A GP prior is defined by a mean function m(x) — often set to zero — and a covariance (kernel) function k(x,x') that encodes how outputs at two inputs co-vary. Together they specify a distribution over all functions, from which the model samples candidate solutions before any data are seen.

Formula
f(\mathbf{x}) \sim \mathcal{GP}(m(\mathbf{x}),\, k(\mathbf{x},\mathbf{x}'))
2

The kernel is the central modelling choice. Common kernels include the squared-exponential (RBF), Matern, periodic, and linear kernels, each encoding different assumptions about smoothness, periodicity, or trend. Domain knowledge is used to select or compose kernels that match the expected behaviour of the target function.

3

Given training observations (X, y), Bayes' rule is applied in closed form for regression to obtain the posterior GP. The posterior mean provides the point prediction at any new input x*, while the posterior variance quantifies remaining uncertainty — shrinking near observations and widening in unexplored regions.

Formula
p(f^* | \mathbf{X}, \mathbf{y}, \mathbf{x}^*) = \mathcal{N}(\mu^*, \sigma^{*2})
4

Kernel hyperparameters (length-scales, output variance, noise level) are learned by maximising the log marginal likelihood, which automatically balances data fit against model complexity. This built-in Occam's razor avoids separate cross-validation for hyperparameter selection in many settings.

Formula
\hat{\boldsymbol{\theta}} = \arg\max \log p(\mathbf{y}|\mathbf{X},\boldsymbol{\theta})
5

The final predictive distribution at a new point is Gaussian for regression, returning a mean and a variance. For classification, an additional approximation step (Laplace, expectation propagation, or variational inference) converts the latent GP into class probabilities.

Common pitfalls

  • Using a default RBF kernel without considering whether the data are actually smooth and stationary.
  • Ignoring posterior predictive variance and reporting only the mean, discarding the principal advantage of the model.
  • Applying exact GP inference to large datasets (n > 10,000) without sparse or inducing-point approximations, causing memory and speed failures.
  • Confusing GP hyperparameter optimisation (marginal likelihood) with full Bayesian inference over hyperparameters — the latter requires MCMC and is rarely done in practice.
  • Treating classification GP outputs as if the latent posterior were exact, without accounting for the approximation error in non-Gaussian likelihoods.

Frequently asked questions

Is a Gaussian process inherently Bayesian?

Yes. A GP defines a prior over functions, and conditioning on data yields a posterior via Bayes' rule. The predictive distribution is the marginalisation of all plausible functions weighted by their posterior probability, which is the defining property of Bayesian inference.

How do I choose the kernel?

Start by asking whether the function is expected to be smooth (RBF/squared-exponential), periodic (periodic kernel), or piecewise irregular (Matern 3/2 or 5/2). Kernels can be combined additively or multiplicatively to capture composite structure. Validate choices using the log marginal likelihood.

Can I use a GP on thousands of data points?

Exact GP inference has O(n^3) cost, which is feasible up to a few thousand points on modern hardware. For larger datasets, use sparse GP approximations (inducing-point methods such as FITC or VFE) or stochastic variational GPs, which scale to millions of observations.

What is the difference between GP regression and GP classification?

GP regression uses a Gaussian likelihood, so the posterior is also Gaussian and can be computed in closed form. GP classification uses a Bernoulli (or softmax) likelihood, which is non-Gaussian, so the posterior must be approximated — commonly with Laplace approximation or expectation propagation.

How does a GP compare to a neural network?

GPs are better suited to small data, provide calibrated uncertainty out of the box, and are interpretable through the kernel. Neural networks dominate on large, high-dimensional data. Infinitely-wide neural networks with certain activations are mathematically equivalent to GPs, revealing a deep theoretical connection between the two.

Related methods

Sources

  1. Rasmussen, C. E., & Williams, C. K. I. (2006). Gaussian Processes for Machine Learning. MIT Press.

    ISBN: 978-0-262-18253-9
    Copied.
  2. Bishop, C. M. (2006). Pattern Recognition and Machine Learning (Ch. 6). Springer.

    ISBN: 978-0-387-31073-2
    Copied.