Gaussian Process
- Introduced by
- Rasmussen, C. E. & Williams, C. K. I.
- Year
- 2006 (book); roots in Kriging, 1951)
- Type
- Probabilistic non-parametric model
- Family
- Machine learning
What does it do?
A Gaussian Process (GP) is a non-parametric, fully probabilistic machine learning model that places a prior distribution directly over functions.
Rather than predicting a single value, it returns a predictive mean and a calibrated uncertainty estimate at every test point, making it especially valuable for regression on small to medium datasets and for Bayesian optimization tasks.
The intuition behind the method
Instead of fitting a fixed curve to your data, a Gaussian Process maintains a whole distribution over possible curves that are consistent with what you have observed. Before seeing data, it assumes functions should be smooth (as defined by a kernel). After seeing data, it narrows that distribution: nearby test points receive tight, confident predictions, while distant or data-sparse regions produce wide uncertainty bands. This built-in uncertainty quantification is what sets GPs apart from most other models.
History
Gaussian Processes trace their roots to Kriging, developed by Danie Krige in geostatistics in the 1950s and formalised by Georges Matheron. Their adoption in machine learning was driven largely by Carl Rasmussen and Christopher Williams whose 2006 MIT Press book became the definitive reference, establishing GP as a principled Bayesian non-parametric alternative to neural networks and kernel SVMs, particularly for settings where uncertainty quantification is essential.
When to use it
Gaussian Processes excel on small to medium datasets (up to a few thousand rows) where uncertainty quantification matters — for example, scientific experiments, engineering design, or surrogate modelling for expensive simulations. They are the method of choice for Bayesian optimization. Prefer GP over simpler models when you need calibrated prediction intervals rather than just point estimates, and when the relationship between inputs and outputs is expected to be smooth. Avoid GP when n exceeds roughly 10,000 observations, as exact inference scales as O(n^3) making it computationally prohibitive without approximations. GP also struggles with very high-dimensional input spaces and with discontinuous or highly non-stationary functions unless carefully tailored kernels are used.
Limitations and considerations
- Exact inference is O(n^3) in time and O(n^2) in memory, making it impractical for large n without sparse or approximate GP methods.
- Performance degrades in very high-dimensional input spaces due to the curse of dimensionality in kernel distances.
- Choosing and combining kernels requires domain knowledge; a poorly chosen kernel leads to poor uncertainty calibration.
- Classification (GP classifiers) requires an additional Laplace or expectation-propagation approximation, adding complexity.
Source conditions do not replace independent scientific review or validation by an analysis engine.
Strengths
- Returns calibrated predictive uncertainty intervals alongside point predictions.
- Extremely sample-efficient — performs well on small datasets where tree-based models overfit.
- Kernel choice flexibly encodes domain knowledge about smoothness, periodicity, or scale.
- Hyperparameters are learned from data via marginal likelihood, providing automatic regularisation.
- Native framework for Bayesian optimization and active learning experimental design.
Applications
- Regression on small to medium scientific and engineering datasets requiring calibrated confidence intervals.
- Bayesian optimization of expensive black-box functions such as hyperparameter tuning or experimental design.
- Surrogate modelling for costly computer simulations in aerospace, materials science, and climate modelling.
- Spatio-temporal interpolation and geostatistical mapping (Kriging).
- Active learning: using the posterior variance to select the most informative next measurement.
- Time series modelling with periodic or composite kernels encoding seasonal structure.
How it works
How it works
A kernel function (commonly squared-exponential / RBF) encodes prior beliefs about function smoothness and length scale. It measures the similarity between any two input points: similar inputs should produce similar outputs. The choice of kernel shape strongly governs what kinds of functions the GP can represent.
Formula
k(\mathbf{x}, \mathbf{x}') = \sigma^2 \exp\!\left(-\frac{\|\mathbf{x}-\mathbf{x}'\|^2}{2l^2}\right)Before any data are observed, the GP defines a prior distribution over functions: every finite collection of function values is jointly Gaussian with mean function m(x) and covariance given by the kernel. This prior is the starting belief about how the unknown function behaves.
Formula
f(\mathbf{x}) \sim \mathcal{GP}(m(\mathbf{x}),\, k(\mathbf{x}, \mathbf{x}'))Conditioning the prior on the observed training data yields the posterior distribution, also Gaussian. The posterior mean becomes the point prediction at any new input x*, and the posterior variance provides a calibrated confidence interval, automatically narrowing where training data are dense.
Formula
f_* \mid \mathbf{X}, \mathbf{y}, \mathbf{x}_* \sim \mathcal{N}(\mu_*, \sigma^2_*)Kernel hyperparameters (length scale, output variance, noise level) are estimated by maximising the log marginal likelihood of the training data. This optimisation balances data fit against model complexity, performing automatic regularisation without a separate penalty term.
Formula
\hat{\boldsymbol{\theta}} = \arg\max_{\boldsymbol{\theta}} \log p(\mathbf{y} \mid \mathbf{X}, \boldsymbol{\theta})At prediction time the posterior mean is returned as the point estimate and the posterior standard deviation as the uncertainty band. Both can be used directly: report the mean for the best guess and the interval to communicate how confident the model is at each new input location.
Common pitfalls
- Using the default RBF kernel without checking whether the function is truly stationary and smooth — periodic or discontinuous patterns require different kernels.
- Forgetting to standardise continuous inputs and outputs before fitting, which leads to ill-conditioned covariance matrices.
- Treating the posterior mean alone as the deliverable and ignoring the uncertainty bands, which are the model's primary contribution.
- Applying exact GP to datasets with n above ~5,000 without switching to a sparse or inducing-point approximation.
- Conflating aleatoric noise (irreducible measurement error) with epistemic uncertainty (model uncertainty) when interpreting the variance output.
Frequently asked questions
What kernel should I start with?
The squared-exponential (RBF) kernel is the default starting point because it produces infinitely differentiable, smooth functions and has interpretable length-scale and output-variance hyperparameters. Switch to a Matern kernel if you expect the function to be less smooth, or add a periodic component for seasonal data.
My dataset has 50,000 rows. Can I still use a GP?
Exact GP inference would be prohibitive at that scale. Consider sparse GP methods (e.g., inducing points / FITC) or stochastic variational GP, both of which bring training cost down to roughly O(nm^2) where m is the number of inducing points (m << n). Libraries such as GPyTorch implement these efficiently.
How is a GP different from Bayesian linear regression?
Bayesian linear regression places a prior over a fixed finite set of coefficients; the model's flexibility is therefore limited by the chosen basis functions. A GP places a prior directly over functions in an infinite-dimensional space, so its effective complexity adapts to the data without requiring you to pre-specify basis expansions.
Are the uncertainty estimates truly calibrated?
Under the correct model and kernel, GP posterior uncertainties are well-calibrated by construction. In practice, miscalibration arises from kernel misspecification or ignoring noise structure. Always perform a posterior predictive check — plot observed vs. predicted with coverage of the credible intervals.
Can Gaussian Processes handle categorical or mixed-type inputs?
Standard GP kernels operate on continuous inputs. Categorical inputs require special kernels (e.g., Hamming distance kernel) or encoding. For predominantly categorical or very high-dimensional mixed data, tree-based models such as Random Forest or gradient boosting are usually a better starting point.
Related methods
Sources
Rasmussen, C. E., & Williams, C. K. I. (2006). Gaussian Processes for Machine Learning. MIT Press.
ISBN: 978-0-262-18253-9Copied.Gaussian process. Wikipedia.
Open publication referenceCopied.