Bayesian Optimization — Sequential Model-Based Hyperparameter Tuning
- Introduced by
- Mockus (1975); popularised for ML by Snoek, Larochelle & Adams (2012)
- Year
- 1975 (foundational); 2012 (ML standard)
- Type
- Sequential model-based black-box optimization
What does it do?
Bayesian Optimization is a sequential, model-based strategy for finding the optimum of expensive black-box functions with as few evaluations as possible.
Rooted in the work of Mockus (1975) and brought to mainstream machine-learning practice by Snoek, Larochelle, and Adams (2012), it fits a probabilistic surrogate model — typically a Gaussian Process — to past observations and uses an acquisition function to decide where to probe next, balancing exploration of unknown regions with exploitation of promising ones.
The intuition behind the method
Imagine you have to find the best settings for a machine-learning model, but each trial takes hours to run. You cannot afford to test every combination. Bayesian Optimization acts like a careful scientist: it builds a running map of what has been tried and what results came back, uses that map to guess where the optimum probably lies, and then picks the next experiment that is most likely to improve the best result seen so far. As more experiments complete, the map grows more accurate and the choices smarter — all without ever needing to know the mathematical formula inside the black box.
History
The theoretical foundations of Bayesian Optimization were laid by Jonas Mockus and colleagues in the 1970s through the concept of expected improvement for sequential experimental design. The approach remained largely academic until 2012, when Snoek, Larochelle, and Adams published "Practical Bayesian Optimization of Machine Learning Algorithms" at NeurIPS, demonstrating that GP-based Bayesian Optimization consistently outperformed expert hand-tuning and random search on standard ML benchmarks. This work catalysed the development of dedicated libraries and established Bayesian Optimization as the de facto standard for hyperparameter search in deep learning and AutoML pipelines. Peter Frazier's 2018 tutorial (arXiv:1807.02811) later provided a unified pedagogical treatment that broadened the method's reach beyond the ML community.
When to use it
Bayesian Optimization is the right choice when the objective function is expensive or slow to evaluate — training a large model, running a physical simulation, or conducting a laboratory experiment — and the budget is limited to tens or low hundreds of evaluations. The search space must be defined with explicit bounds for each parameter. The method handles continuous and ordinal parameters naturally; purely categorical or very high-dimensional spaces (more than roughly 20 parameters) can degrade surrogate quality and may call for extensions such as random forests as the surrogate or dimensionality-reduction preprocessing. If evaluations are cheap and parallelism is available, simpler random search or grid search may be competitive.
Limitations and considerations
- Scales poorly to very high-dimensional hyperparameter spaces: Gaussian Process fitting becomes computationally expensive and the surrogate loses accuracy beyond roughly 10–20 dimensions.
- Sensitive to the choice of surrogate kernel and acquisition function: poor defaults can impede convergence, especially on non-stationary or multi-modal objectives.
- Sequential by default: each evaluation must complete before the next candidate is selected, limiting parallelism unless batch-mode or asynchronous extensions are used.
- Requires bounded search spaces: parameters without explicit ranges cannot be handled directly.
Source conditions do not replace independent scientific review or validation by an analysis engine.
Strengths
- Sample-efficient: finds good configurations in far fewer evaluations than grid search or random search, which matters when each evaluation is slow or costly.
- Principled exploration–exploitation balance: the acquisition function provides a formal, interpretable criterion for where to search next.
- Model-based uncertainty: the Gaussian Process not only predicts performance but also quantifies its own confidence, enabling informed stopping decisions.
- Applicable to any black-box objective: the method requires no gradient information and no mathematical knowledge of the objective's interior.
Applications
- Tuning hyperparameters of machine-learning and deep-learning models where each training run is computationally expensive.
- Neural Architecture Search (NAS): selecting architectural choices such as layer widths, activation functions, and skip connections.
- Experimental design in engineering and natural sciences where each physical experiment is costly or time-consuming.
- Automatic Machine Learning (AutoML) pipelines that select both algorithms and their hyperparameters jointly.
- Drug discovery and materials science, where evaluating a candidate configuration requires laboratory work.
How it works
How it works
The practitioner defines the search space: every hyperparameter to be tuned is given a type (continuous, ordinal) and a range — lower and upper bounds — within which the optimizer may search. A clear space definition is prerequisite to everything else and prevents the surrogate from being asked to extrapolate beyond safe regions.
A small number of points in the search space are evaluated at random or via a space-filling design (e.g., Latin hypercube sampling). These initial observations — typically five to ten — give the surrogate enough data to form a first, rough map of the objective landscape before the sequential loop begins.
All observations collected so far are used to fit a probabilistic surrogate model, most commonly a Gaussian Process. The GP returns both a mean prediction and an uncertainty estimate at every point in the space, capturing both what is known and how confident the model is. The surrogate is cheap to evaluate, so it can be queried thousands of times in place of the costly true objective.
An acquisition function — such as Expected Improvement (EI) or Upper Confidence Bound (UCB) — is maximised over the surrogate to select the next candidate point. EI quantifies the expected gain over the current best; UCB adds a confidence-weighted exploration bonus. This step is where the exploration–exploitation trade-off is resolved: regions with high predicted performance and regions with high uncertainty both attract attention.
The candidate point selected by the acquisition function is passed to the true objective — for example, training and validating a neural network with those hyperparameters. This evaluation is the expensive step. Its result (the score) is recorded alongside the candidate's coordinates.
The new observation is added to the dataset, the surrogate is refitted, and the loop returns to the acquisition step. After the iteration budget is exhausted, the hyperparameter configuration that produced the best observed score is returned as the final recommendation.
Common pitfalls
- Too few initial random probes leave the surrogate poorly calibrated; use at least five initial points, and more in higher-dimensional spaces.
- Running the optimizer for very few total iterations (fewer than roughly ten) means the surrogate never becomes reliable; in such cases a simple random search is preferable.
- Ignoring acquisition-function choice: EI favours exploitation and can get stuck near early good results; UCB adds exploration through a tunable confidence weight — neither is universally best, and the default should be verified against the problem structure.
- Treating Bayesian Optimization as a black box itself: always inspect the convergence trace (best-seen score vs. iteration) to detect stagnation early.
- Applying the method when evaluations are cheap — random or grid search will converge just as well at a fraction of the implementation overhead.
Frequently asked questions
How is Bayesian Optimization different from grid search or random search?
Grid search evaluates a fixed lattice of configurations and random search samples uniformly at random — both ignore the results of previous trials when choosing the next point. Bayesian Optimization actively uses all past observations to build a surrogate model and selects the next point most likely to improve over the current best, making it far more sample-efficient when evaluations are expensive.
What acquisition function should I use?
Expected Improvement (EI) is the most widely used default: it measures the expected gain over the current best and tends to be reliable across problem types. Upper Confidence Bound (UCB) adds a tunable exploration bonus and is often preferred when the landscape is poorly understood. Probability of Improvement (PI) is more exploitation-heavy and can converge prematurely. Start with EI and switch to UCB if the optimizer appears to stagnate.
How many iterations are needed?
There is no universal rule, but fewer than about ten total evaluations (including initial random probes) give the surrogate too little data to be reliable. Practical deployments typically use five to ten initial random points followed by twenty to fifty sequential Bayesian steps. When the best-seen score stops improving over several consecutive iterations, the budget can be stopped.
Can Bayesian Optimization be parallelised?
Standard Bayesian Optimization is inherently sequential. Batch-mode extensions allow proposing a set of candidates simultaneously — common strategies include Local Penalization and the Kriging Believer heuristic — enabling parallel evaluation. Asynchronous variants launch a new candidate whenever a worker becomes free rather than waiting for the full batch. These extensions are available in libraries such as Optuna and BoTorch.
Related methods
Sources
Snoek, J., Larochelle, H., & Adams, R.P. (2012). Practical Bayesian Optimization of Machine Learning Algorithms. Advances in Neural Information Processing Systems (NeurIPS), 25.
Open publication referenceCopied.Frazier, P.I. (2018). A Tutorial on Bayesian Optimization. arXiv:1807.02811.
Open publication referenceCopied.