Skip to content

Compare methods

Compare source explanations, strengths, limitations and references for up to four methods.

10 results
Criterion Bayesian Optimization Neural Architecture Search Stochastic Optimization
Overview
  • Bayesian Optimization is a sequential, model-based strategy for finding the optimum of expensive black-box functions with as few evaluations as possible. Rooted in the work of Mockus (1975) and brought to mainstream machine-learning practice by Snoek, Larochelle, and Adams (2012), it fits a probabilistic surrogate model — typically a Gaussian Process — to past observations and uses an acquisition function to decide where to probe next, balancing exploration of unknown regions with exploitation of promising ones.
  • Neural Architecture Search (NAS), introduced by Zoph and Le in 2017, automatically optimizes architectural decisions such as a network's depth, width, and connection structure instead of hand-designing them. Leading methods in the field include DARTS, ENAS, and Once-for-All.
  • Stochastic optimization is a family of iterative methods that minimize an objective function by computing gradients on randomly sampled subsets of data — mini-batches — rather than on the entire dataset at once. Pioneered by Robbins and Monro in 1951 as stochastic approximation, the approach became the standard engine for training large-scale machine-learning models through variants such as SGD with momentum, AdaGrad, RMSProp, and Adam.
Use and interpretation
  • Bayesian Optimization is the right choice when the objective function is expensive or slow to evaluate — training a large model, running a physical simulation, or conducting a laboratory experiment — and the budget is limited to tens or low hundreds of evaluations. The search space must be defined with explicit bounds for each parameter. The method handles continuous and ordinal parameters naturally; purely categorical or very high-dimensional spaces (more than roughly 20 parameters) can degrade surrogate quality and may call for extensions such as random forests as the surrogate or dimensionality-reduction preprocessing. If evaluations are cheap and parallelism is available, simpler random search or grid search may be competitive.
  • Use NAS when you need to design a high-performing deep network for prediction or classification on continuous or text data, you have a large dataset (roughly 1000 observations or more), the search space is well defined, and a large computational budget with GPU/TPU resources is available. With small samples the search space cannot be evaluated reliably and candidate architectures overfit; below about 1000 observations a Random Forest is a safer choice, and below 500 a standard model such as gradient boosting is preferable.
  • Stochastic optimization applies whenever you need to minimise a differentiable (or sub-differentiable) loss function over a large dataset where computing exact gradients on the full dataset at every step is computationally prohibitive. It is the standard approach for training neural networks, linear models with large data, and any parametric model fitted by gradient descent. The objective must admit a gradient or subgradient; the method does not apply to non-differentiable black-box problems. Learning rate selection is critical — too large a rate causes divergence, too small a rate slows convergence.
Distribution assumption
  • No
Sample size
  • 1000
Output
  • Converged parameter vector minimizing the objective function
Strengths
  • Sample-efficient: finds good configurations in far fewer evaluations than grid search or random search, which matters when each evaluation is slow or costly.
  • Principled exploration–exploitation balance: the acquisition function provides a formal, interpretable criterion for where to search next.
  • Model-based uncertainty: the Gaussian Process not only predicts performance but also quantifies its own confidence, enabling informed stopping decisions.
  • Applicable to any black-box objective: the method requires no gradient information and no mathematical knowledge of the objective's interior.
  • Automates architecture design, removing slow manual trial and error.
  • Can discover networks that outperform hand-crafted designs on the target task.
  • Assumption-light on the data: no normality requirement; handles continuous and text inputs.
  • Flexible across tasks and data structures once the search space is specified.
  • Weight-sharing and super-network variants (ENAS, Once-for-All) sharply reduce search cost.
  • Scales to arbitrarily large datasets because each update uses only a mini-batch, not the full data.
  • Gradient noise from mini-batch sampling can help escape shallow local minima and saddle points.
  • Adaptive variants (Adam, RMSProp) reduce the burden of manual learning-rate tuning by adjusting per-parameter step sizes automatically.
  • Widely supported across all major deep-learning frameworks and well-understood theoretically.
Limitations
  • Scales poorly to very high-dimensional hyperparameter spaces: Gaussian Process fitting becomes computationally expensive and the surrogate loses accuracy beyond roughly 10–20 dimensions.
  • Sensitive to the choice of surrogate kernel and acquisition function: poor defaults can impede convergence, especially on non-stationary or multi-modal objectives.
  • Sequential by default: each evaluation must complete before the next candidate is selected, limiting parallelism unless batch-mode or asynchronous extensions are used.
  • Requires bounded search spaces: parameters without explicit ranges cannot be handled directly.
  • Requires a large computational budget and often a GPU/TPU cluster.
  • Needs a large dataset (about 1000+ observations) to evaluate candidates reliably.
  • Quality depends heavily on a well-defined search space, which itself requires expertise.
  • On small data, candidate architectures overfit and the search becomes unreliable.
  • Learning rate is a critical hyperparameter; poor choices lead to divergence or extremely slow convergence.
  • Mini-batch gradient noise can prevent convergence to an exact minimum — the optimiser typically oscillates near the optimum rather than settling precisely.
  • Saddle points and poor local minima are genuine risks in non-convex landscapes, though gradient noise and momentum help mitigate them.
  • Adam and other adaptive methods can converge to sharper minima that generalise worse than the flatter minima found by plain SGD with careful tuning.
Common pitfalls
  • Too few initial random probes leave the surrogate poorly calibrated; use at least five initial points, and more in higher-dimensional spaces.
  • Running the optimizer for very few total iterations (fewer than roughly ten) means the surrogate never becomes reliable; in such cases a simple random search is preferable.
  • Ignoring acquisition-function choice: EI favours exploitation and can get stuck near early good results; UCB adds exploration through a tunable confidence weight — neither is universally best, and the default should be verified against the problem structure.
  • Treating Bayesian Optimization as a black box itself: always inspect the convergence trace (best-seen score vs. iteration) to detect stagnation early.
  • Applying the method when evaluations are cheap — random or grid search will converge just as well at a fraction of the implementation overhead.
  • Running NAS on a small dataset, where candidates overfit and rankings are meaningless.
  • Defining a search space that is too broad to explore or too narrow to contain good designs.
  • Underestimating the compute cost and stopping the search before it converges.
  • Selecting an architecture on a validation split and not retraining and testing it cleanly.
  • Reaching for NAS when a standard model would already be sufficient for the problem.
  • Setting the learning rate too high causes the loss to diverge or oscillate wildly; always monitor the loss curve during training.
  • Forgetting to shuffle the data before each epoch can introduce periodic bias into the gradient estimates.
  • Using a fixed learning rate throughout training often leaves performance on the table — learning rate schedules (warm-up, cosine decay) consistently improve final results.
  • Comparing optimisers fairly requires matching the total number of gradient evaluations (or wall-clock time), not just the number of epochs.
  • Adam's adaptive rates can suppress useful gradient signal in sparse settings; SGD with momentum sometimes generalises better for image classification tasks.
Introduced by
  • Mockus (1975); popularised for ML by Snoek, Larochelle & Adams (2012)
  • Zoph, B. & Le, Q.V.
  • Herbert Robbins & Sutton Monro (SGD, 1951); Diederik Kingma & Jimmy Ba (Adam, 2015)
Year
  • 1975 (foundational); 2012 (ML standard)
  • 2017
  • 1951 (SGD); 2014 (Adam)