Skip to content
Back to search results

Stochastic Optimization — SGD and Variants

Year
1951 (SGD); 2014 (Adam)
Type
Gradient-based iterative optimization
Font size

What does it do?

Stochastic optimization is a family of iterative methods that minimize an objective function by computing gradients on randomly sampled subsets of data — mini-batches — rather than on the entire dataset at once.

Pioneered by Robbins and Monro in 1951 as stochastic approximation, the approach became the standard engine for training large-scale machine-learning models through variants such as SGD with momentum, AdaGrad, RMSProp, and Adam.

The intuition behind the method

Imagine you want to find the lowest point in a vast hilly landscape, but measuring the slope of the entire landscape at every step is prohibitively expensive. Stochastic optimization works by measuring the slope at a small random patch of ground instead, taking a step downhill, and repeating the process many times. Each individual step is noisier than measuring the whole landscape, but the sheer number of cheap steps gets you to the bottom faster than a few expensive exact measurements would. Adaptive variants like Adam go further by remembering how bumpy each direction has been and scaling their steps accordingly.

History

Stochastic optimisation traces to Robbins and Monro (1951), who proved that iterative gradient estimates converge to a function's stationary point under mild conditions — founding the field of stochastic approximation. Stochastic gradient descent became widely used in machine learning through Bottou's work in the 1990s on large-scale linear models. The deep-learning era intensified demand for efficient adaptive methods: Duchi et al. introduced AdaGrad in 2011, Tieleman and Hinton proposed RMSProp in 2012, and Kingma and Ba published Adam in 2014, which rapidly became the default optimiser for neural network training due to its robustness across diverse architectures and tasks.

When to use it

Stochastic optimization applies whenever you need to minimise a differentiable (or sub-differentiable) loss function over a large dataset where computing exact gradients on the full dataset at every step is computationally prohibitive. It is the standard approach for training neural networks, linear models with large data, and any parametric model fitted by gradient descent. The objective must admit a gradient or subgradient; the method does not apply to non-differentiable black-box problems. Learning rate selection is critical — too large a rate causes divergence, too small a rate slows convergence.

Limitations and considerations

  • Learning rate is a critical hyperparameter; poor choices lead to divergence or extremely slow convergence.
  • Mini-batch gradient noise can prevent convergence to an exact minimum — the optimiser typically oscillates near the optimum rather than settling precisely.
  • Saddle points and poor local minima are genuine risks in non-convex landscapes, though gradient noise and momentum help mitigate them.
  • Adam and other adaptive methods can converge to sharper minima that generalise worse than the flatter minima found by plain SGD with careful tuning.

Source conditions do not replace independent scientific review or validation by an analysis engine.

Strengths

  • Scales to arbitrarily large datasets because each update uses only a mini-batch, not the full data.
  • Gradient noise from mini-batch sampling can help escape shallow local minima and saddle points.
  • Adaptive variants (Adam, RMSProp) reduce the burden of manual learning-rate tuning by adjusting per-parameter step sizes automatically.
  • Widely supported across all major deep-learning frameworks and well-understood theoretically.

Applications

  • Training neural networks of all depths — from logistic regression to large language models.
  • Fitting large-scale linear and generalised linear models where full-batch gradient descent is computationally infeasible.
  • Online learning settings where data arrive in a stream and the model must update incrementally.
  • Reinforcement learning policy gradient methods, where Adam and RMSProp are standard choices.
  • Recommendation systems and embedding models trained on massive sparse datasets.

How it works

How it works

1

The parameter vector is initialised — typically with small random values. The learning rate (step size) and, for adaptive variants, moment decay rates are set as hyperparameters before training begins. These choices have a large effect on convergence speed and final performance.

2

At each iteration a mini-batch — a small random subset of the full dataset — is drawn. Mini-batch size controls a trade-off: smaller batches introduce more gradient noise but update parameters more frequently and often generalise better, while larger batches give lower-variance estimates at higher per-iteration cost.

3

The gradient of the objective (loss) function with respect to the current parameters is computed using the mini-batch. For adaptive optimisers, first and second moment estimates of the gradient are also maintained across iterations to track per-parameter scale and curvature information.

4

Parameters are updated by moving opposite to the gradient direction, scaled by the learning rate. Adaptive variants (AdaGrad, RMSProp, Adam) divide each parameter's step by a running estimate of past gradient magnitudes, effectively giving larger steps to infrequently updated parameters and smaller steps where the gradient has been large and consistent.

5

The process repeats — sampling, gradient computation, and parameter update — until a convergence criterion is met: loss ceases to decrease, a maximum number of iterations is reached, or validation performance stops improving. Early stopping on a held-out validation set guards against overfitting.

Common pitfalls

  • Setting the learning rate too high causes the loss to diverge or oscillate wildly; always monitor the loss curve during training.
  • Forgetting to shuffle the data before each epoch can introduce periodic bias into the gradient estimates.
  • Using a fixed learning rate throughout training often leaves performance on the table — learning rate schedules (warm-up, cosine decay) consistently improve final results.
  • Comparing optimisers fairly requires matching the total number of gradient evaluations (or wall-clock time), not just the number of epochs.
  • Adam's adaptive rates can suppress useful gradient signal in sparse settings; SGD with momentum sometimes generalises better for image classification tasks.

Frequently asked questions

Which optimiser should I choose — SGD, Adam, or something else?

Adam is a safe default for most deep learning tasks: it is robust to learning rate choice and converges quickly. SGD with momentum and a tuned learning rate schedule often achieves slightly better final generalisation on image classification benchmarks, but requires more careful tuning. AdaGrad suits sparse gradient problems (e.g., NLP with bag-of-words features), while RMSProp works well in recurrent network training.

How do I choose the learning rate?

The learning rate is the most important hyperparameter. Common practice is to start with 1e-3 for Adam and 1e-1 for SGD, then use a learning rate finder or grid search. Applying a warm-up phase followed by cosine or step decay nearly always improves results over a fixed rate.

What mini-batch size should I use?

Batch sizes between 32 and 256 are common. Smaller batches introduce more gradient noise (which can aid generalisation) and allow more frequent updates; larger batches reduce noise but may generalise less well and require more careful learning-rate scaling. When doubling the batch size, scale the learning rate proportionally (linear scaling rule) as a starting point.

How do I know when to stop training?

Monitor validation loss and stop when it ceases to improve for a set number of consecutive epochs — a technique called early stopping. This is more reliable than stopping at a fixed number of epochs and guards against overfitting.

Related methods

Sources

  1. Robbins, H. & Monro, S. (1951). A Stochastic Approximation Method. Annals of Mathematical Statistics, 22(3), 400-407.

    Open DOI reference
    Copied.
  2. Kingma, D.P. & Ba, J. (2015). Adam: A Method for Stochastic Optimization. International Conference on Learning Representations (ICLR 2015).

    Open publication reference
    Copied.