Bagging (Bootstrap Aggregating)
- Introduced by
- Breiman, L.
- Year
- 1996
- Type
- Ensemble meta-algorithm (variance reduction via bootstrap aggregation)
What does it do?
Bagging, short for Bootstrap Aggregating, is an ensemble meta-algorithm introduced by Leo Breiman in 1996 that trains multiple copies of a base learner on independently drawn bootstrap samples of the training data and combines their predictions — by averaging for regression or majority vote for classification — to produce a final predictor with substantially lower variance than any single base learner.
The intuition behind the method
Imagine asking one expert for an opinion: their answer depends heavily on their particular experiences. Now imagine asking one hundred experts, each with a slightly different background, and averaging their answers. Individual quirks cancel out and the group consensus is far more reliable. Bagging does exactly this for machine-learning models: it generates many slightly different training sets by resampling the original data with replacement, trains one model per set, and merges all predictions. Each model sees a different slice of reality, so their individual errors tend not to align, and the aggregate is much more stable than any single model.
History
Leo Breiman published Bagging Predictors in Machine Learning in 1996, motivated by the observation that unstable learners — those whose output changes substantially under small data perturbations — could be stabilised by averaging over bootstrap replicates of the training set. The theoretical justification rests on the bias-variance decomposition: averaging reduces variance while leaving bias unchanged. Breiman demonstrated consistent accuracy improvements over single decision trees on a battery of classification and regression benchmarks. Bagging rapidly became a foundational ensemble technique and directly inspired Random Forest (Breiman, 2001), which adds random feature subsampling at each split to further de-correlate the trees and achieve even greater variance reduction. Today, bagging serves as the conceptual backbone for a broad family of ensemble methods and is a standard topic in graduate-level machine-learning curricula.
When to use it
Bagging is appropriate whenever the base learning algorithm is high-variance — that is, sensitive to small perturbations in training data — and reducing that variance is the main source of error. Fully grown, unpruned decision trees are the canonical example. Bagging is suitable for both classification and regression on tabular data, requires no assumptions about data distribution, and handles mixed feature types. It is most beneficial when the dataset is large enough to support B independent bootstrap samples of quality (at least about 30 observations is a practical lower bound, though hundreds are preferable). Bagging is not beneficial when the base learner is already stable (such as linear regression on well-specified problems) or when the dominant error source is bias rather than variance — in such cases boosting methods are more appropriate.
Limitations and considerations
- Does not reduce bias: if the base learner underfits, bagging cannot correct it.
- Ensemble predictions are harder to interpret than a single model; no analogue of a single decision tree's visual structure.
- Memory and storage scale linearly with B: keeping hundreds of large models in memory can be prohibitive.
- Gains diminish rapidly as B grows; performance usually plateaus well before B = 500.
- On stable base learners (e.g. linear regression with correct specification) bagging yields negligible improvement.
Source conditions do not replace independent scientific review or validation by an analysis engine.
Strengths
- Directly reduces estimator variance without increasing bias, producing more stable predictions than any single base model.
- Trivially parallelisable: each bootstrap iteration is independent, making large ensembles computationally tractable.
- Built-in OOB error estimation provides an honest generalisation estimate at no extra data cost.
- Model-agnostic meta-algorithm: compatible with any sufficiently unstable base learner (trees, neural networks, nearest neighbours).
- Robust to outliers in the training set: any single unusual observation is absent from most bootstrap samples.
Applications
- Constructing stable classification and regression models on noisy tabular data where a single decision tree would overfit.
- Generating prediction intervals by inspecting the spread of predictions across the B base learners.
- Feature importance estimation via the decrease in OOB accuracy when each feature is permuted across the ensemble.
- Ensemble baseline: a quick benchmark to quantify how much a more sophisticated method (boosting, stacking) improves over simple aggregation.
How it works
How it works
B independent bootstrap samples are drawn from the original training set of size n. Each sample is generated by drawing n observations with replacement from the original data, so on average each sample contains about 63.2% of the unique training instances; the remaining ~36.8% are left out and form the out-of-bag (OOB) set for that iteration.
Formula
\mathcal{D}_b = \{(\mathbf{x}_i^*, y_i^*)\}_{i=1}^{n}, \quad (\mathbf{x}_i^*, y_i^*) \overset{\text{i.i.d.}}{\sim} \hat{F}_n, \quad b = 1, \dots, BA base learning algorithm — most often a decision tree grown to full depth, but any sufficiently unstable learner qualifies — is fitted independently on each bootstrap sample, yielding B distinct models. The key requirement is instability: if small data perturbations cause large model changes, bagging yields the greatest variance reduction.
Formula
\hat{f}_b = \mathcal{A}(\mathcal{D}_b)For regression tasks the B model outputs are averaged. This averaging step is the primary mechanism for variance reduction: under independence assumptions the variance of the average of B identically distributed models is 1/B of the variance of a single model, and even under positive correlation the reduction is substantial.
Formula
\hat{f}_{\text{bag}}(\mathbf{x}) = \frac{1}{B} \sum_{b=1}^{B} \hat{f}_b(\mathbf{x})For classification tasks each base learner casts one vote for a class label, and the class with the most votes is returned as the final prediction. Ties are broken arbitrarily. Plurality voting is asymptotically equivalent to averaging the class probability estimates produced by each base learner.
Formula
\hat{y}(\mathbf{x}) = \underset{k}{\operatorname{arg\,max}} \sum_{b=1}^{B} \mathbf{1}\!\left[\hat{f}_b(\mathbf{x}) = k\right]Because each base learner is trained on roughly 63.2% of the data, the remaining observations can serve as an honest held-out test set for that learner. Averaging these OOB predictions across all learners that did not see each observation gives a nearly unbiased estimate of generalisation error without requiring a separate validation fold or cross-validation.
Formula
\widehat{\text{Err}}_{\text{OOB}} = \frac{1}{n} \sum_{i=1}^{n} L\!\left(y_i,\; \frac{1}{|\mathcal{B}_i^c|} \sum_{b \in \mathcal{B}_i^c} \hat{f}_b(\mathbf{x}_i)\right)Common pitfalls
- Applying bagging to a low-variance base learner (such as ridge regression) and expecting large accuracy gains.
- Setting B too small (e.g. B < 10) so that the aggregate is still noisy and OOB estimates are unreliable.
- Ignoring class imbalance: bagging preserves the class distribution of the original data in each bootstrap sample, so severe imbalance is replicated across all learners.
- Conflating bagging with boosting: bagging trains models in parallel on independent samples to reduce variance, while boosting trains models sequentially to reduce bias — they address different error sources.
- Treating OOB error as a substitute for proper held-out evaluation when the dataset is very small, as OOB estimates can be optimistic in that regime.
Frequently asked questions
How many bootstrap samples B should I use?
Breiman (1996) found that performance stabilises relatively quickly; B = 50 to 100 is often sufficient for decision trees. Increasing B beyond a few hundred yields diminishing returns. Monitor the OOB error as B grows and stop when it plateaus.
What is the difference between bagging and Random Forest?
Random Forest extends bagging by adding random feature subsampling at each split of every tree. This additional randomisation further de-correlates the base learners, reducing variance beyond what bagging alone achieves. Bagging with fully grown trees is therefore a special case of Random Forest where all features are considered at every split.
Can bagging be applied to learners other than decision trees?
Yes. Any learner that is unstable — whose predictions change substantially when the training data change slightly — can benefit from bagging. Neural networks, k-nearest neighbours, and unpruned regression trees all qualify. Stable learners such as linear regression with a well-specified model benefit little because there is little variance to reduce.
Is the OOB error a reliable substitute for cross-validation?
For moderate to large datasets the OOB error is a nearly unbiased estimate of generalisation error and is computationally free since no extra model fits are required. On very small datasets (n below about 50) OOB estimates can be optimistic and cross-validation is preferable.
Related methods
Sources
Breiman, L. (1996). Bagging Predictors. Machine Learning, 24(2), 123–140.
Open DOI referenceCopied.Hastie, T., Tibshirani, R. & Friedman, J. (2009). The Elements of Statistical Learning (2nd ed., Ch. 8.7). Springer.
ISBN: 978-0-387-84857-0Copied.James, G., Witten, D., Hastie, T. & Tibshirani, R. (2013). An Introduction to Statistical Learning (Ch. 8.2). Springer.
ISBN: 978-1-4614-7138-7Copied.