Bagging Ensemble
- Introduced by
- Leo Breiman
- Year
- 1996
- Type
- parallel ensemble
- Family
- Ensemble
What does it do?
Bagging, short for bootstrap aggregating, is an ensemble method that reduces variance by training multiple copies of a single learning algorithm on different random subsets of the training data.
Each subset is created via bootstrap sampling—randomly drawing samples with replacement. Predictions are combined through majority voting (classification) or averaging (regression). Introduced by Leo Breiman in 1996, bagging forms the foundation for random forests and is particularly effective for reducing overfitting in high-variance models.
The intuition behind the method
The core insight is that by training the same model on different random data subsets and averaging their predictions, you reduce variance without increasing bias. If a model overfits to specific training samples, different bootstrap samples expose different subsets of this overfitting, and averaging cancels out spurious patterns. Bagging works by exploiting randomness: each bootstrap sample is slightly different, so each trained model is slightly different, and their aggregate is more stable than any individual model.
History
Bagging was introduced by Leo Breiman in 1996, building on the bootstrap method from statistics (Efron, 1979). Breiman demonstrated that bagging reduces the prediction error of unstable learners by reducing variance. The method became foundational with the introduction of Random Forests (Breiman, 2001), which combine bagging with random feature selection. Bagging remains a standard technique in machine learning, especially for ensemble methods.
When to use it
Use bagging when working with high-variance models (e.g., decision trees, neural networks) that tend to overfit. It is most effective when base learners are unstable—sensitive to small changes in training data. Bagging is less effective for low-bias, low-variance models (e.g., k-NN with small k, linear regression on stable data). It is also well-suited for parallel computing since each base learner can be trained independently.
Limitations and considerations
- Provides no bias reduction; only effective for high-variance base learners.
- Computational cost scales linearly with the number of base learners.
- Bootstrap samples lack diversity compared to deliberate sampling strategies.
- May not improve performance when base models have high bias (e.g., linear models on complex data).
Source conditions do not replace independent scientific review or validation by an analysis engine.
Strengths
- Reduces variance without increasing bias, improving generalization.
- Works with any base learner type; can use unstable models like trees.
- Easily parallelizable—each bootstrap sample can be processed independently.
- Provides variance estimates through out-of-bag error calculation.
- Simple to implement with minimal hyperparameter tuning.
Applications
- Medical diagnosis: bagging decision trees for disease classification to improve robustness to noisy patient data.
- Financial risk modeling: ensemble of decision trees for credit default prediction across market conditions.
- Remote sensing: bagging image classifiers to classify satellite imagery with reduced overfitting.
- Genomics: ensemble methods for gene selection and disease prediction from high-dimensional microarray data.
- Manufacturing quality control: bagging neural networks for defect detection in production processes.
How it works
How it works
Create B bootstrap samples by randomly sampling from the training data with replacement, each of size equal to the original dataset.
Train an independent base learner (e.g., decision tree, neural network) on each bootstrap sample.
For a new input, pass it through all B trained base learners and collect their predictions.
Combine predictions by majority voting (classification) or averaging (regression).
Common pitfalls
- Using too few base learners (B < 20) reduces variance reduction benefit; diminishing returns typically occur around B = 100-500.
- Training base learners on overlapping subsets (e.g., consecutive data splits) instead of random bootstrap samples.
- Assuming bagging eliminates overfitting; it reduces variance but biased models remain biased.
- Not leveraging out-of-bag predictions for validation, missing an efficient alternative to cross-validation.
Frequently asked questions
How many bootstrap samples (B) should I use?
Start with B = 50-100 and monitor performance; typically, benefits plateau around B = 100-500. Increase B if variance reduction stalls. For critical applications, B > 1000 may be justified despite computational cost.
What is out-of-bag error and how do I use it?
Each bootstrap sample excludes approximately 1/3 of the original data (about 36.8% on average). These out-of-bag samples can be used to validate that specific model without needing a separate validation set. Average out-of-bag errors across all models estimate generalization error.
Is bagging better than cross-validation?
They serve different purposes. Cross-validation estimates model performance; bagging improves it. Use bagging to train an ensemble and out-of-bag estimates to validate, avoiding the overhead of k-fold cross-validation.
Can bagging improve low-variance models like k-NN?
Not effectively. Bagging is designed for high-variance models. Low-variance models (stable, simple) benefit little from variance reduction. Consider other ensemble methods or focus on bias reduction instead.
Related methods
Sources
Breiman, L. (1996). Bagging predictors. Machine Learning, 24(2), 123-140.
Open DOI referenceCopied.Efron, B. (1979). Bootstrap methods: another look at the jackknife. The Annals of Statistics, 7(1), 1-26.
Open DOI referenceCopied.