Skip to content
Back to search results

Stacking

Introduced by
Wolpert, D.H.
Year
1992
Type
Ensemble (heterogeneous meta-learning)
Font size

What does it do?

Stacking, or stacked generalization, is an ensemble method introduced by David Wolpert in 1992 that combines the outputs of several different base models (Level-0) through a separate meta-model (Level-1).

Unlike bagging and boosting, it deliberately uses heterogeneous model types, and it is the standard final-stage strategy in Kaggle competitions.

The intuition behind the method

Different model families make different kinds of mistakes: a tree ensemble, a support vector machine and a linear model each capture the data in their own way. Stacking treats their predictions as new features and trains one more model to learn how best to blend them, so the meta-model can lean on each base model where it is strongest and discount it where it is weak — producing a combined prediction that usually beats any single model.

History

Stacked generalization was introduced by David Wolpert in 1992 as a way to combine multiple generalizers through a higher-level model. The idea was later formalised and given strong theoretical footing by van der Laan, Polley and Hubbard in 2007 under the name Super Learner, which uses cross-validation to combine a library of base learners. Stacking has since become the standard last-stage strategy in machine-learning competitions, where ensembles of heterogeneous models routinely top the leaderboards.

When to use it

Use stacking when predictive accuracy is the priority and you have several reasonably strong, diverse base models to combine, on cross-sectional data with continuous, categorical or binary features and at least about 100 observations. It does not require normally distributed data. Below roughly 100 observations the k-fold cross-validation cannot generate enough reliable Level-0 predictions and the meta-model overfits, so a single tuned model such as random forest or XGBoost is safer.

Limitations and considerations

  • Needs a fairly large sample (about 100+) so that k-fold cross-validation can generate reliable Level-0 predictions.
  • Gains shrink when the base models are highly correlated rather than diverse.
  • More complex and compute-heavy than a single model, and harder to interpret as a whole.
  • On small samples the diversity of base models cannot be achieved and a single model is usually enough.

Source conditions do not replace independent scientific review or validation by an analysis engine.

Strengths

  • Often achieves the highest predictive accuracy by exploiting heterogeneous base models that make different errors.
  • Flexible: any mix of model families can serve as base learners, with a simple meta-model on top.
  • Assumption-light — no normality requirement, and it handles continuous, categorical and binary features.
  • The proven final-stage strategy in competitive prediction settings such as Kaggle.

Applications

  • Final-stage ensembling in predictive-modelling competitions such as Kaggle.
  • Combining heterogeneous models for high-accuracy classification and prediction.
  • Super Learner ensembles in biostatistics and epidemiology.
  • Blending strong but diverse models when squeezing out extra accuracy matters.

How it works

How it works

1

A diverse set of base learners (Level-0) is trained on the data — for example a random forest, a gradient-boosting model and an SVM. The more varied and mutually independent these models are, the more the stack can gain; when the base models are highly correlated with one another, the benefit of stacking shrinks.

2

Level-0 predictions are produced through k-fold cross-validation rather than on the same data the base models were fitted on. This out-of-fold scheme is essential to prevent data leakage: each observation's meta-features come from models that did not see it during training.

3

A meta-model (Level-1) is trained on the cross-validated base-model outputs to produce the final prediction. The meta-learner is usually kept simple — logistic regression, ridge regression or a small XGBoost — so that it learns sensible weights for combining the base models without itself overfitting.

Formula
\hat{y} = g\big(f_1(x), f_2(x), \dots, f_M(x)\big)

Common pitfalls

  • Generating Level-0 predictions on the same data the base models were trained on, causing data leakage into the meta-model.
  • Stacking many highly correlated base models and expecting a large gain.
  • Using an overly complex meta-model that overfits the base-model outputs instead of a simple regressor.
  • Applying stacking to a small dataset where base-model diversity and reliable cross-validation are impossible.

Frequently asked questions

How is stacking different from bagging and boosting?

Bagging and boosting build many models of the same type, whereas stacking deliberately combines heterogeneous model families and learns how to blend them with a separate meta-model. That diversity is where stacking's advantage comes from.

Why use cross-validation to build the meta-features?

If the meta-model is trained on predictions the base models made on their own training data, information leaks and the stack overfits. Producing Level-0 predictions out-of-fold via k-fold cross-validation gives the meta-model honest inputs.

What should the meta-model be?

Keep it simple — logistic regression, ridge regression or a small XGBoost. A simple meta-learner learns sensible blending weights without overfitting the base-model outputs.

When should I not use stacking?

When you have fewer than about 100 observations, cross-validation cannot produce enough reliable Level-0 predictions and base-model diversity is hard to achieve, so a single tuned random forest or XGBoost is the safer choice.

Related methods

Sources

  1. Wolpert, D.H. (1992). Stacked Generalization. Neural Networks, 5(2), 241–259.

    Open DOI reference
    Copied.
  2. van der Laan, M.J., Polley, E.C. & Hubbard, A.E. (2007). Super Learner. Statistical Applications in Genetics and Molecular Biology, 6(1), Article 25.

    Open DOI reference
    Copied.