Gradient Boosting
- Introduced by
- Friedman, J. H.
- Year
- 2001
- Type
- Ensemble (sequential boosting of decision trees)
What does it do?
Gradient Boosting is an ensemble learning method, formalised by Jerome H. Friedman in 2001, that combines a sequence of weak learners — typically shallow decision trees — so that each new tree is fitted to minimise the residual errors of the trees before it.
It is the core algorithm behind popular implementations such as XGBoost, LightGBM and CatBoost.
The intuition behind the method
Rather than building one strong model at once, Gradient Boosting builds many small trees in sequence, where each new tree focuses on the mistakes the current ensemble still makes. By repeatedly nudging the prediction in the direction that most reduces the remaining error, a chain of weak learners adds up to one highly accurate predictor.
History
Gradient Boosting was formalised by Jerome H. Friedman in his 2001 paper Greedy Function Approximation: A Gradient Boosting Machine, which framed boosting as gradient descent in function space. The idea — adding weak learners that each fit the residuals of the current ensemble — became the foundation for the most successful tabular-data algorithms in practice, including XGBoost, LightGBM and CatBoost.
When to use it
A strong choice for accurate classification and prediction on cross-sectional or panel tabular data with at least about 100 observations, across continuous, categorical, binary and ordinal features. Feature scaling is generally not required, but hyperparameter tuning of the number of trees, learning rate and maximum depth is critical, and early stopping is recommended to guard against overfitting. With fewer than about 100 observations the sequential trees tend to memorise noise, and Random Forest or a single decision tree is safer.
Limitations and considerations
- Hyperparameter tuning of the number of trees, learning rate and maximum depth is critical and time-consuming.
- Sequential boosting is prone to overfitting if not regularised, which is why early stopping is recommended.
- On small samples (n below about 100) the sequential trees memorise noise, and even early stopping is insufficient.
- With too little data there is not enough cross-validation material to optimise hyperparameters reliably.
Source conditions do not replace independent scientific review or validation by an analysis engine.
Strengths
- Very high predictive accuracy on tabular data for both classification and prediction.
- Feature scaling is generally unnecessary, and it handles continuous, categorical, binary and ordinal features.
- Forms the core algorithm behind widely used libraries such as XGBoost, LightGBM and CatBoost.
- Early stopping and a small learning rate give fine control over the bias–variance trade-off.
Applications
- High-accuracy classification and prediction on cross-sectional and panel tabular data.
- Risk scoring and outcome prediction where maximum predictive performance is the priority.
- Modelling complex nonlinear relationships among mixed-type features without manual scaling.
- Serving as the engine behind production gradient-boosting libraries such as XGBoost and LightGBM.
How it works
How it works
Gradient Boosting trains weak learners — usually shallow decision trees — one after another rather than independently. The ensemble starts from a simple initial guess, and every subsequent tree is added to correct what the current model still gets wrong, so accuracy improves step by step.
At each round the method looks at the residual errors of the current ensemble and fits the next tree to those residuals (the negative gradient of the loss). The new tree is added with a small learning-rate weight, so the prediction is updated cautiously: F_m(x) = F_{m-1}(x) + nu * h_m(x). This sequential, gradient-guided correction is what gives the method its name and its accuracy.
Formula
F_{m}(x) = F_{m-1}(x) + \nu \, h_m(x)The final prediction is the sum of the initial estimate and all the small tree corrections accumulated along the way. Because the trees are added greedily to chase the remaining error, tuning the number of trees, the learning rate and the tree depth — together with early stopping — is critical to avoid overfitting.
Common pitfalls
- Skipping early stopping and letting the boosting run for too many rounds until it overfits.
- Setting the learning rate too high so each tree overcorrects instead of nudging the prediction gently.
- Applying it to a sample below about 100 observations where the sequential trees memorise noise.
- Accepting default hyperparameters instead of tuning n_estimators, learning_rate and max_depth.
- Treating it as needing feature scaling like distance-based models, when scaling is generally unnecessary.
Frequently asked questions
How is Gradient Boosting different from Random Forest?
Random Forest builds many independent trees in parallel and averages them, whereas Gradient Boosting builds trees sequentially, each one fitted to correct the residual errors of the ones before it. Boosting often reaches higher accuracy but is more sensitive to overfitting and needs careful tuning.
Why is early stopping recommended?
Because each new tree chases the remaining error, an over-long boosting run starts fitting noise. Early stopping halts training once validation performance stops improving, which is the main safeguard against overfitting.
Do I need to scale my features?
Feature scaling is generally not required for tree-based gradient boosting, since splits depend on order rather than magnitude. The critical knobs are instead the number of trees, the learning rate and the tree depth.
What sample size do I need?
At least about 100 observations. Below that, the sequential trees memorise noise and even early stopping is insufficient, so Random Forest or a single decision tree is the safer choice.
Related methods
Sources
Friedman, J. H. (2001). Greedy Function Approximation: A Gradient Boosting Machine. Annals of Statistics, 29(5), 1189–1232.
Open DOI referenceCopied.