Skip to content
Back to search results

XGBoost

Introduced by
Chen, T. & Guestrin, C.
Year
2016
Type
Ensemble (gradient-boosted decision trees)
Font size

What does it do?

XGBoost (Extreme Gradient Boosting) is a scalable tree-boosting algorithm introduced by Tianqi Chen and Carlos Guestrin in 2016.

It builds a strong predictor by adding decision trees one at a time, each correcting the errors left by the trees before it, and is a powerful prediction method widely used in competitions.

The intuition behind the method

Instead of growing many independent trees and averaging them, XGBoost grows trees in sequence: each new tree focuses on the cases the current model still gets wrong. Step by step, the ensemble chips away at the remaining error, while a regularisation penalty keeps the individual trees simple so the combined model generalises rather than memorising the training data.

History

XGBoost was introduced by Tianqi Chen and Carlos Guestrin in their 2016 paper 'XGBoost: A Scalable Tree Boosting System' at ACM SIGKDD. By combining regularised gradient boosting of decision trees with a scalable, efficient implementation, it became one of the most widely adopted prediction algorithms, especially prominent in data-science competitions.

When to use it

Use XGBoost for accurate classification or prediction on tabular data with continuous, categorical, binary, or count features and at least about 100 observations, when predictive performance is the priority. It does not require normally distributed data; sufficient data and thoughtful feature engineering are the main practical assumptions. With fewer than ~100 cases, gradient boosting tends to overfit and a simpler model (such as logistic regression or a single decision tree) is more dependable.

Limitations and considerations

  • Needs enough data — roughly 100+ observations — or it overfits and cross-validation results become unreliable.
  • On small samples there is not enough data to tune the many hyperparameters, so a simpler model generalises better.
  • Many hyperparameters and a black-box ensemble make it harder to set up and interpret than a single model.

Source conditions do not replace independent scientific review or validation by an analysis engine.

Strengths

  • Strong predictive accuracy on tabular data, which is why it is so common in competitions.
  • Built-in regularisation penalises tree complexity and helps control overfitting.
  • Tolerates missing values and handles mixed feature types (continuous, categorical, binary, count).
  • Scales well and benefits directly from feature engineering.

Applications

  • Classification and prediction on tabular data with many features.
  • Competition-grade predictive modelling where accuracy is paramount.
  • Risk scoring and prediction tasks on continuous, categorical, binary, and count features.
  • Modelling problems where missing values and engineered features are common.

How it works

How it works

1

XGBoost is a gradient-boosting method: it combines many shallow decision trees into one ensemble. Trees are added one after another, and every new tree is fit to reduce the prediction error that remains after the trees already in the model.

2

The model minimises an objective that sums a loss term measuring fit to the data plus a regularisation term that penalises tree complexity. This explicit complexity penalty is what helps XGBoost control overfitting while still fitting flexible, nonlinear relationships.

Formula
\mathcal{L} = \sum_i l(\hat{y}_i, y_i) + \sum_k \Omega(f_k)
3

At each boosting round the current prediction is updated by adding the output of a newly fitted tree. Each tree is chosen to move the predictions in the direction that most reduces the objective, so the ensemble's accuracy improves round by round.

Formula
\hat{y}_i^{(t)} = \hat{y}_i^{(t-1)} + f_t(x_i)
4

After all rounds the final prediction is the sum of the contributions of every tree, used for classification or for predicting a target. Because the model tolerates missing values and benefits from feature engineering, it performs strongly on tabular data once enough observations are available.

Common pitfalls

  • Applying it to small samples (n < 100) where gradient boosting overfits and CV estimates are not trustworthy.
  • Skipping hyperparameter tuning via cross-validation (e.g. grid search) and accepting defaults.
  • Leaving missing-value handling and categorical encoding undecided even though XGBoost tolerates gaps.
  • Failing to set up a proper train/test split or k-fold cross-validation before reporting performance.

Frequently asked questions

How much data does XGBoost need?

Around 100 observations is a sensible minimum. With fewer than 100 cases gradient boosting tends to overfit and cross-validation results become unreliable, so a simpler model such as logistic regression or a single decision tree is more dependable.

Does XGBoost require normally distributed data?

No. It makes no normality assumption and works with continuous, categorical, binary, and count features. The main practical requirements are having enough data and doing some thoughtful feature engineering.

How is XGBoost different from Random Forest?

Random Forest grows trees independently and averages them to reduce variance. XGBoost grows trees sequentially, each one correcting the errors of the previous ones, and adds a regularisation penalty on tree complexity, which can reach higher accuracy with careful tuning.

How do I interpret an XGBoost model?

Since the model is an ensemble of many trees, contributions are usually examined with feature-attribution analysis such as SHAP, alongside test-set metrics like RMSE or AUC and cross-validated hyperparameter tuning.

Related methods

Sources

  1. Chen, T. & Guestrin, C. (2016). XGBoost: A Scalable Tree Boosting System. Proceedings of the 22nd ACM SIGKDD, 785–794.

    Open DOI reference
    Copied.