Skip to content
Back to search results

LightGBM

Introduced by
Ke, G. et al. (Microsoft)
Year
2017
Type
Gradient boosting decision tree ensemble
Font size

What does it do?

LightGBM is Microsoft's gradient boosting decision tree implementation, introduced by Ke and colleagues in 2017, that grows trees leaf-wise and bins features into histograms for speed.

On large datasets it is much faster than XGBoost while retaining strong predictive accuracy.

The intuition behind the method

Gradient boosting builds an ensemble one tree at a time, where each new tree corrects the errors left by the trees before it. LightGBM keeps this core idea but makes it fast: instead of testing every possible split point on continuous features, it groups feature values into a fixed number of histogram bins, and instead of growing every tree level by level, it always grows the leaf that promises the biggest improvement. The result is a model that converges quickly and scales to large tabular data.

History

LightGBM was introduced by Guolin Ke and colleagues at Microsoft in a 2017 NeurIPS paper, 'LightGBM: A Highly Efficient Gradient Boosting Decision Tree.' Its two distinguishing choices — histogram-based feature binning and leaf-wise (best-first) tree growth — were designed to make gradient boosting dramatically faster and more memory-efficient on large datasets, and it became one of the most widely used boosting libraries for tabular data alongside XGBoost.

When to use it

A strong choice for prediction, classification, and forecasting on large tabular datasets — cross-sectional, panel, or time-series — with continuous, categorical, binary, or ordinal features and at least about 200 observations. It is most valuable when training speed and scale matter and you can tune regularisation. Below roughly 200 observations the leaf-wise growth tends to overfit badly, and a level-wise method such as XGBoost or a more stable Random Forest is safer.

Limitations and considerations

  • Leaf-wise growth overfits on small samples unless carefully regularised.
  • Requires tuning num_leaves and min_data_in_leaf to control complexity.
  • On datasets below about 200 observations a level-wise method (XGBoost) or Random Forest is more reliable.
  • Like other boosted trees, it is a black box with no directly interpretable coefficients.

Source conditions do not replace independent scientific review or validation by an analysis engine.

Strengths

  • Much faster than XGBoost on large datasets thanks to histogram-based splitting.
  • Leaf-wise growth converges quickly and often reaches high accuracy.
  • Handles categorical features directly, without manual one-hot encoding.
  • Works for prediction, classification, and forecasting across cross-sectional, panel, and time-series data.

Applications

  • Prediction and classification on large tabular datasets.
  • Forecasting tasks on cross-sectional, panel, and time-series data.
  • Fast, scalable alternative to XGBoost when training time matters.
  • Modelling data with many categorical features without heavy preprocessing.

How it works

How it works

1

Trees are added sequentially: each new tree is fit to the residual errors of the current ensemble and added with a learning-rate shrinkage, so the model gradually reduces its loss step by step. Categorical features can be fed to LightGBM directly without manual one-hot encoding.

Formula
F_m(x) = F_{m-1}(x) + \eta\, h_m(x)
2

Continuous feature values are bucketed into a fixed number of discrete histogram bins before splitting. The algorithm then searches for the best split over these bins rather than every raw value, which is the main reason LightGBM is so much faster than level-wise implementations such as XGBoost on large data.

3

Rather than expanding all leaves at the same depth (level-wise), LightGBM grows leaf-wise: it always splits the leaf that yields the largest loss reduction. This converges faster but can overfit on small samples, so the number of leaves (num_leaves) and the minimum data per leaf (min_data_in_leaf) must be set to regularise the trees.

Common pitfalls

  • Using LightGBM on a small sample where leaf-wise growth severely overfits.
  • Leaving num_leaves and min_data_in_leaf at values too permissive for the data size.
  • One-hot encoding categorical variables manually instead of passing them to LightGBM directly.
  • Trusting in-sample accuracy without a held-out or cross-validated evaluation.

Frequently asked questions

How is LightGBM different from XGBoost?

Both are gradient boosting decision tree libraries. LightGBM uses histogram-based splitting and leaf-wise (best-first) tree growth, which makes it much faster on large datasets. XGBoost grows level-wise, which is often safer on smaller samples.

Why does LightGBM overfit on small datasets?

Its leaf-wise growth always splits the leaf with the largest gain, which can carve the data into too many leaves relative to its size. On small samples (below about 200 observations) this leads to serious overfitting, so a level-wise method or Random Forest is recommended.

Do I need to one-hot encode categorical variables?

No. LightGBM can take categorical features directly, so manual one-hot encoding is usually unnecessary and can even hurt performance.

Which settings control overfitting?

The number of leaves (num_leaves) and the minimum data per leaf (min_data_in_leaf) are the key regularisation controls; tightening them limits tree complexity relative to the dataset size.

Related methods

Sources

  1. Ke, G., Meng, Q., Finley, T., Wang, T., Chen, W., Ma, W., Ye, Q. & Liu, T.-Y. (2017). LightGBM: A Highly Efficient Gradient Boosting Decision Tree. Advances in Neural Information Processing Systems (NeurIPS) 30, 3146–3154.

    Open publication reference
    Copied.