Skip to content
Back to search results

Random Forest

Introduced by
Breiman, L.
Year
2001
Type
Ensemble (bagging of decision trees)
Font size

What does it do?

Random Forest is an ensemble learning method, introduced by Leo Breiman in 2001, that grows many decision trees on bootstrap samples of the data and combines their votes to produce strong classification and regression.

By pooling many slightly different trees, it produces more accurate and more stable predictions than any single tree.

The intuition behind the method

A single decision tree grown deeply fits the training data closely but is unstable: a small change in the data can produce a very different tree. Random Forest tackles this by building many trees, each on a slightly different random view of the data, and letting them vote. The errors of individual trees tend to cancel out, so the combined prediction is far more reliable than any one tree on its own.

History

Random Forest was proposed by Leo Breiman in 2001, combining bootstrap aggregation with random feature selection at each split. It became one of the most widely used and reliable off-the-shelf machine-learning methods for tabular data, and is a standard reference in texts such as An Introduction to Statistical Learning (Chapter 8). More recently, tools like SHAP values have brought its interpretability closer to that of logistic regression, increasing its acceptance in the social sciences.

When to use it

A good fit for accurate, robust classification, regression, or prediction on tabular data with many — possibly nonlinear and interacting — features and at least about 50 observations, when predictive performance matters more than reporting explicit coefficients. Random Forest is assumption-light: it does not require normally distributed data and handles continuous, categorical, binary, ordinal, and count features. Hyperparameter tuning is recommended. When sample sizes fall below about 50, simpler models such as logistic or linear regression are safer.

Limitations and considerations

  • No coefficients to interpret; its black-box nature can clash with reviewers who expect explicit effects.
  • Hyperparameter tuning is recommended, and large forests cost memory and compute.
  • On small samples (n below about 50) many trees overfit and cross-validation results become misleading.
  • Feature-importance estimates can be unstable and non-reproducible at small sample sizes.

Source conditions do not replace independent scientific review or validation by an analysis engine.

Strengths

  • Strong off-the-shelf accuracy on tabular data for both classification and regression.
  • Resistant to overfitting thanks to bagging plus random feature selection at each split.
  • Built-in feature importance integrates variable ranking directly into the model.
  • Out-of-bag error gives an honest validation estimate without a separate cross-validation pass.
  • Assumption-light: no normality requirement, and it handles mixed feature types and nonlinear interactions.

Applications

  • Classification and regression on tabular data with many features.
  • Variable selection and ranking via native feature importance.
  • Risk scoring and prediction where robustness matters more than explicit coefficients.
  • Baseline modelling before trying more complex ensembles such as gradient boosting.

How it works

How it works

1

Many bootstrap samples are drawn from the training set — each is a random sample taken with replacement, so the trees see slightly different data. Observations left out of a given sample (the out-of-bag set) provide a built-in validation estimate without needing a separate hold-out fold.

2

At each split in every tree, only a random subset of the available features is considered as candidates. This random feature selection de-correlates the trees so they do not all latch onto the same dominant predictors, which is what lets the ensemble cut variance below that of any single tree.

3

Each tree makes its own prediction, and the forest combines them: a majority vote across the trees for classification, or the average of their outputs for regression. This aggregation is what turns many noisy trees into one robust predictor.

Formula
\hat{y} = \text{mode/mean of } T \text{ trees}

Common pitfalls

  • Skipping a train/test split or cross-validation and trusting in-sample performance.
  • Ignoring class imbalance instead of re-weighting classes (class_weight='balanced') or resampling.
  • Reporting only accuracy without F1 and AUC alongside it.
  • Treating a small-sample feature-importance ranking as stable.
  • Accepting default hyperparameters blindly instead of tuning the number of trees.

Frequently asked questions

How many trees should the forest have?

More trees reduce variance and stabilise the vote at the cost of compute and memory. A common starting point is around 100 trees, increasing until the out-of-bag error stops improving.

Does Random Forest overfit?

It is highly resistant to overfitting because bagging and random feature selection de-correlate the trees and average out their individual errors. The main risk is fitting many deep trees on a very small dataset, where the model overfits and cross-validation becomes misleading.

Can I trust the feature importance?

Feature importance is useful for ranking predictors, but at small sample sizes it can be unstable and non-reproducible, so treat small-sample rankings with caution. Pairing it with SHAP analysis gives a clearer view of each feature's contribution.

What should I check before and after fitting?

Beforehand, handle missing values, encode categorical variables, and set up a train/test split or k-fold cross-validation. Afterwards, evaluate the out-of-bag error, run cross-validation, and inspect feature contributions (for example with SHAP).

Related methods

Sources

  1. Breiman, L. (2001). Random Forests. Machine Learning, 45, 5–32.

    Open DOI reference
    Copied.
  2. James, G., Witten, D., Hastie, T. & Tibshirani, R. (2013). An Introduction to Statistical Learning (Ch. 8). Springer.

    ISBN: 978-1-4614-7138-7
    Copied.