Extra Trees
- Introduced by
- Geurts, P.; Ernst, D.; Wehenkel, L.
- Year
- 2006
- Type
- Ensemble (extremely randomized decision trees)
- Family
- Machine learning
What does it do?
Extra Trees (Extremely Randomized Trees), introduced by Geurts, Ernst, and Wehenkel in 2006, is an ensemble of decision trees that pushes randomisation further than Random Forest.
Both the candidate features and the split thresholds are chosen completely at random at each node, eliminating the greedy search over thresholds. This extra randomness reduces variance, often matches or exceeds Random Forest accuracy, and runs substantially faster at training time.
The intuition behind the method
Random Forest already de-correlates trees by sampling features randomly at each split, but it still searches for the best threshold among those features. Extra Trees goes one step further: for each candidate feature it picks a random threshold rather than optimising it. At first this sounds like it would hurt accuracy, but because the resulting trees are even less correlated with one another, the ensemble average cancels out more variance. The loss from any single bad split is tiny once you average over many trees, and the speed gain from not searching for an optimal threshold is substantial.
History
Extra Trees was proposed by Pierre Geurts, Damien Ernst, and Louis Wehenkel in a 2006 Machine Learning paper. The authors showed that introducing full randomisation of split thresholds — rather than searching for optimal cuts — substantially reduces variance while trading a small increase in bias. The method has since become a standard alternative to Random Forest in the scikit-learn ecosystem and is widely used as a fast, high-accuracy baseline for tabular-data benchmarks.
When to use it
Use Extra Trees when you want fast training and strong accuracy on tabular data with continuous or mixed-type features. It is particularly useful when training time is a constraint — its removal of threshold optimisation makes it considerably faster than Random Forest at comparable accuracy. It suits classification and regression tasks with moderate to large samples (roughly 50 or more observations). Prefer Random Forest or gradient boosting if you need built-in out-of-bag estimates or if dataset noise is very high and you suspect fully random thresholds hurt accuracy. For very small datasets, simpler models such as logistic regression or a single decision tree are safer and more interpretable.
Limitations and considerations
- No built-in out-of-bag estimate (unlike Random Forest), so cross-validation is required for unbiased error assessment.
- Fully random thresholds can hurt on very noisy data where optimal splits are informative.
- Black-box: provides no explicit coefficients, limiting interpretability for audiences expecting effect sizes.
- Memory usage grows with the number of trees and tree depth.
Source conditions do not replace independent scientific review or validation by an analysis engine.
Strengths
- Faster training than Random Forest because no threshold optimisation is performed at each split.
- High diversity among trees leads to strong variance reduction and often matches or beats Random Forest accuracy.
- Handles mixed feature types and nonlinear interactions without preprocessing assumptions.
- Provides native feature importances for variable ranking and selection.
- Scales well to high-dimensional data with many irrelevant features.
Applications
- Fast classification and regression baselines on tabular data.
- Feature selection and importance ranking in high-dimensional datasets.
- Predictive modelling in bioinformatics, remote sensing, and sensor-data analysis.
- Pipeline hyperparameter search where training speed is critical.
- Large-scale data scenarios where Random Forest training time is prohibitive.
How it works
How it works
Unlike Random Forest, Extra Trees by default trains each tree on the entire training set rather than a bootstrap sample. This reduces bias introduced by sampling but means there is no built-in out-of-bag set; cross-validation is therefore needed for honest error estimation.
At each internal node, a random subset of features is drawn. For each candidate feature a split threshold is sampled uniformly at random from the observed range of that feature's values — no search for the optimal cut-point is performed. The feature-threshold pair yielding the best score (e.g., Gini impurity reduction) among those random candidates is selected.
Trees are grown to full depth by default, meaning leaves are as pure as possible. Full depth, combined with the randomised splits, is what drives variance down across the ensemble: individual trees are high-variance but highly diverse.
Predictions from all trees are combined: majority vote for classification and mean output for regression. The strong diversity among trees means that individual errors tend to cancel, making the ensemble more stable than any single tree.
Formula
\hat{y} = \text{mode/mean of } T \text{ trees}Common pitfalls
- Skipping cross-validation and relying on in-sample accuracy, since no OOB estimate is produced by default.
- Ignoring class imbalance — use class_weight='balanced' or resampling before fitting.
- Reporting only accuracy without complementary metrics such as F1, AUC, or RMSE.
- Treating feature importance rankings as stable without checking reproducibility across seeds.
Frequently asked questions
How does Extra Trees differ from Random Forest?
Both are ensembles of decision trees with random feature subsets at each split. The key difference is that Extra Trees picks split thresholds completely at random rather than optimising them, and by default trains on the full training set rather than bootstrap samples. This makes Extra Trees faster and typically produces more diverse, less correlated trees.
Which should I prefer — Extra Trees or Random Forest?
When training speed matters or the data are relatively clean, Extra Trees often matches or exceeds Random Forest accuracy at lower computational cost. If you want a built-in out-of-bag estimate or your data are noisy (where better split thresholds help), Random Forest is safer.
How many trees do I need?
Start with 100 trees and increase until performance stabilises on a validation set or in cross-validation. More trees improve stability but increase memory and prediction latency.
Does Extra Trees overfit?
Individual trees are grown to full depth and can overfit in isolation, but the ensemble average strongly reduces variance. Overfitting of the full ensemble is generally rare, though on very small datasets with many features it can still occur.
How do I assess model performance without an OOB set?
Use k-fold cross-validation — typically 5 or 10 folds. Evaluate classification tasks with accuracy, F1, and AUC; evaluate regression tasks with RMSE and R-squared.
Related methods
Sources
Geurts, P., Ernst, D. & Wehenkel, L. (2006). Extremely randomized trees. Machine Learning, 63(1), 3–42.
Open DOI referenceCopied.Extra-Trees. Wikipedia.
Open publication referenceCopied.