Skip to content
Back to search results

Voting Ensemble

Introduced by
Lam & Suen; Kuncheva, L. I. (systematic treatment)
Year
1990s–2004
Type
Ensemble (combination of multiple classifiers by vote)
Family
Machine learning
Font size

What does it do?

A voting ensemble trains several diverse classifiers independently and combines their predictions by a vote: hard voting picks the class chosen by the most models, while soft voting averages their class-probability estimates, optionally with per-model weights.

The combination usually outperforms any individual member, and requires no additional training after the base models are fitted.

The intuition behind the method

Imagine asking five doctors for a diagnosis and going with the majority opinion. Each physician may err in different ways, but pooling their verdicts cancels idiosyncratic mistakes. A voting ensemble works the same way: diverse models fail on different examples, so their combined vote is more reliable than any single model. Soft voting sharpens this by letting more confident models contribute proportionally more to the final decision through their probability scores.

History

Combining classifiers by majority vote was studied systematically from the early 1990s, with notable theoretical grounding by Lam and Suen (1994) and Kittler et al. (1998). Ludmila Kuncheva's 2004 monograph provided the definitive treatment of combining pattern classifiers, covering hard and soft voting, weighted voting, and the role of diversity. Soft voting gained practical popularity through scikit-learn's VotingClassifier, which made it straightforward to combine arbitrary estimators in a single API call.

When to use it

Use a voting ensemble when you have two or more already-trained diverse classifiers and want a quick, interpretable combination with no extra training cost. It is especially valuable when individual models have similar average accuracy but different error profiles — combining them yields genuine gains. Soft voting is preferable when base models output calibrated probabilities. Do not use voting ensembles as a substitute for proper model selection: if all base learners are weak or make correlated errors, combining them will not help. Avoid when interpretability of a single model is required by stakeholders or regulation.

Limitations and considerations

  • Gains are minimal or absent when base learners make highly correlated errors.
  • Soft voting requires that all base learners output calibrated, comparable probability estimates.
  • Inference cost scales linearly with the number of base learners; large ensembles can be slow.
  • No mechanism to learn optimal combination weights automatically — unlike stacking, weights must be set manually.

Source conditions do not replace independent scientific review or validation by an analysis engine.

Strengths

  • Simple to implement: no meta-learner training needed beyond fitting the base models.
  • Often improves accuracy and robustness over any single base learner when members are diverse.
  • Soft voting exploits probability calibration, typically outperforming hard voting.
  • Supports heterogeneous model families, allowing any combination of classifiers.
  • Easy to add or remove base learners without retraining the ensemble infrastructure.

Applications

  • Combining logistic regression, SVM, and tree-based models for clinical risk prediction.
  • Robust text classification by pooling predictions from multiple language-model fine-tunes.
  • Competition machine learning (Kaggle-style) where combining top-k diverse models improves leaderboard score.
  • Production settings where ensemble robustness is more important than single-model simplicity.
  • Baseline ensemble before investing effort in stacking or boosting.

How it works

How it works

1

A set of K base learners — which may be of entirely different model families, such as a logistic regression, a k-nearest-neighbours classifier, and a decision tree — are each trained independently on the full training set or on their own tuned configuration. Diversity among the base learners is essential: models that make similar errors gain little from being combined.

2

At inference time every base learner produces a prediction for the new input. For hard voting this is simply a class label; for soft voting each learner must output a vector of class probabilities, which imposes the constraint that all chosen base learners can be calibrated to emit meaningful probability estimates.

3

In hard voting the ensemble counts how many models predicted each class and assigns the label that received the plurality of votes. Ties are broken by a fixed rule (e.g., lowest class index). This mode works even when base learners cannot produce reliable probabilities.

Formula
\hat{y} = \arg\max_c \sum_{k=1}^{K} \mathbf{1}[h_k(x)=c]
4

In soft voting the ensemble averages (or takes a weighted average using weights w_k) the class-probability vectors across all base learners and predicts the class with the highest averaged probability. Because soft voting exploits probability magnitudes rather than just winners, it typically outperforms hard voting when base learners are well calibrated.

Formula
\hat{y} = \arg\max_c \sum_{k=1}^{K} w_k \, p_k(c \mid x)
5

The ensemble returns the winning class label and, in soft-voting mode, the averaged probability vector that can be used for threshold adjustment, ranking, or calibration diagnostics downstream. No additional training step is required once base models are fitted.

Common pitfalls

  • Combining base learners that were tuned on the same data without cross-validation leads to optimistic error estimates.
  • Using soft voting with poorly calibrated probability outputs (e.g., raw SVM decision values) can hurt rather than help.
  • Selecting base learners that are too similar — all variants of the same model family — eliminates diversity and wastes computation.
  • Ignoring class imbalance: if base learners are not balanced individually, their votes will collectively amplify the majority-class bias.
  • Treating voting ensemble accuracy as final without checking whether a simpler single model achieves the same result.

Frequently asked questions

How many base learners should I include?

There is no fixed rule, but three to seven diverse classifiers is a common practical range. Adding more learners beyond a point of diminishing returns increases inference latency without meaningfully improving accuracy.

When should I prefer soft voting over hard voting?

Use soft voting when all base learners produce well-calibrated probability outputs — it typically yields higher accuracy because it uses the full probability distribution rather than just the winning class. If probabilities are poorly calibrated or unavailable, hard voting is safer.

How do I choose the weights in weighted soft voting?

A simple approach is to weight each model by its cross-validated accuracy or AUC on the training set. More principled approaches optimise weights via a held-out validation set, but risk overfitting if the validation set is small.

How is voting ensemble different from stacking?

Voting ensembles combine base model outputs with fixed rules (majority or average). Stacking trains a meta-learner on the base model outputs to learn the optimal combination — more powerful but more complex and prone to data leakage if cross-validation is not used correctly.

Does diversity always improve the ensemble?

Diversity is necessary but not sufficient. Base learners must also be individually accurate; an ensemble of weak, diverse classifiers can still perform poorly. The goal is diverse errors among accurate models.

Related methods

Sources

  1. Kuncheva, L. I. (2004). Combining Pattern Classifiers: Methods and Algorithms. Wiley-Interscience.

    ISBN: 978-0-471-21078-8
    Copied.
  2. Dietterich, T. G. (2000). Ensemble Methods in Machine Learning. In J. Kittler & F. Roli (Eds.), Multiple Classifier Systems (MCS 2000), Lecture Notes in Computer Science, vol 1857, pp. 1–15. Springer.

    Open DOI reference
    Copied.