Skip to content
Back to search results

Active Learning Voting Ensemble

Introduced by
Seung, H. S., Opper, M., & Sompolinsky, H.
Year
1992
Type
Active learning with ensemble voting
Family
Machine learning
Font size

What does it do?

Active Learning Voting Ensemble — formally known as Query by Committee — is an active learning strategy that trains a committee of diverse models and selects the unlabeled examples where the committee members disagree most for human annotation.

By focusing labeling effort on the most informative points, it achieves high accuracy with far fewer labeled examples than passive learning requires.

The intuition behind the method

Imagine asking five experts to classify an unlabeled sample. If four agree and one dissents, the answer is probably clear — no need to pay for a label. But if they split three-to-two, that sample sits right on the decision boundary where labeling it will teach all the models the most. Active learning voting ensemble makes this intuition precise: build a committee of models, let them vote, and only pay for labels when the vote is close. The result is a highly efficient learning loop that stretches a small annotation budget as far as possible.

History

The Query by Committee (QBC) framework was introduced by Seung, Opper, and Sompolinsky at COLT 1992 as a principled approach to active learning with version-space committees. It was one of the first theoretical accounts of using ensemble disagreement to guide data collection, and it remains the canonical reference for committee-based active learning. The broader active learning literature was consolidated in Burr Settles's widely cited 2009 technical survey, which systematized pool-based, stream-based, and membership-query scenarios and established QBC as a cornerstone strategy alongside uncertainty sampling.

When to use it

Use active learning voting ensemble when labeling is expensive or time-consuming — clinical annotation, legal document tagging, specialized image review — and a large pool of unlabeled data is available alongside a small labeled seed set. It works well for classification tasks where committee disagreement is a meaningful signal, and when at least 3–5 diverse models can be fielded. Avoid it when labels are cheap and plentiful (passive learning on the full dataset will then be equally or more efficient), when the unlabeled pool is very small, when real-time online annotation is infeasible, or when a single interpretable model is required rather than an ensemble.

Limitations and considerations

  • Requires an annotation loop, which adds infrastructure complexity: a pool manager, oracle interface, and retraining pipeline.
  • Performance depends heavily on committee diversity; a homogeneous committee produces uninformative disagreement.
  • Pool-based active learning assumes the full unlabeled pool fits in memory; very large pools require approximations.
  • Convergence is not guaranteed if the oracle makes systematic labeling errors.

Source conditions do not replace independent scientific review or validation by an analysis engine.

Strengths

  • Dramatically reduces the number of labeled examples needed to reach a target accuracy compared with random sampling.
  • Ensemble voting provides the dual benefit of lower variance predictions at inference time.
  • Model-agnostic: any set of diverse classifiers can form the committee.
  • The disagreement score provides a built-in uncertainty signal that is easy to monitor and visualize.
  • Well-founded theoretically through the Query by Committee framework introduced in 1992.

Applications

  • Medical image annotation where expert radiologists are scarce and labeling is costly.
  • Named entity recognition and text classification with limited annotated corpora.
  • Sentiment analysis with domain-specific jargon requiring specialist review.
  • Drug discovery and molecular property prediction with expensive wet-lab validation.
  • Satellite imagery classification where only a fraction of tiles can be manually reviewed.

How it works

How it works

1

A committee of k diverse base models (the voting ensemble) is initialized on a small seed set of labeled examples. Diversity is key: the models may differ in architecture, hyperparameters, or training subset so their disagreements are informative rather than random.

2

For every unlabeled candidate x in the pool U, disagreement D(C, x) among the committee is measured — common metrics include vote entropy (how spread the votes are) or KL divergence between predictions. The sample with the highest disagreement score is selected: x* = argmax D(C, x).

Formula
x^* = \arg\max_{x \in U} D(C, x)
3

The selected unlabeled example is submitted to a human oracle (expert annotator, crowdsourcing platform, or automated labeler) for a ground-truth label. This is the only step that incurs annotation cost.

4

The newly labeled example is added to the training set and all committee members are retrained (or updated incrementally). The loop returns to the query step until the annotation budget is exhausted or a performance threshold is reached.

5

At inference time, the trained committee produces a majority vote across all k members: the predicted class is the mode of the individual predictions h_1(x) through h_k(x), combining both the active-learning efficiency gain and the ensemble's variance reduction.

Formula
\hat{y} = \text{mode}(h_1(x), h_2(x), \ldots, h_k(x))

Common pitfalls

  • Using identical or near-identical committee members — diversity is the engine of the method and without it disagreement collapses.
  • Retraining from scratch after every single query; batch-mode querying (selecting b examples per round) is far more practical.
  • Forgetting to evaluate on a fixed held-out test set throughout the loop — only measuring on the growing training set hides overfitting.
  • Treating the disagreement score as a confidence measure at inference without proper calibration.
  • Ignoring the informativeness–representativeness trade-off: highly uncertain samples near the boundary may be outliers, not typical examples.

Frequently asked questions

How many committee members should I use?

Three to ten diverse models is a common practical range. Fewer members yield noisier disagreement estimates; more members increase compute costs. Empirically, five diverse classifiers often provide a good balance between stability and efficiency.

How do I measure disagreement in the committee?

Vote entropy is the most common measure: H = -sum_c p(c) log p(c) where p(c) is the fraction of committee members predicting class c. For probabilistic models, average KL divergence from the mean prediction is an alternative. Higher scores indicate more informative samples.

Can I use this with deep learning models?

Yes. Deep ensembles or MC-dropout approximations can serve as the committee. The main challenge is retraining cost — strategies like fine-tuning a shared backbone with diverse heads reduce this considerably.

What if my annotation budget is very tight — say, fewer than 20 labels?

With very few labels, random sampling or stratified seed selection often performs comparably to QBC because all methods start from nearly the same information-poor baseline. QBC's advantage emerges more clearly once the seed set has at least 20–50 examples per class.

Does active learning voting ensemble work for regression?

Yes, with disagreement measured as the variance or standard deviation of committee predictions rather than vote entropy. The same query-retrain loop applies; the final output is the mean of committee predictions.

Related methods

Sources

  1. Seung, H. S., Opper, M., & Sompolinsky, H. (1992). Query by committee. In Proceedings of the Fifth Annual Workshop on Computational Learning Theory (COLT '92), pp. 287–294. ACM.

    Open DOI reference
    Copied.
  2. Settles, B. (2009). Active Learning Literature Survey. Computer Sciences Technical Report 1648, University of Wisconsin–Madison.

    Open publication reference
    Copied.