Active Learning Logistic Regression
- Introduced by
- Lewis, D. D. & Gale, W. A.; Settles, B. (survey)
- Year
- 1994–2010
- Type
- Active learning framework with logistic regression base learner
- Family
- Machine learning
What does it do?
Active Learning with Logistic Regression is an iterative label-efficient framework in which a logistic regression model selects the unlabeled examples it is most uncertain about, an oracle (human annotator) labels them, and the model is retrained — repeating until a labeling budget or accuracy target is met.
It dramatically reduces annotation cost compared to random labeling.
The intuition behind the method
Labeling data is expensive. Instead of annotating a random sample, imagine the logistic model pointing to only the examples where its predicted probability is closest to 0.5 — meaning it genuinely cannot decide. An expert labels only those borderline cases, and the model is retrained. Because each labeled example carries maximum information, the model reaches good accuracy with far fewer labels than passive learning. Logistic regression is favored here because it provides calibrated probabilities that directly reflect uncertainty.
History
Active learning as a framework was formalized in the early 1990s. Lewis and Gale (1994) introduced uncertainty sampling for text classification using a Naive Bayes learner, and the approach was quickly adapted to logistic regression. Settles' 2010 survey became the canonical reference, cataloguing query strategies including least confidence, margin sampling, and entropy. Logistic regression remains the most commonly cited base learner in pool-based active learning benchmarks because its probabilities are calibrated and its retraining cost is low.
When to use it
Use active learning with logistic regression when annotation is costly (medical records, legal documents, specialist surveys) and unlabeled data is abundant, but the total labeling budget is limited. It excels for binary or multi-class text classification, sentiment analysis, and clinical coding when features are reasonably linearly separable. Avoid it when (a) the decision boundary is highly nonlinear and logistic regression cannot model it even with feature engineering, (b) the oracle is unreliable or expensive to query repeatedly, (c) the labeled pool is already large enough for full supervision, or (d) the unlabeled pool is too small to benefit from selective querying.
Limitations and considerations
- Performance degrades when the true decision boundary is nonlinear; in such cases a kernel SVM or neural active learner is preferable.
- Uncertainty sampling can create a biased labeled set that over-represents borderline instances and underrepresents the bulk of each class.
- Requires repeated interaction with the oracle, which adds logistical overhead compared to a single labeling batch.
- Cold-start: the initial seed set strongly influences early queries, so a poorly chosen seed can slow convergence.
Source conditions do not replace independent scientific review or validation by an analysis engine.
Strengths
- Dramatically reduces the number of labels needed to reach a target accuracy compared to random sampling.
- Logistic regression provides natively calibrated probabilities, making uncertainty estimates reliable without extra calibration.
- Transparent and auditable: the linear model and the query criterion are both inspectable.
- Computationally cheap to retrain after each query, enabling real-time or near-real-time annotation workflows.
- Transfers seamlessly to text classification with TF-IDF or bag-of-words feature representations.
Applications
- Text classification and sentiment analysis with limited labeled corpora.
- Medical record coding and clinical NLP where expert annotation is expensive.
- Active annotation pipelines for social science surveys and content moderation.
- Information retrieval relevance feedback where users label only uncertain documents.
- Biomedical entity recognition bootstrapping with minimal seed annotations.
How it works
How it works
A small seed set of labeled examples is assembled — often 10 to a few hundred instances depending on the domain. A logistic regression model is trained on this seed set to establish an initial decision boundary. The remaining unlabeled instances form the pool from which queries will be drawn.
Logistic regression fits a linear decision boundary by maximizing the log-likelihood of the labeled set. The output is a probability P(y=1|x) = sigmoid(w^T x) for each instance. Because logistic regression produces calibrated probabilities, its confidence scores are directly interpretable as uncertainty estimates — a key reason it pairs naturally with active learning.
Formula
P(y=1 \mid x) = \sigma(\mathbf{w}^\top x)Each unlabeled instance in the pool is scored by an uncertainty criterion. The most common is least-confidence sampling: the model selects the instance whose maximum class probability is lowest (i.e., the model is least sure). For binary problems this reduces to selecting points nearest the decision boundary (predicted probability closest to 0.5).
Formula
x^* = \arg\min_{x \in U} \max_y P(y \mid x, \hat{\theta})The most uncertain instance (or a batch of the top-k most uncertain instances) is presented to a human annotator or oracle for labeling. The oracle provides the true class label, which is added to the labeled training set. A fixed labeling budget determines how many queries the loop is allowed to make.
The logistic regression model is retrained on the expanded labeled set. Performance is evaluated on a held-out validation set. If the budget is not exhausted and the accuracy target has not been met, the loop returns to the uncertainty scoring step. The process terminates when the budget runs out or performance plateaus.
Common pitfalls
- Evaluating performance only on the actively selected labeled set rather than a separate i.i.d. held-out set, which inflates accuracy estimates.
- Ignoring class imbalance in the seed set — an imbalanced seed causes the model to query almost exclusively from the minority class boundary.
- Applying uncertainty sampling to a pool with distributional shift between labeled and unlabeled data without density weighting or representativeness correction.
- Treating the final model as a standard logistic regression without accounting for the biased sampling when reporting confidence intervals or hypothesis tests.
Frequently asked questions
Why use logistic regression instead of a more powerful model?
Logistic regression provides well-calibrated probabilities, retrains quickly after each query, and is interpretable — all critical in active learning loops where the model is updated many times and uncertainty scores must be trustworthy. More complex models can be used but increase retraining cost and may require probability calibration (e.g., Platt scaling for SVMs).
How many seed labels do I need to start?
There is no fixed rule, but 10–50 labeled examples per class are common starting points in the literature. Too few seeds produce an unstable initial model whose uncertainty estimates are unreliable; too many defeat the purpose of active learning.
What is the difference between uncertainty sampling and query by committee?
Uncertainty sampling uses a single model's prediction confidence to select queries. Query by committee trains a committee of models and selects points where they disagree most. For logistic regression, uncertainty sampling (least confidence or entropy) is more common because it needs only one model.
Can I use batch queries instead of one at a time?
Yes. Batch-mode active learning selects the top-k most uncertain instances per round to reduce oracle interaction frequency. A common heuristic is to use a batch size of 10–50, retraining after each batch rather than after every single query.
How do I evaluate the active learning curve?
Plot classification accuracy (or F1, AUC) on a fixed held-out test set as a function of the number of labels queried. Compare this curve to a random-sampling baseline. The area under the learning curve relative to random sampling quantifies the labeling efficiency gain.
Related methods
Sources
Settles, B. (2010). Active Learning Literature Survey. Computer Sciences Technical Report 1648, University of Wisconsin–Madison.
Open publication referenceCopied.Lewis, D. D., & Gale, W. A. (1994). A sequential algorithm for training text classifiers. Proceedings of the 17th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, 3–12.
Open DOI referenceCopied.