Semi-supervised Logistic Regression
- Introduced by
- Nigam, K.; McCallum, A. et al. (EM variant); Yarowsky, D. (self-training)
- Year
- 1995–2000
- Type
- Semi-supervised classifier
- Family
- Machine learning
What does it do?
Semi-supervised logistic regression extends the standard logistic classifier by incorporating unlabeled data during training.
Using self-training, expectation-maximization, or label-propagation wrappers, it iteratively assigns soft labels to unlabeled examples and refines model parameters, improving generalization when labeled data are scarce relative to the full dataset.
The intuition behind the method
Imagine training a disease classifier with only 50 confirmed diagnoses but 5,000 patient records where the outcome is unknown. Standard logistic regression ignores those unlabeled records entirely. Semi-supervised logistic regression uses the model's own probabilistic predictions to assign tentative labels to the unlabeled cases, retrains on the enlarged pseudo-labeled set, and repeats. Each iteration nudges the decision boundary toward regions of high data density, effectively letting the unlabeled data guide where the boundary belongs without requiring expensive expert annotation.
History
Semi-supervised learning for probabilistic classifiers was systematized in the late 1990s. Yarowsky (1995) popularized self-training for NLP tasks, and Nigam et al. (2000) formalized the EM approach for text classification with naive Bayes and logistic-like models. Chapelle, Scholkopf, and Zien's 2006 MIT Press volume consolidated the theoretical foundations. Semi-supervised logistic regression is now a standard technique in settings where annotation budgets are limited, implemented in scikit-learn's SelfTrainingClassifier wrapper and dedicated SSL libraries.
When to use it
Choose semi-supervised logistic regression when labeled examples are few — typically when the labeled fraction is under 20% of the total available data — but a large pool of unlabeled data from the same distribution exists and annotation is expensive. It is well-suited to text classification, clinical outcome prediction, and survey research where outcomes are partially observed. Do not use it when the unlabeled data come from a different distribution than the labeled set (covariate shift), when labels are missing not at random and missingness is informative, or when the labeled set is already large enough that a standard logistic regression achieves good performance — in those cases, the pseudo-label noise adds more harm than benefit.
Limitations and considerations
- If initial labeled data are too few or unrepresentative, early pseudo-labels are noisy and errors compound across iterations.
- Assumes that the unlabeled data are drawn from the same distribution as the labeled data — violated in many real-world collection scenarios.
- Convergence to a local optimum is not guaranteed to improve over the supervised baseline.
- Confidence thresholds in self-training require tuning and interact with class imbalance in non-obvious ways.
- Interpreting coefficients becomes harder because the effective training set includes algorithmically generated labels.
Source conditions do not replace independent scientific review or validation by an analysis engine.
Strengths
- Exploits large pools of unlabeled data to improve a logistic classifier without additional labeling cost.
- Preserves the interpretability of logistic regression coefficients and odds ratios.
- Self-training wrapper is model-agnostic and easy to add to any existing logistic regression pipeline.
- EM variant provides a principled probabilistic framework with a formal convergence guarantee.
- Regularization (L1/L2) can be combined straightforwardly to control overfitting on pseudo-labeled data.
Applications
- Clinical outcome prediction where only a subset of patients have confirmed diagnoses.
- Text classification with large corpora and small expert-labeled training sets.
- Survey research where respondents answer only a subset of outcome questions.
- Sentiment analysis and opinion mining with partially annotated social-media data.
- Fraud detection where confirmed fraud cases are rare but transaction volumes are large.
- Medical imaging studies where radiologist labels are available for a subset of scans.
How it works
How it works
A standard logistic regression model is fit on the available labeled examples. This gives initial coefficient estimates and class-probability estimates for every observation. Even with few labels, this warm start produces a reasonable starting decision boundary that the subsequent iterations will refine.
The trained model scores every unlabeled example, producing a posterior probability P(y=k | x) for each class. In self-training, examples whose maximum probability exceeds a confidence threshold are assigned a hard pseudo-label. In EM-based approaches, soft fractional memberships are assigned rather than a hard class, keeping uncertainty explicit throughout.
Formula
\hat{y}_u = \arg\max_k \, p(y=k \mid x_u; \theta)The logistic regression is refitted on the combined set of truly labeled examples and pseudo-labeled (or soft-labeled) unlabeled examples. Regularization — L1 or L2 — is applied to prevent the expanded dataset from driving the coefficients toward the noisy pseudo-label signal too aggressively.
Steps of scoring unlabeled data and retraining are repeated until convergence — typically when the change in log-likelihood or coefficient values between iterations drops below a small tolerance. The EM variant alternates formally between an E-step (computing expected labels) and an M-step (maximizing the complete-data log-likelihood).
Once converged, the final model is evaluated on a held-out labeled test set using standard metrics such as accuracy, AUC, and F1. Cross-validation must be conducted only on the labeled partition to avoid label leakage from the unlabeled pool into the validation estimate.
Common pitfalls
- Including unlabeled examples in the validation or test set, which inflates performance estimates by leaking distributional information.
- Setting a low confidence threshold that floods the training set with uncertain pseudo-labels, compounding early errors.
- Ignoring class imbalance when assigning pseudo-labels, causing the majority class to dominate successive iterations.
- Treating pseudo-labeled data as equivalent to truly labeled data in downstream uncertainty quantification or significance testing.
- Skipping regularization on the expanded training set, allowing pseudo-label noise to drive overfitting.
Frequently asked questions
How is semi-supervised logistic regression different from standard logistic regression?
Standard logistic regression uses only labeled examples. The semi-supervised variant iteratively assigns tentative labels to unlabeled data and includes them in retraining, so the decision boundary is shaped by both confirmed labels and the structure of the larger unlabeled dataset.
What confidence threshold should I use for self-training?
Common starting points are 0.85–0.95 for binary classification. The right threshold depends on class balance and label quality. Tune it on the labeled validation set and monitor the fraction of unlabeled examples added per iteration to avoid a flood of uncertain pseudo-labels.
Can I still interpret the coefficients as odds ratios?
Yes, but with caution. The estimated coefficients retain the log-odds interpretation, but their confidence intervals are underestimated because pseudo-labeled data are treated as real observations. Inference about significance should rely only on the truly labeled portion.
What if my unlabeled data come from a different time period or source?
Distribution shift between labeled and unlabeled sets is one of the main failure modes. If the unlabeled data come from a different population or period, pseudo-labels may reinforce systematic errors. Check for covariate shift before proceeding.
Is there a ready-made implementation?
Yes. scikit-learn's SelfTrainingClassifier wraps any probabilistic classifier, including LogisticRegression, for self-training. The EM variant requires manual implementation or dedicated SSL libraries such as SKSSL or LASSL.
Related methods
Sources
Nigam, K., McCallum, A., Thrun, S., & Mitchell, T. (2000). Text classification from labeled and unlabeled documents using EM. Machine Learning, 39, 103–134.
Open DOI referenceCopied.Chapelle, O., Scholkopf, B., & Zien, A. (Eds.) (2006). Semi-supervised Learning. MIT Press.
ISBN: 978-0-262-03358-9Copied.