Logistic Regression (ML)
- Introduced by
- Cox, D. R.
- Year
- 1958
- Type
- Probabilistic linear classifier
- Family
- Machine learning
What does it do?
Logistic regression is a foundational probabilistic classifier that models the log-odds of a binary (or multinomial) outcome as a linear function of the predictors.
Introduced by D. R. Cox in 1958, it remains one of the most widely used and interpretable classification methods in both statistics and machine learning, valued for its calibrated probability outputs and clear coefficient interpretation.
The intuition behind the method
Logistic regression asks: given a set of features, what is the probability that an observation belongs to a particular class? It fits a straight line to the log-odds of the outcome, then squashes that linear score through a sigmoid (S-shaped) curve to produce a probability between 0 and 1. Every predictor's coefficient tells you directly how a one-unit change in that feature shifts the log-odds of the positive class, making the model highly transparent. A predicted probability above 0.5 is classified as the positive class, though this threshold is adjustable.
History
David Cox formalised logistic regression for binary outcomes in 1958, extending earlier work on probit models. Through the 1970s–1980s it became the standard classifier in epidemiology and social science. The maximum-likelihood framework, extended to multinomial outcomes and regularised variants, was consolidated by Tibshirani's Lasso (1996) and by Hastie, Tibshirani, and Friedman's The Elements of Statistical Learning (2001). Today it remains the canonical interpretable classifier and is the required baseline in many peer-review guidelines.
When to use it
Use logistic regression when the outcome is binary or categorical, the sample size is at least moderate (n > 10 events per predictor as a rule of thumb), and interpretable coefficients or calibrated probabilities are required. It is the default baseline classifier in social science, medicine, and epidemiology and should be tried before more complex models. Avoid when the decision boundary is strongly nonlinear, when many interaction terms would be needed, or when prediction accuracy on complex tabular data matters more than interpretability — in those cases tree-based ensembles or SVMs typically outperform it.
Limitations and considerations
- Assumes a linear relationship between features and the log-odds of the outcome; nonlinear patterns require manual feature engineering or a nonlinear model.
- Sensitive to multicollinearity among predictors: correlated features inflate coefficient standard errors and destabilise estimates.
- Requires a reasonable number of events per predictor (rule of thumb: at least 10); sparse outcome classes cause convergence problems and overfitting.
- May underperform ensemble methods (random forest, gradient boosting) on complex tabular data with many interactions.
Source conditions do not replace independent scientific review or validation by an analysis engine.
Strengths
- Produces calibrated class probabilities, not just labels, which are directly usable for ranking and decision thresholding.
- Coefficients are interpretable as log-odds ratios, meeting the standards expected in clinical, social-science, and policy research.
- Computationally fast and memory-efficient even on large datasets.
- Regularised variants (Ridge, Lasso, Elastic Net) handle high-dimensional features and perform automatic feature selection.
- Assumption-light relative to discriminant analysis: does not require multivariate normality of predictors.
- Robust and well-understood; serves as a strong, credible baseline against which complex models should be compared.
Applications
- Disease diagnosis and clinical risk scoring (e.g., predicting readmission or mortality).
- Credit risk and fraud detection in finance.
- Binary survey-response modelling in social science and epidemiology.
- Spam and sentiment classification as a fast, interpretable text baseline.
- Baseline classifier before deploying more complex ensemble models.
How it works
How it works
A weighted sum of all input features is computed. Each feature receives a coefficient (weight) that is learned from data; the intercept captures the baseline log-odds when all features are zero. This is identical in form to linear regression but the output is interpreted as a log-odds score rather than a direct numeric prediction.
Formula
z = \beta_0 + \beta_1 x_1 + \cdots + \beta_p x_p
The linear score is transformed by the sigmoid function, which maps any real number to the open interval (0, 1). This gives a well-calibrated class probability. For multi-class problems the softmax function generalises this step to produce a probability distribution over all classes.
Formula
P(y=1|\mathbf{x}) = \sigma(z) = \frac{1}{1 + e^{-z}}Model parameters are estimated by maximising the log-likelihood of the observed labels given the predicted probabilities. This is equivalent to minimising binary cross-entropy loss. Unlike ordinary least squares, there is no closed-form solution: an iterative optimisation algorithm (typically Newton–Raphson or gradient descent) is used.
Formula
\ell(\boldsymbol{\beta}) = \sum_{i=1}^{n} \left[ y_i \log \hat{p}_i + (1-y_i) \log(1-\hat{p}_i) \right]An iterative solver — such as L-BFGS, Newton–Raphson, or stochastic gradient descent — updates the coefficient vector until convergence. Regularisation terms (L1 or L2) can be added to the objective to shrink coefficients, preventing overfitting and enabling automatic feature selection (L1 / Lasso).
The fitted probability is compared to a classification threshold (default 0.5) to assign a class label. In imbalanced settings or when false-positive and false-negative costs differ, this threshold should be tuned using a precision-recall curve or ROC analysis rather than left at 0.5.
Formula
\hat{y} = \mathbf{1}[\hat{p} \geq 0.5]Common pitfalls
- Omitting a train/test split or cross-validation and evaluating on training data, which inflates apparent performance.
- Using accuracy as the sole metric with imbalanced classes instead of reporting AUC-ROC, F1, and a confusion matrix.
- Ignoring the linearity assumption and applying logistic regression to data with strongly nonlinear class boundaries without diagnostic checks.
- Failing to apply regularisation when the number of predictors is large relative to sample size, leading to overfitting.
- Not checking for multicollinearity (via VIF) before interpreting coefficient magnitudes.
Frequently asked questions
When should I prefer logistic regression over a random forest?
When you need calibrated probabilities, interpretable coefficients, or must satisfy reporting standards that require explicit odds ratios — as in clinical trials or policy evaluation. Random forest typically achieves higher predictive accuracy but provides no direct coefficient interpretation.
How many observations do I need?
A widely cited rule of thumb is at least 10 events (positive outcomes) per predictor variable. With fewer events, parameter estimates become unstable and confidence intervals unreliable. Regularisation or penalised likelihood can extend the usable range somewhat.
Should I standardise my features?
Standardising features is not required for prediction, but it is important when comparing coefficient magnitudes across predictors or when using regularisation (L1 or L2), because penalties treat all coefficients on the same scale.
What is the difference between binary and multinomial logistic regression?
Binary logistic regression models a two-class outcome using the sigmoid function. Multinomial (softmax) logistic regression generalises this to three or more unordered classes, estimating one set of coefficients per class relative to a reference class.
How do I handle class imbalance?
Set class_weight='balanced' in scikit-learn to re-weight the loss function, or oversample the minority class (SMOTE). Always evaluate with AUC-ROC and precision-recall curves rather than accuracy, and tune the classification threshold on a validation set.
Related methods
Sources
Cox, D. R. (1958). The regression analysis of binary sequences. Journal of the Royal Statistical Society, Series B, 20(2), 215–242.
Open DOI referenceCopied.James, G., Witten, D., Hastie, T. & Tibshirani, R. (2013). An Introduction to Statistical Learning (Ch. 4). Springer.
ISBN: 978-1-4614-7138-7Copied.