Skip to content
Back to search results

Regularized Logistic Regression

Introduced by
Tibshirani, R. (lasso); Hoerl & Kennard (ridge); Zou & Hastie (elastic net)
Year
1996–2005
Type
Penalized classification model
Family
Machine learning
Font size

What does it do?

Regularized logistic regression extends standard logistic regression by adding an L1 (lasso), L2 (ridge), or elastic net penalty to the log-likelihood, shrinking coefficients toward zero and preventing overfitting.

It is the default choice for binary or multinomial classification when you want interpretable, sparse, or stable coefficient estimates in high-dimensional or collinear feature spaces.

The intuition behind the method

Standard logistic regression maximizes the probability of observed class labels. In high-dimensional settings — many features, few observations, or correlated predictors — the estimated coefficients can blow up or fluctuate wildly, causing poor generalization. Regularization constrains the coefficient magnitudes by adding a penalty that grows with coefficient size, trading a small amount of training-set fit for much better generalization. L1 penalty drives some coefficients exactly to zero (automatic feature selection), L2 penalty shrinks all coefficients smoothly without zeroing them, and elastic net blends both behaviors.

History

Ridge regression was formalized by Hoerl and Kennard in 1970 and applied to generalized linear models including logistic regression. Tibshirani's 1996 lasso paper introduced L1 regularization with sparse solutions, transforming high-dimensional modeling. Zou and Hastie (2005) combined both penalties into elastic net, addressing lasso's limitation with grouped correlated features. Today, regularized logistic regression is a foundational algorithm in scikit-learn, glmnet (R), and LIBLINEAR, and is routinely used as the benchmark classifier in biomedical and NLP applications.

When to use it

Use regularized logistic regression for binary or multinomial classification when you need interpretable probabilistic outputs, when the number of features is large relative to the sample size, or when predictors are correlated. It is the preferred starting model in clinical prediction, text classification with bag-of-words features, and high-dimensional biomarker studies. Choose L1 when automatic feature selection is needed, L2 for stable coefficient estimation, and elastic net when features arrive in correlated groups. Avoid it when the outcome–feature relationship is highly nonlinear or when there are strong interaction effects that the linear log-odds formulation cannot capture; tree ensembles or kernel methods may then be more appropriate.

Limitations and considerations

  • Assumes a linear relationship between features and log-odds; cannot capture nonlinear or interaction effects without manual feature engineering.
  • Requires the outcome to be categorical; continuous outcomes need linear (not logistic) regression.
  • Performance degrades when relevant features strongly interact, since the model is additive in log-odds.
  • Optimal lambda must be selected by cross-validation, adding computation and a tuning step.
  • Severely class-imbalanced data can bias predicted probabilities toward the majority class without reweighting.

Source conditions do not replace independent scientific review or validation by an analysis engine.

Strengths

  • Produces calibrated class probabilities and interpretable, signed coefficients unlike most black-box classifiers.
  • L1 penalty performs automatic feature selection, yielding sparse models that are easy to report.
  • Handles high-dimensional feature spaces (p >> n) robustly through coefficient shrinkage.
  • Elastic net penalty gracefully handles correlated feature groups, spreading weight across related predictors.
  • Very fast to fit even on large datasets using coordinate descent solvers; scales to millions of observations.

Applications

  • Clinical risk scores and diagnostic classifiers from high-dimensional biomarker panels.
  • Text classification with bag-of-words or TF-IDF features, where L1 produces interpretable sparse word weights.
  • Genomics and proteomics where the number of features vastly exceeds the sample size.
  • Credit scoring and fraud detection requiring probability outputs for threshold-based decisions.
  • Baseline classification benchmark before evaluating more complex ensemble or deep learning models.

How it works

How it works

1

A linear combination of features predicts the log-odds of the positive class. The sigmoid function converts log-odds to a probability between 0 and 1, making output directly interpretable as class probability.

Formula
\log\frac{p}{1-p} = \mathbf{x}^\top\boldsymbol{\beta}
2

The optimization objective is the negative log-likelihood (cross-entropy) plus a penalty term. The penalty scales with the coefficient magnitudes, controlled by the hyperparameter lambda (strength) and alpha (L1/L2 mix). A larger lambda imposes heavier shrinkage; alpha=1 gives pure lasso, alpha=0 gives pure ridge.

Formula
\mathcal{L}(\boldsymbol{\beta}) = -\ell(\boldsymbol{\beta}) + \lambda\left[\alpha\|\boldsymbol{\beta}\|_1 + \tfrac{1-\alpha}{2}\|\boldsymbol{\beta}\|_2^2\right]
3

The choice of penalty determines behavior: L1 (lasso) produces sparse solutions by zeroing out irrelevant features and is preferred when feature selection is desired. L2 (ridge) is preferred when all features are expected to contribute and collinearity is high. Elastic net (alpha between 0 and 1) handles grouped correlated features better than lasso alone.

4

The penalized loss is minimized using iterative algorithms such as coordinate descent (scikit-learn's liblinear/saga solvers). The optimal lambda is determined by cross-validation, typically over a logarithmic grid. Standardizing features before fitting is essential so the penalty is applied consistently across all predictors.

5

Predicted probabilities are converted to class labels using a decision threshold, typically 0.5 by default. In imbalanced settings the threshold should be tuned to balance precision and recall according to the application's cost structure.

Formula
\hat{y} = \mathbf{1}[\hat{p} \ge t]

Common pitfalls

  • Fitting without standardizing features: the penalty treats all coefficients equally, so features on different scales receive unequal shrinkage, distorting the solution.
  • Using the default threshold of 0.5 without checking calibration and class balance, which silently degrades recall on the minority class.
  • Reporting only accuracy instead of AUC, F1, and calibration metrics, masking poor minority-class performance.
  • Skipping cross-validation for lambda selection and instead evaluating on training data, leading to overfitted models.
  • Interpreting lasso-selected coefficients as causal without accounting for omitted-variable bias and the instability of variable selection near the decision boundary.

Frequently asked questions

How do I choose between L1, L2, and elastic net?

Use L1 when you believe only a subset of features are relevant and you want automatic selection. Use L2 when all features likely contribute or predictors are highly correlated, since ridge distributes weight across correlated groups. Use elastic net when you need sparsity but also have blocks of correlated features; tune alpha alongside lambda via nested cross-validation.

Must I standardize features before fitting?

Yes, always. The penalty term treats all coefficients equally, so a feature with large numeric values will receive proportionally less shrinkage unless standardized. Center and scale each feature to zero mean and unit variance before fitting and apply the same transformation to test data.

How is lambda chosen?

Lambda is a hyperparameter selected by cross-validation, typically k-fold (k=5 or 10) over a logarithmic grid. In R's glmnet, cv.glmnet automates this. In scikit-learn, LogisticRegressionCV or GridSearchCV with cross-validation on the inverse regularization strength C (C=1/lambda) should be used.

Does regularized logistic regression give calibrated probabilities?

Generally yes, better than tree ensembles, because logistic regression directly models class probabilities via the sigmoid. However, extreme regularization can compress predicted probabilities toward 0.5. Platt scaling or isotonic regression can recalibrate if needed.

How do I handle class imbalance?

Set class_weight='balanced' (scikit-learn) so minority class observations receive higher loss weight, or manually resample the training set. Also tune the decision threshold using the precision-recall curve rather than fixing it at 0.5.

Related methods

Sources

  1. Tibshirani, R. (1996). Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society: Series B, 58(1), 267–288.

    Open DOI reference
    Copied.
  2. Hastie, T., Tibshirani, R., & Friedman, J. (2009). The Elements of Statistical Learning (2nd ed., Ch. 4, 18). Springer.

    ISBN: 978-0-387-84857-0
    Copied.