Regularized Linear Regression
- Introduced by
- Hoerl & Kennard (Ridge, 1970); Tibshirani (Lasso, 1996); Zou & Hastie (Elastic Net, 2005)
- Year
- 1970–2005
- Type
- Penalized linear model
- Family
- Machine learning
What does it do?
Regularized linear regression adds a penalty term to the ordinary least-squares objective, shrinking or zeroing out coefficients to reduce overfitting and handle multicollinearity.
The three main variants — Ridge (L2 penalty), Lasso (L1 penalty), and Elastic Net (combined L1+L2) — make linear regression usable even when features outnumber observations or predictors are highly correlated.
The intuition behind the method
Ordinary least squares gives each predictor exactly the coefficient that minimizes in-sample error, which can produce wildly large, unstable coefficients when predictors are correlated or when the dataset has many features. Regularization pulls those coefficients toward zero — firmly but fairly — so the model loses a little training accuracy in exchange for much better generalization. Ridge shrinks all coefficients proportionally; Lasso goes further and can set some to exactly zero, effectively selecting features; Elastic Net blends both strategies and works when correlated predictors need to be kept together.
History
Ridge regression was introduced by Hoerl and Kennard in 1970 as a remedy for multicollinearity in ordinary least squares. Tibshirani's 1996 Lasso paper added L1 regularization and the ability to shrink coefficients to exactly zero, making automatic feature selection possible. Zou and Hastie extended these ideas in 2005 with Elastic Net, combining L1 and L2 penalties to handle groups of correlated predictors. These three methods form the canonical regularization toolkit for linear models and are staples of the Elements of Statistical Learning and An Introduction to Statistical Learning.
When to use it
Use regularized linear regression when predicting a continuous outcome from many numeric features and at least some multicollinearity is suspected, or when the number of features is large relative to the sample size. Ridge is the default choice when all features are plausibly relevant; Lasso or Elastic Net when automated feature selection is needed. Prefer ordinary (unpenalized) linear regression only when features are few, uncorrelated, and the sample is large. Do not use any linear variant when the outcome is clearly nonlinear or when it is binary or categorical — use logistic regression or tree-based methods instead.
Limitations and considerations
- Assumes a linear relationship between features and the continuous target; misses nonlinear patterns unless features are engineered manually.
- Feature standardization is mandatory before fitting; forgetting this step distorts coefficient estimates.
- Lasso performs poorly when many truly relevant correlated predictors exist — it arbitrarily selects one and discards others; Elastic Net is the safer choice in that setting.
- Ridge retains all features and does not produce a sparse model, which can complicate interpretation when the feature space is very high-dimensional.
Source conditions do not replace independent scientific review or validation by an analysis engine.
Strengths
- Produces fully interpretable linear coefficients with explicit effect sizes and directions.
- Handles multicollinearity gracefully — Ridge redistributes coefficients among correlated predictors; Lasso selects one from a group.
- Lasso and Elastic Net perform built-in feature selection by setting irrelevant coefficients to exactly zero.
- Computationally efficient even on large datasets; scales to millions of observations with coordinate descent.
- Cross-validated lambda selection is well-supported in all major statistical software (scikit-learn, glmnet, statsmodels).
Applications
- Genomics and bioinformatics: predicting continuous phenotypes from thousands of genetic markers where p >> n.
- Economics and social science: estimating effects of many policy variables on continuous outcomes while controlling for multicollinearity.
- Engineering: predicting physical quantities from sensor arrays with correlated measurements.
- Feature selection pipeline: Lasso or Elastic Net used upstream to identify a small predictor set before fitting a more complex model.
- Finance: forecasting returns or risk measures from many macroeconomic and market factors.
- Healthcare: predicting continuous clinical outcomes from large panels of lab values or imaging-derived features.
How it works
How it works
Before fitting, all predictors are standardized (mean zero, unit variance). This is mandatory because the penalty applies equally to every coefficient — without standardization, features on larger scales would be penalized more heavily, distorting the coefficient estimates.
The practitioner selects a penalty type: Ridge uses the sum of squared coefficients (L2 norm), which shrinks all coefficients but never sets any to zero. Lasso uses the sum of absolute values (L1 norm), which can shrink some coefficients to exactly zero, performing automatic feature selection. Elastic Net linearly combines L1 and L2, controlled by a mixing parameter alpha, and is particularly useful when several correlated predictors are relevant.
Formula
\text{Ridge: } \lambda\sum_{j}\beta_j^2 \quad \text{Lasso: } \lambda\sum_{j}|\beta_j| \quad \text{Elastic Net: } \lambda_1\sum|\beta_j|+\lambda_2\sum\beta_j^2The model minimizes the penalized objective — residual sum of squares plus the chosen penalty scaled by the regularization strength lambda. For Ridge this has a closed-form solution; Lasso and Elastic Net require coordinate-descent or subgradient optimization because the L1 penalty is not differentiable at zero.
Formula
\hat{\boldsymbol{\beta}} = \arg\min_{\boldsymbol{\beta}} \left[ \|\mathbf{y} - \mathbf{X}\boldsymbol{\beta}\|^2 + \text{penalty}(\boldsymbol{\beta}) \right]The regularization strength lambda is a hyperparameter that must be tuned via cross-validation, typically over a logarithmic grid. A lambda of zero recovers ordinary least squares; very large lambda forces all coefficients toward zero. The optimal lambda balances bias and variance for the specific dataset.
Once lambda is fixed and coefficients are estimated, prediction for new observations is identical to ordinary linear regression: the dot product of features and estimated coefficients. The model remains fully interpretable in terms of linear effects.
Formula
\hat{y} = \mathbf{x}^\top \hat{\boldsymbol{\beta}}Common pitfalls
- Fitting without standardizing predictors, so features on larger scales are penalized more and coefficients are not comparable.
- Using the default lambda instead of selecting it via cross-validation — a wrong lambda either under-regularizes (near ordinary least squares) or over-regularizes (coefficients collapse to zero).
- Applying regularized linear regression to a binary or count outcome when logistic regression or Poisson regression should be used instead.
- Reporting Lasso variable-selection results as if they were confirmatory — Lasso selection is sensitive to the dataset and lambda, so treat selected variables as exploratory.
- Ignoring the assumption of linearity and reporting a poor-fitting regularized model without checking residuals.
Frequently asked questions
When should I choose Ridge over Lasso?
Choose Ridge when you believe most predictors contribute to the outcome and want stable coefficient estimates without zeroing any out — for example when predictors are highly correlated and you want to keep all of them. Choose Lasso or Elastic Net when you expect many features to be irrelevant and want automatic feature selection.
Do I still get p-values and confidence intervals?
Classical p-values and standard errors are not available directly from penalized regression because the coefficient estimates are biased by design. Post-selection inference methods (e.g., the desparsified Lasso) exist but are rarely used in practice; instead, bootstrap confidence intervals or cross-validated stability measures are reported.
How do I choose the regularization strength lambda?
Select lambda by k-fold cross-validation (typically k = 5 or 10) over a logarithmic grid. Libraries such as scikit-learn's RidgeCV and LassoCV or R's glmnet do this automatically. Report the lambda chosen and note whether it was selected by minimum cross-validation error or by the one-standard-error rule.
Does regularized linear regression require normally distributed features?
No. Like ordinary least squares, the assumption is on the residuals (approximately normal, homoscedastic), not on the predictors. Highly skewed or outlier-heavy predictors can still distort estimates, so log-transforming skewed inputs is a common preprocessing step.
What is Elastic Net and when is it preferred?
Elastic Net combines the L1 and L2 penalties, controlled by a mixing parameter alpha (alpha = 1 is pure Lasso; alpha = 0 is pure Ridge). It is preferred when groups of correlated predictors are all relevant — Lasso would arbitrarily select one from the group and discard the rest, while Elastic Net tends to keep the group together.
Related methods
Sources
Tibshirani, R. (1996). Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society: Series B, 58(1), 267–288.
Open DOI referenceCopied.Hastie, T., Tibshirani, R. & Friedman, J. (2009). The Elements of Statistical Learning (2nd ed., Ch. 3). Springer.
ISBN: 978-0-387-84858-7Copied.