Ridge Regression
- Introduced by
- Hoerl, A.E. & Kennard, R.W.
- Year
- 1970
- Type
- L2-regularized linear regression
What does it do?
Ridge Regression is an L2-regularized linear regression method, introduced by Arthur Hoerl and Robert Kennard in 1970, that reduces multicollinearity by adding a penalty on the size of the coefficients.
It shrinks coefficients toward zero without setting any of them exactly to zero, producing more stable estimates when predictors are highly correlated.
The intuition behind the method
When predictors are strongly correlated, ordinary least squares coefficients become unstable and can swing wildly from sample to sample. Ridge Regression deliberately accepts a small amount of bias in exchange for far lower variance: by adding a penalty on the squared size of the coefficients, it tames these wild swings and keeps all variables in the model, just with smaller, steadier weights.
History
Ridge Regression was introduced by Arthur E. Hoerl and Robert W. Kennard in 1970 as a way to obtain biased but more stable estimates for nonorthogonal (collinear) problems, where ordinary least squares breaks down. The idea of adding an L2 penalty became a foundational tool of regularized regression and a direct precursor to later methods such as LASSO and Elastic Net, which combine or contrast its shrinkage behaviour with explicit variable selection.
When to use it
Use Ridge Regression for prediction with continuous, binary, or count predictors when multicollinearity is present — for example when a variance inflation factor exceeds 10 — and you want to keep all variables rather than select among them. It assumes a roughly linear relationship and at least about 30 observations. Features must be standardized first. When samples are small and collinearity is mild, plain linear regression is more interpretable; when variable selection is the goal, LASSO or Elastic Net is preferable.
Limitations and considerations
- Performs no variable selection; it cannot zero out irrelevant predictors the way LASSO can.
- Requires features to be standardized, otherwise the penalty unfairly favours large-scale variables.
- Assumes a linear relationship between predictors and outcome.
- On small samples (n below about 30) tuning alpha by cross-validation is unreliable and plain OLS may suffice.
- When collinearity is not actually a problem, the shrinkage is unnecessary and reduces interpretability.
Source conditions do not replace independent scientific review or validation by an analysis engine.
Strengths
- Reduces multicollinearity and stabilizes coefficient estimates by adding an L2 penalty.
- Keeps every predictor in the model — coefficients shrink but none are dropped to zero.
- Lowers variance at the cost of a small, controlled increase in bias, often improving prediction.
- Has a single, easily tuned hyperparameter (alpha) chosen by cross-validation.
- Works with continuous, binary, and count predictors without requiring normally distributed data.
Applications
- Prediction with many correlated predictors where multicollinearity inflates OLS variance.
- Stabilizing regression coefficients when variance inflation factors are high (VIF above 10).
- Modelling when all predictors should be retained rather than selected away.
- A regularized baseline before comparing against LASSO or Elastic Net.
How it works
How it works
Ordinary least squares fits coefficients by minimizing the squared error alone. When two or more predictors are nearly redundant (multicollinearity), this leaves the coefficients poorly determined and highly sensitive to noise. Ridge addresses this by discouraging large coefficients, trading a little bias for much greater stability.
Ridge minimizes the usual sum of squared errors plus a penalty equal to a tuning constant alpha times the sum of the squared coefficients. This L2 penalty shrinks all coefficients toward zero in proportion to their size, but unlike LASSO it never forces any coefficient exactly to zero, so every predictor is retained. Standardizing the features beforehand is required so the penalty treats each predictor fairly.
Formula
\hat{\beta}^{\text{ridge}} = \arg\min_{\beta} \; \|y - X\beta\|^2 + \alpha \|\beta\|^2The strength of the penalty is controlled by alpha: larger values shrink the coefficients more. The right value is chosen by cross-validation, typically by scanning a range of alpha and picking the one that minimizes the cross-validated mean squared error.
Common pitfalls
- Fitting Ridge on raw, unstandardized features so the penalty distorts the coefficients.
- Picking alpha by hand instead of selecting it via cross-validation.
- Applying Ridge when no real multicollinearity exists, shrinking coefficients for no benefit.
- Expecting variable selection — Ridge keeps all variables, so reach for LASSO or Elastic Net if you need sparsity.
- Skipping residual diagnostics (normality, homoscedasticity) after fitting.
Frequently asked questions
How is Ridge different from LASSO?
Both add a penalty to shrink coefficients, but Ridge uses an L2 penalty that shrinks coefficients toward zero without ever reaching it, so all predictors stay in the model. LASSO uses an L1 penalty that can set coefficients exactly to zero, performing variable selection. Choose Ridge to stabilize correlated predictors and LASSO when you need a sparse model.
Do I need to standardize my features?
Yes. Standardizing features is required, because the L2 penalty acts on the size of the coefficients; without standardization the penalty would unfairly shrink variables measured on larger scales.
How do I choose the penalty strength alpha?
Choose alpha by cross-validation: scan a range of values and pick the one that minimizes the cross-validated mean squared error. Plotting the CV MSE curve against alpha shows where the optimum lies.
When should I not use Ridge?
If there is no real multicollinearity, Ridge adds unnecessary shrinkage and a plain linear regression is more interpretable. With very small samples (n below about 30), tuning alpha is unreliable, and if you need to drop irrelevant variables you should use LASSO or Elastic Net instead.
Related methods
Sources
Hoerl, A.E. & Kennard, R.W. (1970). Ridge Regression: Biased Estimation for Nonorthogonal Problems. Technometrics, 12(1), 55–67.
Open DOI referenceCopied.