Lasso Regression
- Introduced by
- Tibshirani, R.
- Year
- 1996
- Type
- Regularized linear regression (L1 penalty)
What does it do?
Lasso regression, introduced by Robert Tibshirani in 1996, is a linear regression method that adds an L1 penalty to the loss so that it shrinks coefficients and performs variable selection at the same time, producing a sparse model.
By driving some coefficients exactly to zero it keeps only the predictors that matter.
The intuition behind the method
Ordinary least squares fits every predictor and can spread weight across many correlated or irrelevant variables, which inflates variance and hides which features really drive the outcome. Lasso adds a budget on the total size of the coefficients: as that budget tightens, weak predictors are pushed all the way to zero and drop out of the model. The result is a smaller, more stable set of variables that is easier to interpret.
History
Lasso was introduced by Robert Tibshirani in 1996 as a way to combine the shrinkage of penalised regression with automatic variable selection, using an L1 penalty on the coefficients. It became a cornerstone of high-dimensional statistics and motivated related methods such as ridge regression and the elastic net, which trade off or combine L1 and L2 penalties.
When to use it
A good fit when you want a sparse, interpretable linear model for prediction or explanation from many continuous, binary, or count predictors, and at least about 30 observations. It is especially useful when multicollinearity is present (for example VIF above 10) and you want the method itself to choose which variables to keep. It assumes a roughly linear relationship and requires the features to be standardised. Below about 30 observations the variable selection becomes unstable and ridge regression is safer; when the number of predictors exceeds the sample size, elastic net is preferable.
Limitations and considerations
- Assumes a linear relationship between predictors and outcome.
- Features must be standardised first or the penalty treats variables on different scales unfairly.
- On small samples (n below about 30) the selected variables are unstable and not reproducible.
- When the number of predictors exceeds the sample size it can select at most n variables, so elastic net is preferred.
Source conditions do not replace independent scientific review or validation by an analysis engine.
Strengths
- Performs variable selection and coefficient shrinkage in a single fit, yielding a sparse, interpretable model.
- Drives weak predictors exactly to zero, which simplifies the model and reduces overfitting.
- A natural response to multicollinearity (high VIF) without manually dropping variables.
- Does not require normally distributed data and handles continuous, binary, and count predictors.
- Only one tuning parameter (lambda), which is straightforward to set by cross-validation.
Applications
- Building sparse, interpretable prediction models from many candidate predictors.
- Automatic variable selection when multicollinearity (high VIF) is present.
- High-dimensional settings where only a subset of features is expected to matter.
- A regularised baseline before trying elastic net or more complex models.
How it works
How it works
Lasso minimises the usual sum of squared residuals plus a penalty equal to lambda times the sum of the absolute values of the coefficients. This L1 penalty is the defining feature: the larger lambda is, the more the coefficients are pulled toward zero.
Formula
\min_{\beta} \; \|y - X\beta\|_2^2 + \lambda \sum_j |\beta_j|Because the L1 penalty has sharp corners, it does more than just shrink coefficients toward zero — it sets many of them exactly to zero. Variables with a zero coefficient are effectively removed, so fitting and variable selection happen in a single step and the final model is sparse.
The strength of the penalty lambda controls the trade-off between fit and sparsity, so it is chosen by cross-validation: a range of lambda values is tried and the one minimising the cross-validated mean squared error is kept. Features must be standardised first so the single penalty applies fairly across predictors on different scales.
Common pitfalls
- Fitting Lasso without standardising the features beforehand.
- Picking lambda by hand instead of tuning it with cross-validation.
- Trusting the selected variables on a very small sample where the selection is unstable.
- Forgetting to check residual plots for normality and homogeneity after fitting.
- Ignoring that, with more predictors than observations, Lasso caps the number of selected variables.
Frequently asked questions
How is lambda chosen?
Lambda controls how strongly coefficients are penalised. It is selected by cross-validation: a range of values is tried and the one minimising the cross-validated mean squared error is kept.
Why must I standardise the features?
The single L1 penalty applies equally to every coefficient, so predictors measured on larger scales would otherwise be penalised less. Standardising the features first makes the penalty fair across variables.
How does Lasso differ from ridge regression?
Lasso uses an L1 penalty that can set coefficients exactly to zero, performing variable selection, while ridge uses an L2 penalty that shrinks coefficients but does not zero them out. On small samples or when selection is unstable, ridge is the safer choice.
What should I check after fitting?
Inspect the cross-validation MSE curve to confirm the chosen lambda, review which variables were kept versus set to zero, and examine residual plots for normality and homogeneity.
Related methods
Sources
Tibshirani, R. (1996). Regression Shrinkage and Selection via the Lasso. Journal of the Royal Statistical Society: Series B, 58(1), 267–288.
Open DOI referenceCopied.