Skip to content
Back to search results

Linear Regression (ML)

Introduced by
Legendre, A.-M. & Gauss, C.F.
Year
1805–1809
Type
Supervised regression
Family
Machine learning
Font size

What does it do?

Linear regression fits a straight-line relationship between one or more input features and a continuous numeric outcome by minimising the sum of squared prediction errors.

As a machine-learning model it is trained on labeled examples and evaluated on held-out data, making it the simplest supervised learning baseline for any regression task.

The intuition behind the method

Imagine drawing the best-fitting line through a scatter plot of data points. Linear regression does exactly this for many predictors at once: it finds the combination of feature weights that minimises the total squared distance between the line's predictions and the true values. Because the relationship is expressed as a simple weighted sum of inputs, each coefficient tells you directly how much the predicted outcome changes when that feature increases by one unit, holding everything else constant. This transparency makes linear regression uniquely easy to explain and audit, even though it will underfit whenever the true relationship is curved or involves interactions.

History

Least squares was independently published by Legendre in 1805 and Gauss around 1809. By the mid-twentieth century linear regression had become the foundational technique of applied statistics. Its reframing as a machine-learning model — trained on examples, evaluated on held-out data, and regularised to prevent overfitting — gave it a second life in predictive modelling, where it remains the canonical first-step baseline for any regression problem.

When to use it

Use linear regression when the outcome is continuous, the expected relationship between predictors and outcome is approximately linear, and interpretable coefficients are needed. It is the right starting baseline before trying more complex models, and a good final model when its assumptions hold. Do not use it for binary or categorical outcomes (use logistic regression instead), when strong nonlinear patterns or interactions dominate, or when the number of predictors exceeds the sample size without regularisation.

Limitations and considerations

  • Assumes a linear relationship; misses curves, thresholds, and interactions unless they are manually encoded.
  • Sensitive to outliers because squaring errors amplifies the influence of extreme points.
  • Multicollinearity among predictors inflates standard errors and destabilises coefficient estimates.
  • Requires that residuals be approximately normally distributed and homoscedastic for valid inference.

Source conditions do not replace independent scientific review or validation by an analysis engine.

Strengths

  • Coefficients are directly interpretable: each value is a marginal effect in the outcome's units.
  • Computationally trivial — fits almost instantly even on large datasets.
  • Provides standard errors and p-values, supporting formal hypothesis testing.
  • Serves as the indispensable baseline against which more complex models should be compared.
  • Works well and is fully justified when the linearity assumption holds.
  • Highly compatible with regularisation extensions (Ridge, Lasso, Elastic Net) when predictors are many.

Applications

  • Predicting a continuous numeric outcome from tabular features in any domain.
  • Establishing a performance baseline before testing gradient-boosting or neural-network models.
  • Estimating effect sizes and marginal impacts in observational studies when linearity holds.
  • Demand forecasting, price modelling, and resource estimation in business analytics.
  • Rapid exploratory modelling to identify the most important predictors before more complex analysis.
  • Interpretable scoring functions in settings where model transparency is legally or ethically required.

How it works

How it works

1

The model expresses the prediction as a weighted sum of input features plus an intercept. The target variable must be continuous and numeric; categorical inputs must be encoded as dummy or numeric variables before fitting.

Formula
\hat{y} = \beta_0 + \beta_1 x_1 + \cdots + \beta_p x_p
2

Ordinary least squares finds the coefficient vector that minimises the residual sum of squares. For moderate-sized datasets this is solved analytically via the normal equations; for very large datasets iterative solvers or stochastic gradient descent are used instead.

Formula
\hat{\boldsymbol{\beta}} = (\mathbf{X}^\top \mathbf{X})^{-1} \mathbf{X}^\top \mathbf{y}
3

R-squared (coefficient of determination) summarises how much of the variance in the outcome the model explains. It is complemented by root mean squared error and mean absolute error, which report errors in the original units of the outcome.

Formula
R^2 = 1 - \frac{\sum_i (y_i - \hat{y}_i)^2}{\sum_i (y_i - \bar{y})^2}
4

Fit is evaluated on a hold-out test set or through k-fold cross-validation to detect overfitting. Residual plots — residuals versus fitted values and a Q-Q plot of residuals — are inspected to check linearity, homoscedasticity, and normality of errors before trusting inference.

Common pitfalls

  • Extrapolating predictions far outside the range of training data where the linear approximation breaks down.
  • Ignoring residual diagnostics and assuming the model is valid without checking plots.
  • Including highly correlated predictors without addressing multicollinearity (check VIF).
  • Confusing R-squared with predictive accuracy on new data — always evaluate on a held-out set.
  • Applying linear regression to a binary or count outcome instead of logistic or Poisson regression.

Frequently asked questions

How is linear regression different from logistic regression?

Linear regression predicts a continuous outcome directly. Logistic regression models the log-odds of a binary outcome and outputs probabilities bounded between 0 and 1. Using linear regression for a binary outcome can produce predictions outside [0, 1] and violates the constant-variance assumption.

Do I need to standardise my features?

Standardisation is not required for predictions, but it makes coefficients directly comparable when features have very different scales. It is especially recommended when using regularised variants such as Ridge or Lasso.

How do I handle many predictors relative to sample size?

Switch to regularised linear regression (Ridge for shrinkage, Lasso for sparsity, Elastic Net for both). OLS becomes unreliable or undefined when the number of predictors approaches or exceeds the number of observations.

What residual plots should I inspect?

At minimum: (1) residuals vs. fitted values to check for patterns suggesting nonlinearity or heteroscedasticity, and (2) a Q-Q plot of residuals to assess normality. A scale-location plot and leverage/influence statistics (Cook's distance) are also informative.

Is a high R-squared enough to declare a good model?

No. R-squared measures in-sample fit and can be inflated by irrelevant predictors. Always report cross-validated or test-set RMSE alongside R-squared, and inspect residual diagnostics to confirm the model's assumptions are satisfied.

Related methods

Sources

  1. Hastie, T., Tibshirani, R. & Friedman, J. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction (2nd ed., Ch. 3). Springer.

    ISBN: 978-0-387-84858-7
    Copied.
  2. James, G., Witten, D., Hastie, T. & Tibshirani, R. (2013). An Introduction to Statistical Learning (Ch. 3). Springer.

    ISBN: 978-1-4614-7138-7
    Copied.