Online Linear Regression
- Introduced by
- Widrow, B. & Hoff, M. E. (LMS); Gauss / Plackett (RLS)
- Year
- 1960 (LMS); 1950 (RLS formalization)
- Type
- Incremental supervised regression
- Family
- Machine learning
What does it do?
Online Linear Regression fits a linear model one observation at a time, updating weights incrementally as each new data point arrives.
Unlike batch least-squares, it never needs to store or re-process the full dataset, making it the natural choice for streaming data, very large datasets, and environments where the data-generating process can shift over time.
The intuition behind the method
Imagine adjusting the dials of a linear predictor after every single measurement rather than waiting to collect all data first. Each new observation tells you how wrong your current guess was, and you nudge the weight vector a tiny step in the direction that would have reduced that error. Over many observations the weights converge to values close to the ordinary least-squares solution — but you got there without ever loading the whole dataset into memory, and you can keep adapting if the underlying relationship drifts.
History
The theoretical foundations trace to Gauss's least-squares method (early 19th century) and Plackett's 1950 recursive formulation. The practical online variant emerged with Widrow and Hoff's Least Mean Squares (LMS) algorithm in 1960, originally developed for adaptive signal processing. Online convex optimisation theory, formalised by Zinkevich (2003) and surveyed comprehensively by Shalev-Shwartz (2012), placed online linear regression within a rigorous regret-minimisation framework and demonstrated its near-optimal convergence guarantees.
When to use it
Use online linear regression when data arrives as a continuous stream and predictions must be served immediately, when the dataset is too large to hold in memory for a batch fit, or when the relationship between predictors and outcome may drift over time and the model needs to adapt. It is also a good choice as a lightweight baseline in real-time production systems. Avoid it when the relationship is strongly nonlinear, when feature engineering must be done holistically over the full dataset, or when you need exact OLS coefficients with confidence intervals for statistical inference — use batch linear regression in those cases.
Limitations and considerations
- Sensitive to the choice of learning rate: too large causes oscillation, too small causes slow convergence.
- Provides no closed-form standard errors or p-values, so classical statistical inference is not directly available.
- Can struggle with highly correlated features (multicollinearity) unless L2 regularisation is added.
- Performance degrades relative to batch OLS when all data is already available and there is no concept drift.
Source conditions do not replace independent scientific review or validation by an analysis engine.
Strengths
- Processes data one observation at a time with O(p) memory, where p is the number of features — no full dataset required.
- Adapts continuously to concept drift, making it suitable for non-stationary environments such as financial time series or sensor streams.
- Computationally cheap per update: a single dot product and a weight vector addition.
- Naturally parallelisable across multiple streams or shards.
- Produces predictions in real time without waiting for a batch re-fit.
Applications
- Real-time price prediction and demand forecasting in e-commerce and energy markets.
- Adaptive signal processing and noise cancellation using recursive least squares.
- Continuous model updating in IoT and sensor-network environments where batch re-fits are infeasible.
- Financial tick-data regression where the data-generating process shifts frequently.
- Large-scale recommendation-system feature weights updated as user events arrive.
- Baseline streaming model in AutoML pipelines before more complex models are considered.
How it works
How it works
Set the weight vector to zero (or a small random value) and choose a learning rate eta that controls the step size. Optionally initialise a regularisation term such as L2 (Ridge) to prevent the weights from growing unbounded in non-stationary streams.
Formula
\mathbf{w}_0 = \mathbf{0},\; \eta > 0For each arriving observation x_t, compute the predicted output as the dot product of the current weight vector with the feature vector. This prediction is made before the true label y_t is revealed, mirroring the sequential nature of real streaming settings.
Formula
\hat{y}_t = \mathbf{w}_t^\top \mathbf{x}_tAfter the true label is revealed, compute the squared prediction error. This instantaneous loss drives the gradient signal; the gradient of the squared loss with respect to the weights is simply the error times the feature vector.
Formula
\ell_t = (y_t - \hat{y}_t)^2Apply the LMS (least mean squares) update rule: move the weights in the direction of the negative gradient of the loss. The step size eta controls the trade-off between adaptation speed and noise sensitivity. Smaller eta yields smoother convergence; larger eta adapts faster but risks oscillation.
Formula
\mathbf{w}_{t+1} = \mathbf{w}_t + \eta\,(y_t - \hat{y}_t)\,\mathbf{x}_tProcess the next observation immediately, or buffer a small mini-batch for a mini-batch online variant. In non-stationary settings, a decaying learning rate or a sliding-window forgetting factor keeps the model responsive to concept drift. No full re-fit is ever required.
Common pitfalls
- Using a fixed learning rate on data with very different feature scales — always standardise features online (e.g., running mean and variance) before updating.
- Forgetting to add regularisation in high-dimensional or near-collinear settings, which lets weights diverge on sparse streams.
- Treating the online weights as equivalent to OLS estimates and reporting them with OLS confidence intervals — they are not identical.
- Ignoring concept drift detection: if the data distribution shifts sharply, a fixed learning rate may track noise rather than signal.
- Evaluating accuracy only at the end of a long stream instead of using prequential (test-then-train) evaluation throughout.
Frequently asked questions
How does online linear regression differ from standard (batch) linear regression?
Batch linear regression solves the normal equations on the entire dataset at once, giving the exact OLS solution. Online linear regression updates weights incrementally after each observation using a gradient step, converging to a near-OLS solution without ever loading all data — at the cost of losing exact inference statistics.
What learning rate should I use?
A common starting point is eta = 0.01 divided by the expected squared norm of the feature vectors. Grid-search or adaptive methods (AdaGrad, Adam) automatically scale the learning rate per feature and are often easier to tune than a fixed global rate.
Can online linear regression handle concept drift?
Yes. A constant (non-decaying) learning rate keeps the model responsive to drift. For abrupt shifts, explicit drift-detection tests (e.g., ADWIN) can trigger a partial weight reset. A decaying rate is better when the relationship is stationary and you want eventual convergence.
Is regularisation possible in the online setting?
Yes. Adding an L2 penalty gives the online Ridge (Tikhonov) update: subtract a small fraction of the current weight vector at each step. This prevents weights from growing large on sparse or noisy streams and is strongly recommended in high-dimensional settings.
How do I evaluate an online model fairly?
Use prequential (test-then-train) evaluation: record the prediction error on each observation before using it to update the model. This gives an unbiased, sequential estimate of generalisation error without a separate held-out test set.
Related methods
Sources
Shalev-Shwartz, S. (2012). Online Learning and Online Convex Optimization. Foundations and Trends in Machine Learning, 4(2), 107–194.
Open DOI referenceCopied.Haykin, S. (2002). Adaptive Filter Theory (4th ed.). Prentice Hall.
ISBN: 978-0130901262Copied.