Skip to content
Back to search results

Online Logistic Regression

Introduced by
Rosenblatt, F. / Widrow, B. (perceptron era); modern SGD form: Bottou, L.
Year
1960s (perceptron); formalized for logistic loss ~2000s
Type
Incremental supervised classifier
Family
Machine learning
Font size

What does it do?

Online Logistic Regression fits a logistic classifier one sample (or mini-batch) at a time via stochastic gradient descent, updating model weights as each observation arrives rather than waiting to see the full dataset.

This makes it the standard choice for high-volume, streaming, or memory-constrained binary classification problems where batch training is infeasible.

The intuition behind the method

Classic logistic regression requires loading all data into memory and optimizing over the whole dataset. Online logistic regression instead reads each incoming example, makes a probability prediction, computes the prediction error, and takes a small gradient step to correct its weights before moving to the next example. Over many examples the weights converge to values close to what a batch solver would find, yet the model can be updated continuously without ever storing the full dataset. A shrinking learning rate ensures the updates become finer as the model matures, trading speed of adaptation for stability.

History

The roots of online gradient-based learning trace to Rosenblatt's perceptron (1958) and Widrow and Hoff's LMS rule (1960). The logistic loss variant gained prominence in the large-scale learning literature of the 2000s, particularly through Bottou's work on stochastic gradient descent for machine learning, and Shalev-Shwartz's theoretical framework for online convex optimization. It became a production standard in web-scale click-through-rate prediction and spam filtering, where datasets stream continuously and models must be updated in real time.

When to use it

Use Online Logistic Regression when data arrive as a stream or when the full dataset is too large to hold in memory, when the class boundary is expected to shift gradually over time (concept drift), or when low-latency model updates are required. It is also a fast baseline for very large static datasets where a single SGD pass is acceptable. Avoid it when the dataset is small enough to batch-train (batch logistic regression will converge to a better solution with less tuning), when the decision boundary is highly nonlinear (tree-based or kernel methods will fit better), or when interpretability of exact coefficient standard errors and p-values is required for reporting.

Limitations and considerations

  • Sensitive to learning-rate choice: too large causes divergence, too small causes slow convergence or underfitting.
  • Does not provide closed-form confidence intervals or p-values for coefficients.
  • Converges to a noisier solution than batch gradient descent given the same data; multiple passes or averaging (Polyak-Ruppert) are needed to close the gap.
  • Performance degrades when features are on very different scales unless standardization or AdaGrad is applied.
  • Assumes a linear decision boundary; cannot capture nonlinear structure without feature engineering.

Source conditions do not replace independent scientific review or validation by an analysis engine.

Strengths

  • Constant memory footprint: processes one example at a time without storing the entire dataset.
  • Adapts to concept drift by continuing to update weights as new labeled data arrive.
  • Scales to billions of examples where batch optimization would be prohibitive.
  • Produces calibrated probability outputs through the sigmoid link, suitable for risk scoring.
  • Compatible with L1 and L2 regularization and learning-rate schedules that mirror full batch solutions.

Applications

  • Click-through-rate prediction in online advertising with continuous data streams.
  • Spam and fraud detection where patterns evolve and models must adapt daily.
  • Real-time sentiment classification of social-media posts as they arrive.
  • Large-scale binary risk scoring when the full dataset does not fit in RAM.
  • Concept-drift scenarios in financial transaction monitoring.
  • Baseline streaming classifier before evaluating more complex online ensembles.

How it works

How it works

1

Weight vector and bias are initialized to zero (or small random values). No data pass is needed at this stage — the model is ready to process its first observation immediately.

Formula
\mathbf{w}_0 = \mathbf{0},\; b_0 = 0
2

For each incoming observation the model computes the linear combination of its current weights and features, then passes it through the sigmoid function to obtain a probability that the label is 1. This probability is used both as the prediction and as an ingredient in the gradient computation.

Formula
\hat{p}_t = \sigma(\mathbf{w}_t^\top \mathbf{x}_t + b_t),\quad \sigma(z)=\frac{1}{1+e^{-z}}
3

The binary cross-entropy (log-loss) between the predicted probability and the true label measures how wrong the current prediction is. Cross-entropy is convex in the weights, which guarantees that gradient steps reliably reduce error on average across the data stream.

Formula
\ell_t = -[y_t \log \hat{p}_t + (1-y_t)\log(1-\hat{p}_t)]
4

The gradient of the loss with respect to the weights reduces to the simple residual (predicted probability minus true label) times the feature vector. A regularization term (L2 or L1) is added to penalize large weights and prevent overfitting. The weights are updated by subtracting the scaled gradient in a single pass per example.

Formula
\mathbf{w}_{t+1} = \mathbf{w}_t - \eta_t (\hat{p}_t - y_t)\mathbf{x}_t - \eta_t \lambda \mathbf{w}_t
5

The learning rate is annealed over time — starting large to move quickly in early stages and shrinking so that later updates make fine corrections without oscillating. Common schedules are inverse-time decay or AdaGrad-style per-feature adaptation, which handles features with very different scales automatically.

Formula
\eta_t = \eta_0 / (1 + \alpha t)

Common pitfalls

  • Forgetting to standardize features: online SGD is especially sensitive to scale differences because the same learning rate is applied to all dimensions.
  • Using a fixed learning rate throughout training: weights oscillate near the optimum and never settle.
  • Ignoring class imbalance in the data stream: under-represented classes receive few gradient signals and the model skews toward the majority class.
  • Treating early-phase weight estimates as final: the model needs sufficient examples before its weights stabilize; do not evaluate too early.
  • Confusing online-update capability with automatic concept-drift detection: weights drift with the data but explicit drift monitoring (e.g., ADWIN) is still needed to detect abrupt shifts.

Frequently asked questions

How does online logistic regression differ from standard logistic regression?

Standard logistic regression uses all data at once (batch optimization) and returns exact maximum-likelihood coefficients with standard errors. Online logistic regression updates weights after each example, using only constant memory and tolerating infinitely long streams, but its estimates are noisier and it does not produce inferential statistics.

How do I choose the learning rate?

A common starting point is a small fixed rate (e.g., 0.01) combined with inverse-time decay, or using AdaGrad which adapts per-feature rates automatically. Always standardize features first, then tune the initial rate on a held-out validation fold or by monitoring the running log-loss on a recent window of the stream.

Can online logistic regression handle multiclass problems?

Yes — extend it to one-vs-rest binary classifiers trained in parallel, or use the softmax (multinomial) generalization with categorical cross-entropy loss and per-class weight vectors, applying the same SGD update rule to each.

What regularization should I use?

L2 (ridge) regularization is the default: it shrinks all weights smoothly and plays well with SGD. L1 (lasso) produces sparse weights and is useful when you want automatic feature selection in very high-dimensional spaces, but requires careful truncation or FOBOS updates to stay exactly sparse online.

How do I evaluate a model that is continuously updated?

Use prequential (test-then-train) evaluation: predict each example before using it for a weight update, then report the running log-loss or AUC over a sliding window. This gives an unbiased estimate of current model quality without a fixed held-out set.

Related methods

Sources

  1. Bottou, L. (2010). Large-Scale Machine Learning with Stochastic Gradient Descent. In Proceedings of COMPSTAT 2010, 177–186. Physica-Verlag.

    Open publication reference
    Copied.
  2. Shalev-Shwartz, S. (2012). Online Learning and Online Convex Optimization. Foundations and Trends in Machine Learning, 4(2), 107–194.

    Open DOI reference
    Copied.