Skip to content
Back to search results

Support Vector Regression

Introduced by
Smola, A.J. & Schölkopf, B.
Year
2004
Type
Kernel-based supervised model (epsilon-insensitive regression)
Font size

What does it do?

Support Vector Regression (SVR), described in Smola and Schölkopf's 2004 tutorial, predicts a continuous outcome by fitting a function that stays within an epsilon-wide tube around the data while incurring as little error as possible.

It extends the support vector machine idea from classification to regression, using a kernel to capture nonlinear relationships.

The intuition behind the method

Instead of penalising every deviation from the fitted line, SVR ignores errors smaller than a tolerance epsilon — it draws a tube of width epsilon around the predicted function and only points falling outside the tube count against it. The model then seeks the flattest possible function that keeps most points inside the tube. The handful of points on or outside the tube edges become the support vectors that define the prediction.

History

Support Vector Regression extends the support vector machine, originally a classification method, to the task of regression. The epsilon-insensitive formulation and its kernelised solution are laid out in Smola and Schölkopf's 2004 tutorial in Statistics and Computing, which has become the standard reference for the method.

When to use it

Use SVR for predicting a continuous (or count) outcome from features when the relationship may be nonlinear and you have at least about 50 observations. Features must be normalised, and the kernel and the C parameter need tuning. With fewer than 50 cases the prediction rests on too few support vectors and becomes unstable, so a ridge or linear regression is preferable.

Limitations and considerations

  • Needs at least about 50 observations; with fewer, predictions rest on too few support vectors and are unstable.
  • Performance hinges on tuning the kernel and the C parameter, which requires enough cross-validation data.
  • Features must be normalised beforehand or the kernel distances are distorted.
  • Offers no interpretable coefficients.

Source conditions do not replace independent scientific review or validation by an analysis engine.

Strengths

  • The epsilon-insensitive loss makes the fit robust to small deviations and to outliers within the tube.
  • Kernels let SVR model nonlinear relationships without explicitly engineering new features.
  • Does not require normally distributed data.
  • The final model depends only on the support vectors, keeping the prediction compact.

Applications

  • Predicting continuous outcomes where the relationship with the features is nonlinear.
  • Forecasting and prediction tasks where robustness to small deviations matters.
  • Modelling count or continuous targets in health, social, engineering and business data.
  • Serving as a kernel-based alternative to linear or ridge regression when nonlinearity is suspected.

How it works

How it works

1

SVR defines a margin of tolerance, the epsilon-tube, around the regression function: deviations within epsilon are treated as no error at all. This makes the fit insensitive to small fluctuations and focuses the model on the observations that genuinely fall outside the band.

Formula
|y_i - f(x_i)| \le \varepsilon
2

The model balances flatness of the function against the error incurred by points outside the tube. A regularisation parameter C controls how heavily out-of-tube deviations are penalised, while a kernel (such as the RBF kernel) lets the function bend to capture nonlinear structure. Because both C and the kernel must be tuned, the features should be normalised before fitting.

Formula
\min \tfrac{1}{2}\lVert w\rVert^2 + C\sum_i (\xi_i + \xi_i^*)
3

Predictions for new inputs are formed as a kernel-weighted combination of the support vectors — the training points lying on or beyond the tube edges — plus an offset. Only these support vectors enter the final prediction, which is what makes the fitted model compact.

Formula
f(x) = \sum_i (\alpha_i - \alpha_i^*) k(x_i, x) + b

Common pitfalls

  • Fitting SVR on unnormalised features, which distorts kernel distances.
  • Accepting default kernel and C values without tuning them by cross-validation.
  • Using SVR on a small sample where there is not enough data for hyperparameter optimisation.
  • Treating the kernel-based fit as if it yielded interpretable effect sizes.

Frequently asked questions

What does the epsilon parameter control?

Epsilon sets the half-width of the tolerance tube around the fitted function. Deviations smaller than epsilon are treated as zero error, so a larger epsilon yields a flatter, simpler model with fewer support vectors, while a smaller epsilon fits the data more tightly.

Why must I normalise the features?

SVR relies on distances computed through the kernel. If features are on very different scales those with larger ranges dominate the distances, so the features should be standardised before fitting.

How small a sample is too small?

With fewer than about 50 observations the fit depends on too few support vectors and becomes unstable, and there is not enough data to tune the kernel hyperparameters; a ridge or linear regression is a safer choice.

How does SVR differ from ordinary least squares?

Least squares penalises every residual quadratically, whereas SVR ignores residuals inside the epsilon-tube and penalises only those outside it. Combined with a kernel, this lets SVR capture nonlinear relationships and stay robust to small deviations.

Related methods

Sources

  1. Smola, A.J. & Schölkopf, B. (2004). A Tutorial on Support Vector Regression. Statistics and Computing, 14, 199–222.

    Open DOI reference
    Copied.