Skip to content
Back to search results

Support Vector Machine (Classification)

Introduced by
Cortes, C. & Vapnik, V.
Year
1995
Type
Maximum-margin classifier (kernel method)
Font size

What does it do?

The Support Vector Machine, introduced by Corinna Cortes and Vladimir Vapnik in 1995, is a classifier that finds the optimal separating hyperplane between classes in a high-dimensional space.

It chooses the boundary that leaves the widest possible margin to the nearest training points, which makes its decisions robust on new data.

The intuition behind the method

Imagine separating two groups of points with a straight boundary. Many lines could do it, but SVM looks for the one positioned as far as possible from the closest points of each class — the widest 'street' between them. Only those closest points, the support vectors, actually shape the boundary; everything farther away is irrelevant. When the classes cannot be split by a straight line, a kernel reshapes the space so a clean separation becomes possible.

History

The Support Vector Machine was introduced by Corinna Cortes and Vladimir Vapnik in their 1995 paper 'Support-Vector Networks', which combined the maximum-margin idea with the kernel trick to handle non-linearly-separable data. It became one of the most influential classification methods of its era and a standard benchmark for high-dimensional supervised learning.

When to use it

Use SVM for binary or categorical classification on tabular data with roughly 50 or more observations, especially when the feature space is high-dimensional and predictive accuracy matters more than explicit coefficients. It does not require normally distributed data, but features must be standardised and the kernel and C must be tuned by cross-validation. On very small samples, prefer a simpler classifier such as logistic regression or naive Bayes.

Limitations and considerations

  • Sensitive to the choice of kernel and the C parameter, which must be tuned by cross-validation.
  • Requires careful feature standardisation; unscaled features distort the margin.
  • Becomes unstable on small samples, where there are too few support vectors to define a reliable boundary.
  • Provides no directly interpretable coefficients and no native class-probability estimates.

Source conditions do not replace independent scientific review or validation by an analysis engine.

Strengths

  • Effective in high-dimensional feature spaces, even when there are more features than the boundary's complexity would suggest.
  • The maximum-margin principle and the use of only support vectors give robust, generalisable boundaries.
  • Kernels let it capture nonlinear class boundaries without manually engineering new features.
  • Assumption-light: it does not require normally distributed data.

Applications

  • Binary and multi-class classification on tabular data.
  • High-dimensional classification problems where features outnumber a simple model's capacity.
  • Prediction tasks where a robust, well-separated decision boundary matters more than interpretable coefficients.
  • Problems with nonlinear class boundaries, addressed through kernel functions.

How it works

How it works

1

SVM searches for the hyperplane that separates the classes with the largest margin — the greatest distance to the nearest data points on either side. Those nearest points are the support vectors, and they alone determine where the boundary sits. A wide margin tends to generalise better to unseen data than a boundary squeezed close to the training points.

2

When the classes are not linearly separable, a kernel function (for example the radial basis function) implicitly maps the features into a higher-dimensional space where a separating hyperplane does exist. This is why features must first be standardised (StandardScaler is required), and why the kernel and the regularisation parameter C should be chosen by cross-validation.

3

The boundary is found by minimising the objective above: it shrinks the weight vector (widening the margin) while paying a penalty C for each point that falls on the wrong side or inside the margin (the slack ξ). Large C demands a tight fit to the training data; small C tolerates more violations for a wider, smoother margin. With imbalanced classes, class_weight='balanced' keeps the minority class from being ignored.

Formula
\min_{w,b}\; \tfrac{1}{2}\lVert w\rVert^2 + C\sum_i \xi_i

Common pitfalls

  • Fitting SVM on raw, unscaled features instead of standardising them first.
  • Accepting default kernel and C values rather than selecting them with cross-validation (e.g. grid search).
  • Ignoring class imbalance instead of setting class_weight='balanced'.
  • Applying SVM on a sample below ~50 observations, where the boundary is unstable and prone to overfitting.
  • Reporting only accuracy without a confusion matrix, precision/recall/F1, and ROC-AUC.

Frequently asked questions

Why do I have to standardise the features?

SVM measures distances to the separating hyperplane, so features on larger numeric scales dominate the margin. Standardising every feature (StandardScaler) puts them on comparable scales so the boundary reflects the true class structure rather than arbitrary units.

What does the C parameter control?

C trades off margin width against classification errors. A large C forces a tight fit to the training data with a narrow margin; a small C allows more margin violations in exchange for a wider, smoother boundary. The right value is found by cross-validation.

What is the kernel for?

When the classes cannot be separated by a straight boundary, a kernel (such as the radial basis function) implicitly maps the data into a higher-dimensional space where a separating hyperplane exists, letting SVM model nonlinear boundaries. The kernel choice should be validated by cross-validation.

When should I not use SVM?

On very small samples — below about 50 observations — there are too few support vectors to define a stable boundary and there is not enough data to tune the kernel and C, so the model risks overfitting. In that case a simpler classifier such as logistic regression or naive Bayes is preferable.

Related methods

Sources

  1. Cortes, C. & Vapnik, V. (1995). Support-Vector Networks. Machine Learning, 20, 273–297.

    Open DOI reference
    Copied.