One-Class SVM
- Introduced by
- Scholkopf, B., Platt, J. C., Smola, A. J., Williamson, R. C.
- Year
- 1999–2001
- Type
- Anomaly / novelty detection (unsupervised)
- Family
- Machine learning
What does it do?
One-class SVM is an unsupervised anomaly and novelty detection algorithm that learns a tight boundary around normal training data in a kernel-induced feature space, flagging new observations that fall outside that boundary as outliers.
Introduced by Scholkopf et al. in 1999–2001, it extends the SVM framework to the single-class setting where no labelled anomalies are available.
The intuition behind the method
Imagine inflating a balloon to enclose all your training points in a transformed space defined by a kernel function. Once the balloon is fitted, any new point that lands outside it is declared an anomaly. The parameter nu controls how tight the balloon is: a small nu allows very few training points to be inside the margin (strict boundary), while a large nu allows more points outside (looser boundary). The kernel trick lets the boundary be non-linear in the original input space, so One-class SVM can capture complex shapes of normality that a simple threshold or distance rule would miss.
History
The one-class SVM was introduced by Scholkopf, Platt, Smola, and Williamson in a 1999 technical report and published in Neural Computation in 2001 as 'Estimating the support of a high-dimensional distribution.' A closely related formulation, Support Vector Data Description (SVDD), was developed simultaneously by Tax and Duin (2004). Both methods extended the supervised SVM framework to the unsupervised anomaly detection setting and became foundational references in the novelty detection literature. Scikit-learn's OneClassSVM implementation made the method widely accessible in applied research.
When to use it
Use One-class SVM when labelled anomaly examples are unavailable or very scarce and you want a non-linear boundary around normal behaviour — typical in fraud detection, intrusion detection, industrial fault monitoring, and medical novelty screening with continuous numeric features. It works best on moderately sized datasets (hundreds to tens of thousands of samples) and high-dimensional or complex-shaped normality distributions where linear or distance-based detectors fail. Avoid it when: the training set is very large (n > ~100 000), because the quadratic solve scales poorly; the data are highly categorical; or when fast, interpretable rules are required. In those cases prefer Isolation Forest, Local Outlier Factor, or a threshold-based statistical model.
Limitations and considerations
- Quadratic programming solver scales as O(n^2) to O(n^3) in training samples, making it impractical for very large datasets.
- Sensitive to the choice of kernel and its hyperparameters (e.g., RBF gamma); poor choices produce either too tight or too loose a boundary.
- Requires clean normal training data — contaminated training sets shift the boundary and inflate false-negative rates.
- Does not naturally produce calibrated probability estimates; the decision score is not a probability.
- Interpretability is limited; explaining why a specific point is flagged is difficult without additional post-hoc tools.
Source conditions do not replace independent scientific review or validation by an analysis engine.
Strengths
- Works with no labelled anomaly examples — only normal training data are required.
- Non-linear kernel (typically RBF) captures complex, curved boundaries of normality that distance-based methods miss.
- The nu parameter gives direct, interpretable control over the expected outlier fraction.
- Scales gracefully to high-dimensional feature spaces via the kernel trick.
- The continuous decision score enables flexible downstream threshold calibration.
- Proven in high-stakes domains such as intrusion detection, fault monitoring, and medical imaging.
Applications
- Network intrusion detection: flagging traffic patterns not seen during normal operation.
- Industrial fault monitoring: detecting machine sensor readings that deviate from healthy-state profiles.
- Fraud detection: identifying unusual transaction patterns when fraudulent examples are too rare to train a supervised classifier.
- Medical novelty detection: flagging imaging or biomarker readings outside the distribution of healthy subjects.
- Natural language processing: out-of-domain document or query detection using text embeddings as input features.
How it works
How it works
All training observations are mapped into a high-dimensional (potentially infinite-dimensional) feature space via a kernel function such as the radial basis function (RBF). The inner products in that space can be evaluated without computing the mapping explicitly, which makes the approach tractable even for complex boundaries.
Formula
\phi: \mathcal{X} \to \mathcal{F}A quadratic program finds a hyperplane in the feature space that separates the mapped training data from the origin with maximum margin. The nu parameter (between 0 and 1) simultaneously bounds the fraction of training points treated as outliers and the fraction of support vectors, giving the practitioner direct control over the trade-off between model tightness and training contamination tolerance.
Formula
\min_{w,\rho,\xi}\; \tfrac{1}{2}\|w\|^2 - \rho + \tfrac{1}{\nu n}\sum_i \xi_iThe learned hyperplane defines a signed decision function. Points on the positive side are classified as normal; points on the negative side (below the threshold rho) are flagged as anomalies or novelties. The raw score can also serve as a continuous anomaly score for ranking.
Formula
f(x) = \text{sgn}\bigl(\langle w, \phi(x)\rangle - \rho\bigr)Choosing nu is the key modelling decision: setting it close to the expected contamination fraction in the training set is a practical rule of thumb. Cross-validation on a held-out normal validation set or domain knowledge about the expected outlier rate should guide the choice rather than grid search on unlabelled data alone.
In deployment, an anomaly score threshold is applied to the decision function output. The threshold can be adjusted post-hoc to trade precision against recall for anomalies, making the method suitable for applications where the cost of false positives differs substantially from the cost of false negatives.
Common pitfalls
- Training on data that contains a significant fraction of undetected anomalies, which corrupts the learned boundary.
- Setting nu without reference to the expected contamination rate, leading to either excessive false positives or missed anomalies.
- Using the default RBF gamma without tuning, which can make the boundary either too tight (all new points are anomalies) or too loose (no anomalies detected).
- Applying One-class SVM to very large datasets without considering scalability alternatives such as SGD-based approximations (sklearn's SGDOneClassSVM).
- Reporting only a binary anomaly flag without examining the continuous score distribution to calibrate the threshold.
Frequently asked questions
What is the difference between One-class SVM and Isolation Forest for anomaly detection?
One-class SVM learns a kernel-based boundary around normal data and performs well on high-dimensional, complex-shaped distributions, but scales poorly with dataset size. Isolation Forest uses random partitioning and is much faster on large datasets but is generally less sensitive to subtle, non-linear anomaly patterns.
How should I choose the nu parameter?
Set nu to the fraction of training observations you expect to be contaminated or atypical. If you believe fewer than 5% of your training samples are outliers, start with nu=0.05. Use a held-out set of known normal examples to evaluate boundary tightness rather than relying on grid search alone.
What kernel should I use?
The RBF (Gaussian) kernel is the standard starting point and handles most continuous numeric datasets well. The gamma hyperparameter of the RBF kernel controls the radius of influence; cross-validate over a log-spaced grid. Linear kernels are faster but restrict the boundary to a hyperplane.
Can One-class SVM handle categorical features?
Not directly. Categorical variables must be encoded numerically (e.g., binary or ordinal encoding) before use, which can introduce distance-metric distortions. For datasets dominated by categorical features, consider tree-based detectors like Isolation Forest instead.
My training set may contain some anomalies. Does that break the model?
Contaminated training data shifts the boundary outward and may cause the model to learn anomalous patterns as normal. One-class SVM is more sensitive to this than Isolation Forest. If contamination is suspected, consider using SVDD with a higher nu or cleaning the training set with a preliminary pass of a fast detector.
Related methods
Sources
Scholkopf, B., Platt, J. C., Shawe-Taylor, J., Smola, A. J., & Williamson, R. C. (2001). Estimating the support of a high-dimensional distribution. Neural Computation, 13(7), 1443–1471.
Open DOI referenceCopied.Tax, D. M. J., & Duin, R. P. W. (2004). Support vector data description. Machine Learning, 54(1), 45–66.
Open DOI referenceCopied.