Skip to content
Back to search results

Naive Bayes

Introduced by
Mitchell, T. M. (textbook treatment)
Year
1997
Type
Probabilistic classifier (Bayes' theorem with conditional independence)
Font size

What does it do?

Naive Bayes is a fast probabilistic classifier that applies Bayes' theorem while assuming that the features are conditionally independent given the class — a method given its standard machine-learning treatment in Tom Mitchell's 1997 textbook Machine Learning.

Despite this simplifying ('naive') assumption, it is quick to train and often surprisingly accurate.

The intuition behind the method

Instead of learning complex interactions between features, Naive Bayes asks a simpler question: given each class, how likely is each feature value on its own? It then multiplies these per-feature likelihoods together with the class's base rate and picks the class that comes out most probable. Pretending the features don't interact makes the maths trivial and the training extremely fast — and in practice the resulting classifier still ranks classes well even when the independence assumption is not strictly true.

History

Naive Bayes classifiers grow out of classical Bayesian decision theory and became a staple of applied machine learning, especially for text classification tasks like spam filtering. The method receives its standard textbook treatment in Tom Mitchell's 1997 Machine Learning, which presents the Bayes-theorem formulation and the conditional-independence assumption that give the classifier its name.

When to use it

A good fit for fast classification when you have categorical, binary, continuous, or text features and at least about 30 observations, especially as a strong, cheap baseline. It is assumption-light on distribution overall but each variant carries its own likelihood assumption (e.g. within-class normality for the Gaussian variant), and it is sensitive to class imbalance. When features are strongly correlated or you need calibrated effect estimates, prefer logistic regression.

Limitations and considerations

  • The conditional-independence assumption rarely holds, so probability estimates can be poorly calibrated even when the ranking is good.
  • Sensitive to class imbalance: priors must be adjusted or the data resampled.
  • Gaussian Naive Bayes degrades when the within-class normality assumption is violated.
  • Unreliable with very small samples, where per-class probability estimates are not trustworthy.

Source conditions do not replace independent scientific review or validation by an analysis engine.

Strengths

  • Extremely fast to train and to predict, even on large or high-dimensional data such as text.
  • Works well as a simple, low-cost baseline and often performs surprisingly well despite its naive assumption.
  • Handles categorical, binary, continuous, and text features through its different variants.
  • Needs relatively little training data to estimate its parameters.

Applications

  • Text classification such as spam filtering and document categorisation.
  • Fast baseline classification before trying more complex models.
  • Classification with categorical, binary, continuous, or count features.
  • High-dimensional problems where training speed matters.

How it works

How it works

1

By Bayes' theorem, the probability of a class given the observed features is proportional to the class's prior probability times the likelihood of those features under that class. The 'naive' step is to assume the features are conditionally independent given the class, so the joint likelihood factorises into a simple product of per-feature likelihoods. This assumption rarely holds exactly in practice, yet the classifier's performance can be surprisingly strong.

Formula
P(y \mid x_1,\dots,x_p) \propto P(y)\prod_{i=1}^{p} P(x_i \mid y)
2

From the training data the model estimates the class priors (how common each class is) and the per-feature likelihoods. The likelihood form depends on the variant: Gaussian Naive Bayes assumes continuous features are normally distributed within each class, while Multinomial Naive Bayes suits term-frequency counts in text classification and does not accept negative values. Because each class is estimated separately, there should be enough observations per class — at least about 30 overall.

3

For a new observation, the model combines the prior with the product of the per-feature likelihoods for each candidate class and assigns the class with the highest resulting probability. Since classes are scored independently, an imbalanced dataset can bias the result, so the prior probabilities should be tuned or the data resampled.

Formula
\hat{y} = \arg\max_{y} \; P(y)\prod_{i=1}^{p} P(x_i \mid y)

Common pitfalls

  • Trusting the raw output probabilities as well-calibrated rather than treating them as a ranking.
  • Ignoring class imbalance instead of tuning the priors or resampling.
  • Feeding negative values into Multinomial Naive Bayes, which it does not accept.
  • Applying Gaussian Naive Bayes to clearly non-normal continuous features without checking.
  • Running it on a dataset with too few observations per class (below ~30 overall).

Frequently asked questions

Why is it called 'naive'?

Because it assumes the features are conditionally independent given the class — that is, that knowing one feature tells you nothing about another within a class. This is rarely literally true, but the simplification makes the model trivial to estimate and fast to run, and the classifier often still ranks classes accurately.

Which variant should I use?

Gaussian Naive Bayes is for continuous features assumed normally distributed within each class. Multinomial Naive Bayes suits count or term-frequency data, as in text classification, and does not accept negative values. Match the variant to your feature type.

Can I trust the predicted probabilities?

Use the predicted class with care as a ranking. Because the independence assumption is usually violated, the raw probabilities tend to be pushed toward 0 or 1 and are often poorly calibrated, even when the chosen class is correct.

How much data does it need?

It is economical with data, but you still need enough observations per class — at least roughly 30 overall — so that the class priors and per-feature likelihoods can be estimated reliably. Below that, logistic regression is a safer choice.

Related methods

Sources

  1. Mitchell, T. M. (1997). Machine Learning. McGraw-Hill.

    ISBN: 978-0070428072
    Copied.