Skip to content
Back to search results

Knowledge Distillation

Introduced by
Hinton, G., Vinyals, O. & Dean, J.
Year
2015
Type
Neural network compression (teacher–student)
Font size

What does it do?

Knowledge Distillation is a model-compression technique, introduced by Geoffrey Hinton and colleagues in 2015, that trains a small student model using the soft-label outputs of a large teacher model.

Distilled models such as DistilBERT and TinyBERT reach roughly 97% of the larger model's performance while running far faster.

The intuition behind the method

A large, accurate model is often too slow or heavy to deploy. Instead of training a small model from scratch on hard labels alone, you let it learn from the rich probability outputs of the big model: the teacher's full distribution over classes carries far more information than a single correct answer, revealing which classes look similar and how confident the teacher is. The student absorbs this 'dark knowledge' and ends up much smaller yet nearly as capable.

History

Knowledge Distillation was introduced by Geoffrey Hinton, Oriol Vinyals and Jeff Dean in a 2015 NeurIPS Deep Learning Workshop paper, which framed compressing a large model into a small one by matching the larger model's softened outputs. It became a standard tool for deploying compact deep models; Sanh and colleagues' 2019 DistilBERT showed a distilled version of BERT that is smaller, faster, cheaper and lighter while retaining most of the original's accuracy.

When to use it

Use it when you have a strong but heavy trained teacher model and want a smaller, faster student for deployment on text or continuous-feature tasks, with at least about 100 task-specific observations. It assumes the teacher model is already trained and available and that a task-specific dataset is ready. Below roughly 500 observations the student cannot distil enough from the teacher and overfitting becomes a risk; below about 100, distillation is pointless and classic machine learning is the safer choice.

Limitations and considerations

  • Requires an already-trained, available teacher model — distillation cannot start without one.
  • Needs a task-specific dataset; with too little data the student learns little from the teacher.
  • On small datasets (n below about 500) the student overfits and fails to absorb the teacher's knowledge.
  • Adds a training pipeline and tuning burden (temperature, mixing weight) on top of the original model.

Source conditions do not replace independent scientific review or validation by an analysis engine.

Strengths

  • Yields a much smaller, faster model that can recover around 97% of the teacher's performance.
  • The teacher's soft labels carry richer information than hard labels, improving the student's generalisation.
  • Assumption-light: does not require normally distributed data.
  • Reuses an existing trained teacher, so it suits deployment and serving where speed and size matter.

Applications

  • Compressing large language models into deployable students such as DistilBERT and TinyBERT.
  • Speeding up inference for text classification and prediction where the full teacher is too slow.
  • Shrinking models for resource-constrained or latency-sensitive serving environments.
  • Transferring a heavy model's capability into a lightweight one without retraining from scratch.

How it works

How it works

1

A large, well-trained teacher model is taken as given — it must already be trained and available. This teacher has the predictive power you want to preserve but is too expensive to run at scale.

2

The teacher produces soft-label outputs: full probability distributions over the classes rather than a single hard label. These softened predictions expose the similarities the teacher sees between classes, providing a far richer training signal than the ground-truth labels alone.

3

The small student model is trained to match both the true labels and the teacher's soft outputs at once. The distillation loss combines a standard cross-entropy term on the hard labels with a Kullback–Leibler divergence term that pulls the student's logits toward the teacher's, weighted by a mixing factor alpha. The result is a compact student that retains most of the teacher's accuracy.

Formula
\mathcal{L} = \alpha\, \mathrm{CE}(y, p_s) + (1-\alpha)\, \mathrm{KL}(p_t \,\|\, p_s)

Common pitfalls

  • Attempting distillation without a properly trained teacher to copy from.
  • Running it on a tiny task dataset where classic machine learning would do better.
  • Ignoring the alpha mixing weight between the hard-label and soft-label loss terms.
  • Expecting the student to fully match the teacher rather than approach it.

Frequently asked questions

Why use soft labels instead of just the correct answer?

The teacher's full probability distribution reveals how similar the classes look to it and how confident it is. This extra signal — sometimes called dark knowledge — helps the student generalise better than hard labels alone.

How much smaller can the student be?

Distilled models such as DistilBERT and TinyBERT are substantially smaller and faster than their teachers while reaching roughly 97% of the teacher's performance, though the exact trade-off depends on the task and architecture.

What do I need before distilling?

You need a teacher model that is already trained and available, plus a task-specific dataset. With fewer than about 500 observations the student struggles to distil enough from the teacher and may overfit.

What is the alpha in the loss?

Alpha mixes the two parts of the distillation loss: a cross-entropy term on the true labels and a Kullback–Leibler divergence term that aligns the student's outputs with the teacher's. Tuning it balances learning from ground truth versus learning from the teacher.

Related methods

Sources

  1. Hinton, G., Vinyals, O. & Dean, J. (2015). Distilling the Knowledge in a Neural Network. NeurIPS Deep Learning Workshop.

    Open publication reference
    Copied.
  2. Sanh, V., Debut, L., Chaumond, J. & Wolf, T. (2019). DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. arXiv:1910.01108.

    Open publication reference
    Copied.