Skip to content
Back to search results

BERT Fine-Tuning

Introduced by
Devlin, J. et al.
Year
2019
Type
Transfer learning (fine-tuning a pre-trained transformer)
Font size

What does it do?

BERT fine-tuning, building on the BERT model introduced by Devlin and colleagues in 2019, re-trains a pre-trained BERT model on a small labelled dataset for a target task such as classification, named-entity recognition, or question answering.

Through transfer learning it reaches high performance even with relatively little task-specific data.

The intuition behind the method

Training a language model from scratch needs enormous amounts of text and compute. BERT has already learned rich, general representations of language during pre-training, so instead of starting over you take that model and gently adjust it to your task. With only a small labelled dataset, the model adapts its existing knowledge to your problem, which is why fine-tuning works where training a model from zero would fail.

History

BERT was introduced by Devlin and colleagues in 2019, demonstrating that deep bidirectional transformers pre-trained on large corpora could be fine-tuned to reach state-of-the-art results across many language-understanding tasks. Practical guidance such as Sun and colleagues' 2019 work on how to fine-tune BERT for text classification helped establish fine-tuning as a standard recipe for adapting large pre-trained language models to specific tasks.

When to use it

Use BERT fine-tuning for text tasks — classification, prediction, or relation extraction such as NER and question answering — on text variables, with at least about 50 labelled examples and ideally several hundred or more. It assumes a pre-trained BERT model is available and that you have labelled task data; a GPU is recommended. Below about 500 labelled examples the model tends to overfit, and below about 50 fine-tuning is not viable — classical TF-IDF-based machine learning is the safer choice there.

Limitations and considerations

  • On small labelled datasets (below about 500 examples) fine-tuning overfits and outputs become unreliable.
  • With very little labelled data (below about 50) fine-tuning is not feasible at all.
  • A GPU is recommended, so compute requirements are higher than for classical text models.
  • Depends on having a suitable pre-trained BERT model available for the language and domain.

Source conditions do not replace independent scientific review or validation by an analysis engine.

Strengths

  • Transfer learning delivers high performance even with relatively little labelled data.
  • Captures bidirectional context, handling word meaning that depends on surrounding text.
  • A single pre-trained backbone serves many tasks: classification, NER, question answering.
  • Avoids the enormous data and compute cost of training a language model from scratch.

Applications

  • Text classification such as sentiment analysis and topic labelling.
  • Named-entity recognition (NER) over text.
  • Question answering and span extraction.
  • Adapting a general pre-trained language model to a specific domain with limited labelled data.

How it works

How it works

1

You start from a pre-trained BERT model that has already learned bidirectional representations of language from large text corpora. This pre-trained model carries general linguistic knowledge that transfers to your target task, so you do not learn language structure from scratch.

2

A task-specific output layer is attached on top of BERT for your target task — for example a classification head for sentiment labelling, a token-level head for named-entity recognition, or a span head for question answering. This head turns BERT's contextual representations into predictions for your task.

3

The combined model is re-trained on your small labelled task dataset, updating the weights so they specialise to the task. Typically the model is trained for a few epochs; a GPU is recommended because the underlying network is large. The result is a model that maps text inputs to task predictions.

Formula
\hat{y} = \text{softmax}(W \cdot h_{\text{[CLS]}} + b)

Common pitfalls

  • Fine-tuning on too few labelled examples and trusting the resulting metrics.
  • Skipping a held-out evaluation set, so overfitting on a small dataset goes unnoticed.
  • Reaching for BERT when a TF-IDF-based classical model would be more reliable on tiny data.
  • Training for too many epochs on a small dataset, which amplifies overfitting.

Frequently asked questions

How much labelled data do I need?

Fine-tuning is viable from around 50 labelled examples, but performance is more reliable with several hundred or more. Below about 500 examples the model tends to overfit, and below about 50 fine-tuning is not feasible — a TF-IDF-based classical model is safer there.

Do I need a GPU?

A GPU is recommended because the BERT network is large and training on CPU is slow. Fine-tuning typically runs for only a few epochs, which keeps the compute manageable on a GPU.

Why fine-tune instead of training from scratch?

BERT has already learned general bidirectional language representations during pre-training. Fine-tuning reuses that knowledge through transfer learning, so you reach high performance with far less labelled data and compute than training a model from zero.

What tasks can BERT fine-tuning handle?

By attaching a task-specific output head you can use the same pre-trained backbone for classification, named-entity recognition, and question answering, among other text tasks.

Related methods

Sources

  1. Devlin, J., Chang, M.-W., Lee, K. & Toutanova, K. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. NAACL.

    Open DOI reference
    Copied.
  2. Sun, C., Qiu, X., Xu, Y. & Huang, X. (2019). How to Fine-Tune BERT for Text Classification. CCL.

    Open DOI reference
    Copied.