BERT-based Classification
- Introduced by
- Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (Google AI Language)
- Year
- 2019
- Type
- Pre-trained language model with fine-tuning
- Family
- Deep learning / NLP / CV
What does it do?
BERT-based Classification fine-tunes Google's Bidirectional Encoder Representations from Transformers model on a labelled text dataset, replacing the generic pre-trained head with a task-specific classification layer.
It exploits deep bidirectional context from hundreds of millions of pre-trained parameters to deliver state-of-the-art accuracy on short- and medium-length text classification tasks with relatively modest amounts of labelled data.
The intuition behind the method
BERT reads every word in a sentence with full awareness of all surrounding words in both directions simultaneously — unlike earlier models that read left-to-right or right-to-left. Before fine-tuning, BERT has already absorbed rich language patterns from a massive corpus. To classify text, a thin classification layer is placed on top and the entire model is briefly re-trained on your labelled examples. The model adjusts its internal representations to suit your specific categories while retaining the broad linguistic knowledge it was pre-trained on, allowing strong performance even when labelled data is scarce.
History
BERT was published by Devlin et al. at Google AI Language in 2019 and immediately set new state-of-the-art results on 11 NLP benchmarks including GLUE and SQuAD. Its key innovation was bidirectional pre-training via masked language modelling and next-sentence prediction. Since 2019 it has spawned a family of derivatives — RoBERTa, ALBERT, DistilBERT, domain-specific variants — and became the foundation of modern applied NLP, including text classification in academic, industrial, and clinical settings.
When to use it
Use BERT-based classification when you have labelled text data and want state-of-the-art accuracy for tasks such as sentiment analysis, topic categorisation, intent detection, or clinical text classification. It works well with as few as a few hundred labelled examples per class, making it a strong choice when large annotated corpora are unavailable. Prefer it over simpler models (TF-IDF + logistic regression, CNN, LSTM) when linguistic nuance, long-range context, or subtle semantic distinctions drive the task. Avoid it when sequences exceed 512 tokens without special handling, when GPU compute is unavailable or too costly, when real-time inference latency below a few milliseconds is required, or when the domain is highly specialised and very little in-domain pre-training data exists — in such cases, a domain-specific model (BioBERT, LegalBERT) may be more appropriate.
Limitations and considerations
- Hard sequence-length limit of 512 tokens; longer documents require truncation, chunking, or hierarchical approaches.
- High computational and memory cost at inference time compared to lighter models such as DistilBERT or logistic regression.
- Fine-tuning on very small datasets (fewer than ~100 examples per class) risks catastrophic forgetting or overfitting.
- Model internals remain largely opaque; explaining predictions requires additional tools such as attention visualisation or SHAP.
Source conditions do not replace independent scientific review or validation by an analysis engine.
Strengths
- Bidirectional self-attention captures rich contextual meaning that unidirectional RNN-based models cannot represent.
- Pre-training on large corpora provides strong language priors, enabling good performance with limited labelled data.
- Fine-tuning is fast — typically only 3–5 epochs — compared with training a large model from scratch.
- A single architecture handles many NLP classification tasks without task-specific feature engineering.
- Widely available pre-trained checkpoints (BERT-base, BERT-large) and derivatives for many languages and domains.
Applications
- Sentiment analysis of reviews, social-media posts, and survey responses.
- Topic and intent classification for customer support, chatbots, and content moderation.
- Clinical text classification: diagnosis coding, clinical note triage, and adverse-event detection.
- Scientific abstract categorisation and citation screening in systematic reviews.
- Hate-speech and misinformation detection on social platforms.
How it works
How it works
Input text is split into WordPiece tokens and a special [CLS] token is prepended. A [SEP] token marks sentence boundaries. Each token is mapped to a numerical ID and padded or truncated to a fixed maximum sequence length, typically 128 or 512 tokens.
The token sequence passes through all BERT Transformer layers. Each layer applies multi-head self-attention so every token can attend to every other token in both directions simultaneously, building rich context-aware representations at each position.
Formula
h = \text{BERT}([\texttt{[CLS]}, w_1, \ldots, w_n])The hidden state corresponding to the [CLS] token at the final BERT layer is extracted as a fixed-length vector representing the entire input sequence. This pooled representation captures the overall meaning of the text.
A linear classification head — a weight matrix and bias — maps the [CLS] vector to a score per class. A softmax function converts these scores to class probabilities, and the class with the highest probability is the prediction.
Formula
\hat{y} = \text{softmax}(W h_{\texttt{[CLS]}} + b)The full model — BERT layers and the classification head — is optimised jointly on the labelled training set using cross-entropy loss and a small learning rate (typically 2e-5 to 5e-5 with AdamW) for a small number of epochs (usually 3–5). A learning-rate warm-up schedule prevents large early gradient updates that could destroy pre-trained weights.
Common pitfalls
- Using a learning rate that is too high, which destroys the pre-trained weights and results in no improvement over a random baseline.
- Truncating all inputs without checking whether the most relevant text falls near the beginning or end of the sequence.
- Skipping a proper train/validation/test split and measuring performance only on the training set.
- Ignoring class imbalance — reporting accuracy alone on an unbalanced dataset hides poor minority-class recall.
- Choosing BERT when the deployment environment lacks GPU resources, leading to unacceptable inference latency in production.
Frequently asked questions
How much labelled data do I need to fine-tune BERT?
BERT can yield useful performance with as few as a few hundred labelled examples per class, because pre-training has already instilled rich language priors. Performance improves substantially up to several thousand examples; beyond that, returns diminish. With fewer than roughly 100 examples per class, overfitting is a real risk — consider data augmentation or a lighter model.
What learning rate should I use when fine-tuning?
A learning rate between 2e-5 and 5e-5 with the AdamW optimiser is the standard range recommended by the original paper. Higher learning rates tend to destroy the pre-trained weights; lower ones make convergence very slow. Use a linear warm-up over the first 10% of training steps.
My texts are longer than 512 tokens. What should I do?
Common strategies include truncating to the first or last 512 tokens, splitting the document into overlapping chunks and aggregating the [CLS] representations, or using a long-document model such as Longformer or BigBird that can handle sequences of thousands of tokens.
Should I use BERT-base or BERT-large?
BERT-base (110 M parameters) is the practical default: it trains faster, uses less GPU memory, and its accuracy is sufficient for most classification tasks. BERT-large (340 M parameters) can add a few percentage points on complex tasks but requires substantially more compute and is prone to overfitting on smaller datasets.
How do I evaluate the classifier fairly?
Split data into train, validation, and test sets before any fine-tuning. Report macro F1 (not just accuracy) so that minority-class performance is visible. Use the validation set for hyperparameter search and early stopping, and report the test set result only once at the end.
Related methods
Sources
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of NAACL-HLT 2019 (pp. 4171–4186). Association for Computational Linguistics.
Open DOI referenceCopied.Sun, C., Qiu, X., Xu, Y., & Huang, X. (2019). How to Fine-Tune BERT for Text Classification? In China National Conference on Chinese Computational Linguistics (CCL 2019), Lecture Notes in Computer Science, vol 11856, pp. 194–206. Springer.
Open DOI referenceCopied.