Skip to content
Back to search results

BERT Embeddings — Contextual Text Representations

Introduced by
Devlin, Chang, Lee & Toutanova (Google AI)
Year
2019
Type
Contextual transformer text-representation method
Font size

What does it do?

BERT-based text embeddings, introduced by Devlin and colleagues at Google AI in 2019, turn text into context-sensitive dense vectors using a bidirectional Transformer encoder.

Because the meaning of a word shifts with its context, BERT produces richer representations than static methods such as Word2Vec or topic models like LDA.

The intuition behind the method

A word like "bank" means something different on a river than in finance, yet older embedding methods give it a single fixed vector regardless of where it appears. BERT reads each word together with the words around it, so the same word receives a different vector depending on its sentence. This contextual reading is what lets the embeddings capture meaning that static word vectors miss.

History

BERT was introduced by Devlin, Chang, Lee and Toutanova (2019), who pretrained a deep bidirectional Transformer on large text corpora and showed strong results across many language-understanding tasks. Later probing work by Tenney and colleagues (2019) showed that BERT internally rediscovers steps of the classical NLP pipeline, helping explain why its contextual embeddings carry such rich linguistic information.

When to use it

BERT embeddings fit when you have text data and need representations that respect context, and when a pretrained model matches the language and domain of that corpus. They suit explanatory, classification, and relationship purposes. With fewer than about 100 documents, fine-tuning is not feasible — prefer static pretrained embeddings such as GloVe — and if there is no text data at all, BERT embeddings cannot be produced.

Limitations and considerations

  • A suitable model must be selected — multilingual or domain-specific — to match the corpus.
  • A GPU is recommended for large datasets because encoding is computationally expensive.
  • Fine-tuning is not feasible on very small corpora (under ~100 documents).
  • Embeddings are high-dimensional and less directly interpretable than frequency- or lexicon-based features.

Source conditions do not replace independent scientific review or validation by an analysis engine.

Strengths

  • Context-sensitive: the same word gets different vectors depending on its surrounding text.
  • Richer representations than static methods such as Word2Vec or topic models like LDA.
  • Bidirectional encoding reads each token in light of the full context on both sides.
  • Pretrained models cover many languages and specialised domains, so analysis can start without training from scratch.

Applications

  • Text classification using context-aware document representations.
  • Measuring semantic similarity between documents or passages.
  • Relationship and explanatory analysis over text where word meaning depends on context.
  • Producing features for downstream NLP models in health, social science, economics, and business research.

How it works

How it works

1

An appropriate pretrained BERT model is chosen to match the corpus — a multilingual model for non-English or mixed-language text, or a domain-specific model when the text comes from a specialised field. Picking a model that matches the language and domain of the corpus is the decision that most shapes embedding quality.

2

Each document is split into the sub-word tokens the chosen model expects. Tokenisation breaks words into the units the model was pretrained on, so that even unfamiliar or compound words can be represented.

3

The bidirectional Transformer encoder passes over the tokens and produces a context-sensitive vector for each one, reading every token in light of the surrounding text. The same word yields different vectors in different contexts, which is what gives the representation its richness.

4

The resulting embeddings feed downstream tasks — classification, similarity, or relationship analysis — and on larger corpora the model can be fine-tuned to the task. A GPU is recommended for large datasets because encoding many documents is computationally heavy.

Common pitfalls

  • Applying an English model to text in another language degrades quality; choose a multilingual or matching model.
  • Fine-tuning BERT on a small dataset (under ~100 documents) does not work — fall back to static pretrained embeddings such as GloVe.
  • With no text data at all, BERT embeddings cannot be generated — consider clustering on structured data instead.
  • Encoding large corpora without a GPU can be prohibitively slow.

Frequently asked questions

How is BERT different from Word2Vec or GloVe?

Word2Vec and GloVe assign each word a single static vector regardless of context. BERT is contextual: it reads each word together with its surrounding text and produces a different vector for the same word in different sentences, which captures meaning that static embeddings miss.

How much text do I need?

Generating embeddings from a pretrained model works even on small corpora (around ten documents and up). Fine-tuning the model to your task, however, needs a larger set — with fewer than about 100 documents, prefer static pretrained embeddings such as GloVe instead.

Do I need a GPU?

Not for small datasets, but a GPU is recommended for large ones because encoding many documents through a Transformer is computationally heavy and can be very slow on a CPU.

Can BERT handle languages other than English?

Yes, provided you select a model that matches — a multilingual model, or a model trained for the specific language or domain of your corpus. Applying an English-only model to other-language text is a common cause of poor results.

Related methods

Sources

  1. Devlin, J., Chang, M.-W., Lee, K. & Toutanova, K. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. NAACL-HLT, 4171-4186.

    Open DOI reference
    Copied.
  2. Tenney, I., Das, D. & Pavlick, E. (2019). BERT Rediscovers the Classical NLP Pipeline. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL), 4593-4601.

    Open DOI reference
    Copied.