Skip to content
Back to search results

Sentence Embeddings

Introduced by
Kiros et al. (Skip-Thought, 2015); Reimers & Gurevych (Sentence-BERT, 2019)
Year
2015–2019
Type
Representation learning / embedding
Family
Deep learning / NLP / CV
Font size

What does it do?

Sentence Embeddings convert a sentence or short text into a single fixed-length dense vector that captures its semantic meaning.

These vectors allow downstream tasks — semantic similarity, clustering, retrieval, and classification — to operate on numerical representations instead of raw text, making them one of the most versatile building blocks in modern NLP pipelines.

The intuition behind the method

Imagine every sentence as a point in a high-dimensional space where meaning determines proximity: two sentences that say the same thing in different words land very close together, while unrelated sentences sit far apart. A sentence embedding model learns this geometry from large corpora. Once the model is trained, embedding a new sentence is a single forward pass — no further training required — and any downstream task simply works with these coordinates. This reuse of a pre-trained geometry is what makes sentence embeddings powerful: you can compare, cluster, or classify millions of texts without task-specific labels.

History

Early sentence representations used averaged word vectors (word2vec, GloVe). Skip-Thought Vectors (Kiros et al., 2015) trained an encoder-decoder to predict surrounding sentences, producing the first deep contextual sentence embeddings. InferSent (Conneau et al., 2017) fine-tuned on NLI pairs and showed strong transfer. Sentence-BERT (Reimers & Gurevych, 2019) applied a Siamese BERT architecture to NLI data and dramatically improved the quality of BERT-derived embeddings while cutting similarity-computation time from 65 hours to about 5 seconds for 10,000 sentence pairs.

When to use it

Use sentence embeddings when you need to measure semantic similarity, cluster documents, perform dense retrieval, or feed text into a classifier without task-specific labels for every example. They excel when you have moderate to large amounts of text and want a general-purpose representation that transfers across tasks. Do not use raw sentence embeddings — without fine-tuning or a task head — when you need calibrated probabilities or interpretable feature weights. They are also a poor fit for very short texts (one or two words), for languages with no pre-trained encoder available, or when computational resources are heavily constrained, since encoding millions of long documents can be expensive.

Limitations and considerations

  • Embedding quality depends heavily on the quality and domain of the pre-trained model; out-of-domain text degrades performance.
  • Representing very long documents as a single fixed vector loses fine-grained local information.
  • Cosine similarity thresholds for downstream decisions must be calibrated empirically and are not universal.
  • Large encoder models require GPU memory; storing embeddings for millions of documents requires significant disk space.

Source conditions do not replace independent scientific review or validation by an analysis engine.

Strengths

  • Capture rich semantic meaning that bag-of-words and TF-IDF miss, such as synonymy and paraphrase.
  • Pre-trained models transfer out of the box to new domains with little or no labelled data.
  • Fast at inference: once encoded, similarity search on millions of vectors takes milliseconds with FAISS-style indices.
  • Versatile: the same embeddings serve clustering, retrieval, semantic deduplication, and classification.
  • Multilingual variants (e.g., paraphrase-multilingual-mpnet) handle cross-lingual similarity with a single model.

Applications

  • Semantic textual similarity and paraphrase detection in academic and commercial text corpora.
  • Dense passage retrieval in open-domain question answering and search engines.
  • Document and paragraph clustering for topic discovery without predefined categories.
  • Zero-shot and few-shot text classification by comparing embeddings to label prototypes.
  • Near-duplicate detection and de-duplication in large bibliographic databases.
  • Cross-lingual semantic alignment in multilingual research synthesis.

How it works

How it works

1

The input sentence is split into tokens (subwords or words) according to the encoder vocabulary — typically a BERT-style WordPiece or SentencePiece tokeniser. Special tokens such as [CLS] and [SEP] are inserted to mark sentence boundaries, preparing the sequence for the encoder.

2

A deep encoder — most commonly a pre-trained Transformer such as BERT or RoBERTa — processes the token sequence and produces a contextual hidden state for every position. Each hidden state carries information about its token in the context of the entire sentence.

Formula
\mathbf{s} = \text{Pooling}(\text{Encoder}(w_1, \ldots, w_n)) \in \mathbb{R}^d
3

The per-token hidden states are collapsed into one fixed-size vector through a pooling operation. Mean-pooling (averaging all token vectors) and CLS-token extraction are the two most common choices. Sentence-BERT showed that mean-pooling over all token states consistently outperforms the raw CLS vector.

4

For high-quality embeddings the encoder is fine-tuned with a contrastive or semantic-similarity objective. Sentence-BERT uses a Siamese network trained on natural-language inference pairs, pushing semantically similar sentences together and dissimilar ones apart in the embedding space.

5

At inference time, two sentence vectors are compared with cosine similarity: the dot product of the unit-normalised vectors. Scores near 1 indicate semantic equivalence; scores near 0 indicate orthogonality. This single number drives retrieval, de-duplication, clustering, and classification tasks.

Formula
\text{sim}(\mathbf{s}_i, \mathbf{s}_j) = \frac{\mathbf{s}_i \cdot \mathbf{s}_j}{\|\mathbf{s}_i\|\|\mathbf{s}_j\|}

Common pitfalls

  • Using the raw [CLS] token from a base BERT model as a sentence embedding without fine-tuning — this produces poor-quality, non-isotropic embeddings.
  • Comparing embeddings produced by different encoder architectures or pooling strategies, which makes cosine scores incomparable.
  • Picking a similarity threshold (e.g., 0.85 means 'duplicate') without calibrating it on held-out data from your domain.
  • Ignoring the need to normalise vectors before computing cosine similarity when using dot-product-only search indices.
  • Treating the embedding as interpretable: individual dimensions carry no semantic meaning on their own.

Frequently asked questions

Which pre-trained model should I use?

For English general-purpose similarity, 'all-mpnet-base-v2' from the sentence-transformers library is a strong default. For multilingual tasks, 'paraphrase-multilingual-mpnet-base-v2' covers over 50 languages. Always benchmark on a small sample from your target domain before committing to a model.

How is this different from using BERT directly?

Standard BERT produces per-token representations optimised for token-level tasks. Using its raw output (e.g., the [CLS] vector) as a sentence embedding yields poor cosine-similarity results. Sentence-BERT fine-tunes BERT with a contrastive objective so the resulting vectors are explicitly structured for sentence-level comparison.

How many labelled examples do I need?

Off-the-shelf sentence embedding models from the sentence-transformers library require zero labelled examples for inference. If you fine-tune on your domain, even a few hundred sentence pairs with similarity labels can noticeably improve performance.

Can I use sentence embeddings for long documents?

Most encoders have a token limit (512 tokens for BERT-based models). For longer texts you can chunk the document into sentences or paragraphs, embed each chunk, and pool the chunk vectors. Alternatively, models like Longformer handle longer contexts natively.

How do I evaluate the quality of my embeddings?

Standard benchmarks include STS-B (Semantic Textual Similarity Benchmark) and MTEB (Massive Text Embedding Benchmark). On your own data, human-annotated sentence pairs with similarity scores let you compute Spearman correlation between model scores and human judgements.

Related methods

Sources

  1. Reimers, N., & Gurevych, I. (2019). Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing (EMNLP), 3980–3990.

    Open DOI reference
    Copied.
  2. Kiros, R., Zhu, Y., Salakhutdinov, R., Zemel, R. S., Torralba, A., Urtasun, R., & Fidler, S. (2015). Skip-Thought Vectors. Advances in Neural Information Processing Systems (NeurIPS), 28.

    Open publication reference
    Copied.