Skip to content
Back to search results

Word2Vec — Word Embeddings

Introduced by
Tomas Mikolov et al.
Year
2013
Type
Neural word-embedding model
Font size

What does it do?

Word2Vec is a neural word-embedding technique introduced by Mikolov and colleagues in 2013 that maps each word in a text corpus to a dense numeric vector.

Words that appear in similar contexts end up close together in the vector space, so the embeddings capture semantic similarity that can be measured arithmetically.

The intuition behind the method

A computer cannot compare the meaning of words directly, but it can learn from the company words keep. Word2Vec reads a large corpus and notices which words tend to surround which other words. Words used in similar contexts get similar vectors, so 'king' lands near 'queen' and far from 'banana'. Meaning is encoded as position in space, and similarity becomes a distance you can compute.

History

Word2Vec was introduced by Mikolov and colleagues in 2013 as an efficient way to estimate word representations in vector space, offering the skip-gram and continuous bag-of-words architectures. It popularised dense distributed word embeddings and was followed by related approaches such as GloVe and, later, contextual embeddings.

When to use it

Use Word2Vec when you have a large unlabelled text corpus and want dense vector representations that capture semantic similarity between words. It suits exploratory and relational analysis of text. A reasonably large corpus (roughly a hundred documents or more, ideally far larger) is required; on small corpora the model cannot learn meaningful embeddings and pretrained vectors such as GloVe are the better choice. With no text data at all, word embeddings cannot be built.

Limitations and considerations

  • Needs a large text corpus; small corpora yield unstable, meaningless embeddings.
  • Requires careful text preprocessing before training.
  • Assigns one vector per word, so it cannot distinguish different senses of the same word from context.

Source conditions do not replace independent scientific review or validation by an analysis engine.

Strengths

  • Captures semantic similarity so related words sit close together in vector space.
  • Produces compact dense vectors that serve as numeric features for downstream text tasks.
  • Learns from unlabelled text alone — no manual annotation is needed.

Applications

  • Measuring semantic similarity between words for search and recommendation.
  • Generating numeric word features that feed text clustering and classification.
  • Exploring relationships and analogies within a large text corpus.
  • Building lexical resources from domain-specific corpora in research and industry.

How it works

How it works

1

A large text corpus is cleaned and tokenised into a sequence of words. Word2Vec needs a sizeable corpus — at least a few thousand sentences — because the embeddings are learned purely from co-occurrence patterns, and too little text gives unstable, meaningless vectors.

2

Each word is paired with the words that surround it within a sliding context window. These (target word, context word) pairs are the training signal: they encode which words appear together and form the basis for learning that contextually similar words should have similar vectors.

3

A shallow neural network is trained on the context pairs, using either the skip-gram architecture (predict surrounding words from a target word) or continuous bag-of-words (predict a target word from its surroundings). The network's weights are adjusted until words in similar contexts receive similar vectors.

4

After training, each word is represented by its learned dense vector — commonly 100 to 300 dimensions. These embeddings can then be compared with cosine similarity to find related words, or used as numeric features feeding downstream tasks such as clustering or classification.

Common pitfalls

  • Training on a corpus that is too small (under ~100 documents) produces noise rather than meaning — prefer pretrained GloVe vectors instead.
  • Skipping preprocessing leaves noisy tokens that pollute the learned vectors.
  • Trying to build embeddings without any text data — Word2Vec has nothing to learn from; cluster structured data instead.
  • Treating a single static vector per word as if it captured every sense of that word.

Frequently asked questions

What is the difference between skip-gram and CBOW?

Both are shallow neural architectures for learning embeddings. Skip-gram predicts the surrounding context words from a given target word, while continuous bag-of-words (CBOW) predicts the target word from its surrounding context. They produce comparable embeddings, with skip-gram often working better on smaller corpora and rare words.

How large does my corpus need to be?

Word2Vec learns purely from co-occurrence, so it needs a sizeable corpus. On very small text (under about a hundred documents) it cannot learn meaningful vectors; in that case pretrained embeddings such as GloVe are the safer choice.

How do I measure similarity between two words?

Compare their learned vectors with cosine similarity. Words used in similar contexts get similar vectors, so a high cosine similarity indicates semantically related words.

Can Word2Vec tell apart different meanings of the same word?

No. It assigns a single static vector per word, blending all senses together. Distinguishing word senses from context requires contextual embedding models rather than Word2Vec.

Related methods

Sources

  1. Mikolov, T., Chen, K., Corrado, G. & Dean, J. (2013). Efficient Estimation of Word Representations in Vector Space.

    Open publication reference
    Copied.