Skip to content
Back to search results

Text Segmentation — Topic-Boundary Detection

Introduced by
Marti A. Hearst (TextTiling)
Year
1997
Type
NLP document-structure / topic-boundary detection
Font size

What does it do?

Text segmentation divides a long document into meaningful sections (segments) along topic or discourse boundaries.

Introduced for subtopic passages by Marti A. Hearst's TextTiling (1997), it supports document-structure analysis and the detection of topic transitions in continuous text.

The intuition behind the method

A long text is rarely about one thing from start to finish — it drifts from subtopic to subtopic. Reading it as one undifferentiated block hides that structure. Text segmentation looks for the places where the vocabulary shifts: where one stretch of words stops being similar to what came before and a new theme begins. By scoring how cohesive neighbouring stretches of text are, it marks those shift points as boundaries and carves the document into coherent passages.

History

Text segmentation as topic-boundary detection was established by Hearst's TextTiling (1997), which scored lexical cohesion between adjacent blocks to locate multi-paragraph subtopic passages. Choi (2000) advanced domain-independent linear segmentation, improving the way cohesion is measured and boundaries are placed in continuous text.

When to use it

Use text segmentation when you have a sufficiently long piece of text and want to expose its internal structure or locate where its topics change. It assumes the text is long enough to segment and that genuine topic transitions exist within it; at least around ten units of text are needed. It fits exploratory and descriptive work on documents, transcripts, and other continuous prose.

Limitations and considerations

  • Requires text that is long enough to segment; very short inputs offer too little signal.
  • Assumes topic transitions actually exist — running it on uniform text yields arbitrary cuts.
  • Cohesion-based boundaries can miss gradual topic drift or over-split on stylistic variation.

Source conditions do not replace independent scientific review or validation by an analysis engine.

Strengths

  • Exposes the latent subtopic structure of a single long document without any labelled training data.
  • Domain-independent: linear segmentation methods work across subject areas because they rely on lexical cohesion rather than topic-specific knowledge.
  • Produces an ordered, human-readable list of passages that downstream summarisation or retrieval can use directly.

Applications

  • Analysing the internal structure of long documents and reports.
  • Locating topic transitions in continuous text such as articles or transcripts.
  • Preparing passage-level units for summarisation and information retrieval.
  • Exploratory and descriptive study of document organisation in the humanities and social sciences.

How it works

How it works

1

The document is split into the small ordered units the method compares — tokens grouped into adjacent windows or candidate-break positions. This sequencing is what lets the later steps reason about what comes before and after each possible boundary.

2

Lexical cohesion is scored along the text: adjacent windows are compared and a similarity (or its inverse, a shift) score is computed at each candidate position. A high shift signals that the vocabulary on either side differs — a sign that the topic may be changing.

3

The shift scores are scanned for their telling dips and peaks; positions where cohesion drops sharply are selected as topic boundaries, while smooth stretches are kept inside a single segment.

4

The chosen boundaries cut the document into an ordered list of segments, each one a multi-sentence passage covering a single subtopic, ready for downstream structure analysis.

Common pitfalls

  • Applying segmentation to text that is too short or thematically uniform produces meaningless boundaries.
  • Treating boundary positions as ground truth without inspecting them; lexical-shift cues can be fooled by repeated vocabulary across genuinely different topics.
  • Confusing paragraph or formatting breaks with topic boundaries — the method detects subtopic shifts, not layout.

Frequently asked questions

How is text segmentation different from paragraph splitting?

Paragraph splitting follows the author's formatting, while text segmentation detects where the subtopic actually changes based on lexical cohesion. A single topic can span several paragraphs, and one paragraph can straddle two topics, so the two need not coincide.

How long does a document need to be?

It must be long enough to contain genuine topic transitions — roughly at least ten units of text. On very short inputs there is too little vocabulary signal for the cohesion scores to mark reliable boundaries.

Does it need labelled training data?

No. Classic linear segmentation such as TextTiling is unsupervised: it scores lexical cohesion within the single document being analysed and needs no labelled boundaries to run.

Will it work outside one specific subject area?

Yes. Domain-independent linear segmentation relies on shifts in word usage rather than topic-specific knowledge, so it transfers across subject areas, though results still depend on the text having real topic transitions.

Related methods

Sources

  1. Hearst, M.A. (1997). TextTiling: Segmenting Text into Multi-Paragraph Subtopic Passages. Computational Linguistics, 23(1), 33-64.

    Open publication reference
    Copied.
  2. Choi, F.Y.Y. (2000). Advances in Domain Independent Linear Text Segmentation. NAACL.

    Open publication reference
    Copied.