Skip to content
Back to search results

Delphi Method — Structured Expert Consensus

Introduced by
Norman Dalkey & Olaf Helmer (RAND Corporation)
Year
1963
Type
Structured iterative expert-elicitation process
Font size

What does it do?

The Delphi method is a structured, iterative survey technique developed by Norman Dalkey and Olaf Helmer at the RAND Corporation in 1963 for eliciting and converging expert opinion on complex topics where empirical data are unavailable or insufficient.

It collects independent judgements from a geographically dispersed expert panel over multiple anonymous rounds, feeding aggregated results back to participants after each round so they can revise their views in light of the group's collective position.

The intuition behind the method

Imagine asking a group of specialists to forecast something uncertain — say, which medical treatments will be standard practice in ten years. If they discuss it openly, dominant voices can distort the outcome. The Delphi method solves this by keeping respondents anonymous and anonymous from each other throughout. After each round of questionnaires, every expert sees only the statistical summary of the group's answers, not who said what. That controlled feedback loop lets individuals reconsider their positions without social pressure, gradually pulling the group toward a well-founded consensus.

History

The Delphi method was developed at the RAND Corporation in the early 1950s and first formally described by Norman Dalkey and Olaf Helmer in their 1963 Management Science article on using structured expert elicitation for technological forecasting. The name references the Oracle at Delphi in ancient Greece. Through the 1970s and 1980s the method spread from defence and technology forecasting into health policy, education, and the social sciences. A modified variant — the policy Delphi — was introduced to deliberately surface disagreement rather than convergence. The method remains widely used in clinical guideline development, health technology assessment, and strategic foresight.

When to use it

Use the Delphi method when empirical data are scarce or unavailable and expert opinion is the most defensible evidence base; when you need structured forecasts, priority lists, or criterion weighting for policy, clinical guideline, or strategic planning purposes; and when participants are dispersed and cannot meet face-to-face. The minimum viable panel is 10 experts, and at least two questionnaire rounds must be planned. If the question can be answered with existing quantitative data, a statistical analysis is more appropriate. If you need real-time deliberation among a small, co-located group, a focus group or nominal group technique is a better fit.

Limitations and considerations

  • Requires a minimum panel of 10-15 experts; with fewer participants, consensus statistics become unreliable.
  • The process is slow: multiple questionnaire rounds with inter-round analysis can span weeks or months.
  • The quality of consensus depends entirely on the quality of the expert panel; a poorly recruited panel produces poorly grounded results.
  • Pre-defining the consensus threshold is methodologically important but inherently arbitrary; different thresholds can yield different 'consensus' outcomes.
  • Response attrition across rounds is common; panellists who drop out may bias the final consensus.

Source conditions do not replace independent scientific review or validation by an analysis engine.

Strengths

  • Anonymity neutralises social dominance and authority bias, allowing quieter experts to contribute on equal footing.
  • Controlled, structured feedback guides the panel toward reasoned convergence rather than group-think conformity.
  • Participants can be geographically dispersed, making it practical when assembling a face-to-face panel is impractical or costly.
  • The iterative structure is transparent and auditable: each round's data can be reported, showing how views shifted.
  • Works well for priority-setting and criterion-weighting tasks where no gold-standard empirical answer exists.

Applications

  • Developing clinical practice guidelines and health technology assessment criteria where randomised evidence is absent.
  • Strategic foresight and technology forecasting in policy and industry planning.
  • Curriculum design and competency-framework development in higher education.
  • Priority-setting exercises in public health resource allocation.
  • Weighting criteria in multi-criteria decision analysis when empirical weights are unavailable.

How it works

How it works

1

A panel of 10-15 domain experts is recruited. Panellists must genuinely hold relevant expertise; diversity across sub-disciplines, institutions, or geographic regions strengthens the result. Anonymity is guaranteed from this point onward — participants never learn who else is in the panel or which individual gave which response.

2

An initial open-ended or structured questionnaire is distributed. In the first round, questions are often broad to surface the range of relevant issues. Responses are collected, coded, and analysed — typically using descriptive statistics (medians, interquartile ranges) for scaled items and thematic grouping for open text.

3

Anonymised, aggregated results from round one are shared with all panellists. Each expert sees where their own response sits relative to the group distribution — typically expressed as the group median and IQR — without knowing individual attributions. A minority whose view diverges substantially from the median is invited to provide a written rationale.

4

A second questionnaire (and a third if needed) is sent, incorporating the aggregated feedback and, where applicable, minority rationales. Panellists reconsider their positions in light of the group summary. The process continues until a pre-defined consensus threshold is reached — commonly defined as an IQR of 1 or less on a five- or seven-point Likert scale, or as 70-80 percent of panellists falling within a specified range.

5

When the consensus criterion is met, or after a predetermined maximum number of rounds, the process closes. Final outputs include the consensus ratings or rankings, the degree of agreement (IQR, percentage agreement, content validity ratio), and a summary of persisting minority views. These outputs feed directly into the policy, clinical, or strategic decision being addressed.

Common pitfalls

  • Defining the consensus threshold after the data are collected — always specify the IQR or percentage-agreement criterion in the study protocol before data collection begins.
  • Treating any level of agreement as consensus without reporting the IQR or percentage agreement; always publish the dispersion statistic alongside the median.
  • Running only a single questionnaire round — the method's iterative feedback loop is constitutive, not optional; a single round is a survey, not a Delphi.
  • Recruiting fewer than 10 experts — below this threshold the distribution statistics lose meaning and a focus group is a more appropriate design.
  • Assuming the method is purely qualitative — Delphi produces ordinal ratings that require descriptive and sometimes inferential statistical analysis across rounds.

Frequently asked questions

How many rounds does a Delphi study need?

At least two rounds are required — one to elicit initial positions and one to allow revision in light of group feedback. Most published Delphi studies use two or three rounds. A third round is warranted when consensus has not been reached after round two and the IQR has been narrowing; beyond three rounds, attrition risk typically outweighs the benefit of continued iteration.

What panel size is appropriate?

The source literature sets a practical minimum of 10-15 experts. A panel below 10 produces unreliable distribution statistics and the method loses its inferential basis. There is no fixed upper bound, but panels above 30 generate administrative complexity with diminishing returns on consensus quality. Panel diversity — across sub-disciplines and institutional contexts — matters more than sheer size.

How is consensus defined?

The most common operational definition is an interquartile range (IQR) of 1 or less on a five- or seven-point Likert scale, or 70-80 percent of panellists rating within a specified range. Crucially, the threshold must be defined in the study protocol before data collection, not selected post hoc to fit the results.

When should I use a focus group instead of a Delphi study?

If your expert panel is small (under 10 people), co-located, and real-time deliberation is feasible, a focus group or nominal group technique is more appropriate. The Delphi method's advantages — anonymity and geographic flexibility — only materialise with a panel large enough to make distribution statistics meaningful and dispersed enough to prevent face-to-face assembly.

Related methods

Sources

  1. Dalkey, N. & Helmer, O. (1963). An Experimental Application of the Delphi Method to the Use of Experts. Management Science, 9(3), 458-467.

    Open DOI reference
    Copied.