Triangulated Delphi Technique
- Introduced by
- Norman Dalkey & Olaf Helmer (Delphi); triangulation principle from Norman Denzin
- Year
- Delphi: 1963; triangulation integration: 1970s–1990s
- Type
- Expert-consensus data collection with multi-method validation
- Family
- Data collection
What does it do?
The Triangulated Delphi Technique combines the structured expert-consensus process of the classic Delphi method with deliberate triangulation — integrating data from at least one additional source or method (e.g., systematic literature review, interviews, survey data) to cross-validate findings and enhance the credibility of expert judgments.
It retains the iterative, anonymous, multi-round panel format while embedding verification steps that reduce reliance on panel consensus alone.
The intuition behind the method
The classic Delphi technique asks a panel of experts to converge on a consensus over multiple rounds. The triangulated variant asks: how confident can we be that the panel's consensus actually reflects reality, not just shared assumptions or group think? By bringing in an independent data source — such as interview findings, a literature synthesis, or a parallel survey — researchers can check whether expert opinion aligns with evidence from elsewhere. Where they agree, confidence rises; where they diverge, the disagreement itself becomes a finding worth investigating. Think of it as a Delphi study with a built-in reality check.
History
The Delphi method was developed at the RAND Corporation in the 1950s and formally published by Norman Dalkey and Olaf Helmer in 1963. It became widely adopted in health, education, and policy research through the 1970s–1990s. Triangulation as a systematic methodological strategy was introduced to social science by Norman Denzin in 1970 and elaborated in successive editions of his handbook. The explicit combination of Delphi with triangulation emerged in the nursing and health research literature from the late 1990s onward, most influentially via guidelines published by Hasson, Keeney, and McKenna (2000).
When to use it
Use the Triangulated Delphi Technique when expert consensus is the primary goal but confidence in panel validity is a concern — particularly when the expert pool may share disciplinary biases, when the literature is contested, or when the research will inform high-stakes policy or clinical decisions. It is especially appropriate in health, education, policy, and technology forecasting contexts where both practitioner wisdom and empirical evidence are needed. Avoid it when the supplementary data source cannot be collected in parallel with the Delphi rounds (logistical mismatch), when the expert panel is very small (fewer than 10), or when time and budget do not permit managing two concurrent data-collection streams. A standard Delphi is preferable if the research goal is purely consensus-building without a validation mandate.
Limitations and considerations
- Substantially more resource-intensive than standard Delphi due to the dual data-collection streams running in parallel.
- Integrating findings from two methodologically different sources requires careful interpretive judgment and may introduce researcher subjectivity.
- Panel fatigue is a known issue in multi-round Delphi studies; adding triangulation steps can increase dropout rates.
- Consensus thresholds and the weight given to supplementary evidence are researcher-defined, meaning methodological choices significantly shape conclusions.
Source conditions do not replace independent scientific review or validation by an analysis engine.
Strengths
- Reduces the risk of consensus reflecting shared bias rather than genuine expert knowledge by cross-checking with independent evidence.
- Produces findings with stronger credibility and transferability, supporting defensible policy or clinical recommendations.
- Retains all advantages of the classic Delphi — anonymity, iterative feedback, controlled opinion exchange — while adding an evidential layer.
- Divergence between expert opinion and supplementary data becomes a productive finding, revealing contested areas for further research.
- Flexible: the triangulation source can be qualitative (interviews, focus groups) or quantitative (survey, secondary data analysis).
Applications
- Developing clinical practice guidelines where expert consensus must be corroborated against systematic review evidence.
- Setting educational standards or curriculum priorities by combining teacher-expert panels with student outcome data.
- Technology forecasting in which specialist opinion is checked against patent or publication trend data.
- Health needs assessment combining expert panel priorities with epidemiological surveillance data.
- Policy Delphi studies in which stakeholder consensus is validated against community survey findings.
How it works
How it works
The researcher defines the phenomenon requiring expert consensus, selects a purposive panel of experts (typically 10–30 participants), and identifies at least one complementary data source or method to serve as the triangulation anchor — for example, a systematic review, focus groups with practitioners, or an existing large-scale survey. The triangulation plan is specified before data collection begins.
An open-ended first-round questionnaire is distributed to the expert panel, inviting unstructured responses about the topic. Responses are thematically analyzed to generate a structured item list for subsequent rounds. At the same time, supplementary data collection from the chosen triangulation source commences so that findings can be compared at a defined checkpoint.
Before or during the second Delphi round, the researcher compares emerging panel themes with findings from the supplementary method. Points of convergence between expert opinion and independent evidence are noted as robustly supported; points of divergence are flagged as areas requiring further panel deliberation or additional evidence-gathering.
Second and subsequent Delphi rounds present panelists with aggregated group statistics (medians, interquartile ranges, percent agreement) alongside a summary of findings from the triangulation source. Panelists are invited to revise their ratings in light of both peer opinion and the independent evidence. Rounds continue until a pre-defined consensus threshold — typically 70–80% agreement within an acceptable response band — is reached or stability is achieved.
Final consensus items are reported together with the triangulation evidence, explicitly documenting where expert opinion was confirmed, challenged, or refined by the independent data. This integrated reporting increases transparency and provides decision-makers with both the expert-derived priorities and the degree to which they are corroborated by other evidence.
Common pitfalls
- Adding a triangulation label without genuine integration — if the supplementary data are simply appended in a discussion section rather than systematically fed back to panelists, the study is a standard Delphi with supplementary evidence, not a truly triangulated design.
- Choosing an incommensurable triangulation source — comparing expert opinion on a normative question (what should be done) with descriptive survey data (what is currently done) conflates different knowledge claims; the supplementary source must address the same question type.
- Ignoring divergence — when expert opinion and supplementary data disagree, resolving the discrepancy requires additional analysis, not simply reporting the consensus and footnoting the conflict.
- Under-reporting the triangulation process — reviewers and readers need full methodological transparency about how the supplementary source was collected, analyzed, and integrated across rounds.
Frequently asked questions
What counts as a valid triangulation source for a Delphi study?
Any independently collected data bearing on the same research question can serve as a triangulation anchor: a systematic literature review, semi-structured interviews with practitioners, an existing large-scale survey, administrative records, or observational data. The key requirement is that the source is methodologically independent from the Delphi panel and collected or synthesized before or alongside the iterative rounds so that findings can inform subsequent panel feedback.
How many Delphi rounds are needed in a triangulated design?
The minimum is two rounds — the first to generate items, the second to rate them with feedback. Most triangulated Delphi studies use two to three rounds. The triangulation comparison typically occurs after round one, with integrated feedback provided in round two. Iteration continues until a consensus threshold (commonly 70–80% agreement within a defined response range) is reached or panel responses stabilize between rounds.
What panel size is appropriate?
The classic Delphi literature suggests 10–30 experts as a workable range; larger panels improve representativeness but increase coordination burden. In triangulated designs, maintaining panel engagement across multiple rounds plus the supplementary data collection makes smaller, carefully selected panels (10–15 experts) more practical. Purposive sampling to ensure expertise coverage is more important than raw panel size.
How do I report divergence between expert consensus and the triangulation source?
Divergence should be reported explicitly rather than minimized. Describe the nature and magnitude of the disagreement, explore possible explanations (disciplinary framing, measurement differences, recency of evidence), and present both the consensus position and the contradictory evidence. This transparency strengthens credibility and identifies genuine knowledge gaps for future research.
Is anonymity still preserved in a triangulated Delphi?
Yes. Panelist anonymity between participants is a defining feature of the Delphi method and is maintained in triangulated designs. The researcher knows individual responses to manage the process, but feedback to panelists presents aggregated statistics, not attributable opinions. The triangulation source is separate and does not compromise panel anonymity.
Related methods
Sources
Dalkey, N., & Helmer, O. (1963). An experimental application of the Delphi method to the use of experts. Management Science, 9(3), 458–467.
Open DOI referenceCopied.Hasson, F., Keeney, S., & McKenna, H. (2000). Research guidelines for the Delphi survey technique. Journal of Advanced Nursing, 32(4), 1008–1015.
Open DOI referenceCopied.