Skip to content
Back to search results

Theory-Based Impact Evaluation

Introduced by
Carol Weiss; Howard White (3ie)
Year
2009
Type
Evaluation approach / framework
Family
Theory-driven evaluation
Font size

What does it do?

Theory-based impact evaluation evaluates a programme by first making explicit the theory of change — the causal chain of assumptions and mechanisms through which inputs are expected to produce outcomes and impacts — and then gathering evidence to test whether each link in that chain holds.

Rather than treating the programme as a black box and estimating only the net effect, it asks not just whether a programme worked but why, for whom, and under what conditions. Articulated by Carol Weiss and brought into development practice by Howard White and 3ie, it complements, rather than competes with, counterfactual designs.

The intuition behind the method

A counterfactual estimate can tell you a programme raised test scores by ten percent, but it leaves a black box: was it the textbooks, the teacher training, the meals, or the parental committees? Theory-based evaluation opens the box. It draws the programme's logic as an explicit map — if we train teachers, then teaching improves, then learning rises — surfaces the assumptions at each arrow, and then checks the evidence link by link. If learning rose, did teaching actually improve? Did teachers attend the training? By tracing the chain, the evaluator can distinguish a programme that failed in design from one that failed in implementation, and can credibly attribute (or rule out) the programme's contribution to observed change.

History

Theory-driven evaluation traces to the program-theory tradition of Carol Weiss, Huey-Tsyh Chen, and Peter Rossi from the 1970s onwards; Weiss's 1997 essay crystallised the case for grounding evaluation in explicit theories of how programmes are supposed to work. In development, Howard White — then at the World Bank's evaluation group and later founding executive director of the International Initiative for Impact Evaluation (3ie) — set out the principles and practice in 2009, arguing that theory-based and counterfactual methods should be combined rather than opposed. Allied strands include John Mayne's contribution analysis and process tracing imported from political science, and the approach now underpins much of 3ie's and major donors' evaluation guidance.

When to use it

Use theory-based impact evaluation when you need to understand the mechanism behind a result — why and how a programme works — not merely whether it worked, and especially for complex, multi-component, or context-dependent interventions where a single counterfactual estimate would be uninformative. It is well suited to programmes that cannot be randomised, to explaining heterogeneous or null results, and to drawing transferable lessons. It is weaker as a stand-alone basis for net-effect attribution where a credible counterfactual is feasible; its rigour depends on the quality of the underlying theory and evidence, and a poorly specified theory of change yields a weak evaluation.

Limitations and considerations

  • On its own it provides weaker net-effect attribution than a credible counterfactual design when one is available.
  • Quality depends heavily on the soundness of the articulated theory of change; a weak or biased theory yields a weak evaluation.
  • Judgements of contribution can be subjective and harder to standardise or replicate than a quantitative effect estimate.
  • Mapping and testing a full causal chain with mixed methods can be data-intensive, time-consuming, and demanding of evaluator skill.

Source conditions do not replace independent scientific review or validation by an analysis engine.

Strengths

  • Explains the mechanism — why, how, and for whom a programme works — rather than treating impact as a black box.
  • Helps distinguish theory failure (a flawed programme design) from implementation failure (a sound design poorly executed).
  • Applicable to complex, multi-component, and non-randomisable interventions where pure counterfactual designs are infeasible.
  • Supports transferable learning and policy uptake by clarifying which mechanisms and conditions drive results.

Applications

  • Evaluating complex multi-component programmes (e.g., integrated nutrition or community-driven development) where a single effect estimate is uninformative.
  • Explaining why a rigorously measured impact was large, small, or null by tracing which causal links held.
  • Assessing governance, policy-influence, and advocacy interventions that cannot be randomised.
  • Strengthening randomized evaluations by specifying and testing the mechanisms behind the measured effect.
  • Building contribution stories for donor accountability where multiple actors and factors shape the outcome.

How it works

How it works

1

The evaluation begins by reconstructing the programme's theory of change with stakeholders: the full causal chain from inputs and activities through outputs to intended outcomes and impacts, including the contextual factors and the behavioural responses assumed at each step. This map — often more detailed than a logframe — makes explicit the mechanisms the programme relies on and defines the hypotheses the evaluation will test.

2

Each link in the chain rests on assumptions — that beneficiaries will participate, that a behaviour will change, that a market will respond, that no rival cause is at work. The evaluator surfaces these assumptions and the alternative explanations that could account for any observed change, identifying which links are most uncertain or most critical and therefore most in need of evidence. This step turns a diagram into a testable set of causal claims.

3

Evidence is assembled to test each link using whatever method suits it: quantitative impact estimates (including experimental or quasi-experimental designs) for net effects, administrative and monitoring data for outputs and uptake, and qualitative methods — interviews, process tracing, case studies — for mechanisms, context, and the plausibility of alternative explanations. Triangulating mixed methods across the chain is the heart of the approach.

4

The evaluator weighs the assembled evidence to reach a judgement: does the chain hold link by link, are rival explanations ruled out, and can the programme credibly be said to have contributed to the observed outcomes? Contribution analysis builds this into a structured argument — a 'contribution story' — acknowledging that in complex settings the aim is a defensible claim of contribution rather than a single attributable effect size.

Common pitfalls

  • Drawing a theory of change as a one-off planning ritual and never returning to test its assumptions against evidence.
  • Confirmation bias — assembling evidence that supports the programme's logic while neglecting rival explanations.
  • Treating a logframe as a substitute for a genuine theory of change, omitting the mechanisms and behavioural assumptions.
  • Claiming attribution from a plausible contribution story without seriously testing and ruling out alternative causes.

Frequently asked questions

How does theory-based impact evaluation differ from a randomized controlled trial?

A randomized controlled trial estimates the net effect of a programme by comparing randomly assigned treatment and control groups, answering whether it worked but treating the programme as a black box. Theory-based impact evaluation instead specifies the causal chain and tests each link with mixed evidence, answering why and how it worked. The two are complementary: the strongest evaluations embed an explicit theory of change within an experimental or quasi-experimental design, so that the net effect and the mechanism behind it are estimated together.

What is contribution analysis?

Contribution analysis, developed by John Mayne, is a structured approach to causal inference where attribution is impossible or impractical. It builds a postulated theory of change, gathers evidence on whether each link occurred and whether alternative explanations can be discounted, and assembles a reasoned 'contribution story' that judges how plausibly the intervention contributed to the observed result. Rather than claiming the programme caused an exact effect size, it argues that, given the evidence and the ruling-out of rival causes, the programme made a credible difference.

Is a theory of change the same as a logframe?

No. A logical framework (logframe) is a management matrix linking inputs, activities, outputs, outcomes, and indicators, often in a linear table. A theory of change is richer: it makes explicit the causal mechanisms, behavioural responses, contextual conditions, and assumptions that connect each level, including feedback loops and alternative pathways. A theory of change can underpin a logframe, but a logframe alone usually omits the why — the mechanisms and assumptions — that theory-based evaluation needs to test.

Related methods

Sources

  1. White, H. (2009). Theory-Based Impact Evaluation: Principles and Practice. Journal of Development Effectiveness, 1(3), 271–284.

    Open DOI reference
    Copied.
  2. Weiss, C. H. (1997). Theory-based evaluation: Past, present, and future. New Directions for Evaluation, 1997(76), 41–55.

    Open DOI reference
    Copied.