Skip to content
Back to search results

Realist Evaluation

Introduced by
Ray Pawson & Nick Tilley
Year
1997
Type
Theory-driven, generative evaluation approach
Family
Program evaluation methodology
Font size

What does it do?

Realist evaluation is a theory-driven approach to evaluating programs and policies that asks not simply 'does it work?' but 'what works, for whom, in what circumstances, and why?'.

Developed by Ray Pawson and Nick Tilley in their 1997 book Realistic Evaluation, it treats interventions as theories incarnate: programs offer resources or opportunities that trigger underlying mechanisms of reasoning and response in participants, and those mechanisms only fire in particular contexts. The unit of analysis is the Context-Mechanism-Outcome (CMO) configuration, and the goal is to build and refine middle-range theory that explains differential outcomes across settings.

The intuition behind the method

Imagine a policy such as a CCTV camera in a car park reduces theft. A black-box experiment might tell you crime fell by 20 percent on average, but not why, nor whether it would fall in a different car park. Realist evaluation insists the camera itself causes nothing; what matters is the mechanism it triggers in people's reasoning — would-be offenders deciding the risk of being caught is now too high, or motivated guardians being alerted, or drivers feeling safer and using the car park more. Each mechanism only operates in a supportive context: a well-lit, monitored, low-volume car park versus a crowded, unmonitored one. By pairing the right mechanism with the right context, realist evaluation explains the pattern of wins and failures rather than averaging them away.

History

Realist evaluation grew out of the scientific-realist philosophy of science and dissatisfaction with both positivist experimental evaluation and relativist constructivist evaluation. Ray Pawson and Nick Tilley set it out in their 1997 book Realistic Evaluation, building on earlier criticism that the black-box randomised trial could establish that a program worked but never why. The approach was rapidly taken up in criminology, public health and international development, and Pawson later extended the logic to evidence review in his 2006 Evidence-Based Policy and to realist synthesis. The RAMESES projects subsequently published reporting and quality standards that consolidated realist evaluation as a recognised methodology.

When to use it

Use realist evaluation when an intervention is complex, its effects vary markedly across sites or subgroups, and decision-makers need to understand the conditions under which it succeeds or fails rather than just an average impact. It is well suited to social programs, public-health initiatives, organisational reforms and policies implemented in heterogeneous settings where the same intervention plausibly works through different mechanisms. It is less appropriate when the question genuinely is a clean average treatment effect of a simple, standardised intervention, or when no articulable program theory exists. It complements rather than replaces counterfactual designs: an experiment may estimate whether an effect exists, while realist evaluation explains why and where.

Limitations and considerations

  • Demanding to conduct well: specifying, testing and refining multiple CMO configurations requires substantial time, methodological skill and rich data.
  • Mechanisms are unobservable generative processes that must be inferred, leaving room for divergent interpretation and weakening claims to objectivity.
  • Provides no standardised effect size or confidence interval, which can frustrate audiences expecting quantified, comparable impact estimates.
  • Quality is uneven across published realist evaluations, and the boundary between context and mechanism is contested and frequently applied inconsistently.

Source conditions do not replace independent scientific review or validation by an analysis engine.

Strengths

  • Explains heterogeneity of effects by design, identifying for whom and in what circumstances a program works rather than discarding variation as noise.
  • Produces transferable middle-range theory, so lessons can be applied to new programs sharing the same mechanisms rather than being locked to one trial site.
  • Method-neutral and explicitly mixed-methods, integrating quantitative outcome data with qualitative evidence on mechanisms within a single explanatory frame.
  • Engages stakeholders' own theories, improving the relevance and uptake of findings among practitioners and policymakers.

Applications

  • Evaluating crime-prevention measures such as CCTV, neighbourhood watch and offender programs, where effects depend heavily on local context.
  • Public-health and health-services research on complex interventions like care pathways, screening programs and behaviour-change initiatives.
  • International development and aid programs implemented across diverse country and community settings with variable results.
  • Organisational and workforce reforms, leadership development and quality-improvement initiatives where uptake varies by setting.
  • Education and social-policy programs delivered to heterogeneous populations where designers need to know which subgroups benefit and why.

How it works

How it works

1

The evaluator begins by surfacing the program's underlying theory — the often-implicit assumptions of designers and practitioners about how the intervention is supposed to produce change. This is articulated as a set of candidate explanations rather than a single logic model, drawing on documents, stakeholder interviews, and prior literature to map the hypothesised pathways from program resources to participant responses to outcomes.

2

Each candidate explanation is formalised as a Context-Mechanism-Outcome configuration: a proposition stating that in context C, the program activates mechanism M, which produces outcome O. The mechanism is the generative engine — typically a change in participants' reasoning, motivation or capacity — not the program activity itself. Multiple competing or complementary CMO configurations are specified for the same program.

Formula
\text{Outcome} = f(\text{Mechanism} \mid \text{Context})
3

Mixed-methods data are collected to test the CMO propositions, with the choice of method dictated by the question rather than by paradigm loyalty. Quantitative data establish outcome patterns and subgroup variation; qualitative data probe whether the hypothesised mechanism actually fired and which contextual features enabled or blocked it. Data collection deliberately targets contrasts across contexts that should, in theory, produce different outcomes.

4

Evidence is used to confirm, refute or refine each CMO configuration. Where outcomes diverge from theory, the configuration is revised — perhaps the mechanism is different, or an unrecognised contextual condition is at work. This is an iterative, retroductive process of moving between data and theory, progressively sharpening the account of for-whom-and-where the program works.

5

The refined CMO configurations are synthesised into transferable middle-range theory: an explanation pitched between abstract grand theory and one-off findings that can travel to other programs sharing the same mechanisms. This cumulative theory, rather than a single effect size, is the deliverable that informs policy decisions about where and how to deploy an intervention.

Common pitfalls

  • Confusing the program activity (e.g., 'a mentoring session') with the mechanism (the change in a participant's reasoning or confidence it triggers), which collapses the whole logic.
  • Treating context as mere background setting rather than as the specific conditions that activate or suppress a mechanism.
  • Specifying a single CMO configuration and ignoring competing explanations, turning a realist study into a thinly disguised confirmation of the official program theory.
  • Collecting data without a prior CMO framework, then retrofitting configurations to whatever was found, which forfeits the theory-testing rigour the approach depends on.

Frequently asked questions

What exactly is a mechanism in realist evaluation?

A mechanism is the underlying generative process through which a program produces its outcomes — usually a change in participants' reasoning, motivation, capacity or resources, not the program activity itself. For example, a job-training subsidy is the resource; the mechanism is the shift in an employer's calculation that hiring is now worthwhile, or in a jobseeker's sense of self-efficacy. Mechanisms are not directly observable and must be inferred from evidence, which is why specifying them carefully is the core analytic task.

How does realist evaluation differ from a theory of change or logic model?

A logic model or theory of change typically lays out a single linear chain from inputs to outcomes. Realist evaluation instead works with multiple Context-Mechanism-Outcome configurations and insists that the same program triggers different mechanisms in different contexts. It is explicitly about explaining variation and building transferable theory, whereas a logic model is usually a planning and communication device. The two are complementary: a theory of change can be a starting point that realist evaluation then interrogates and refines.

Can realist evaluation use randomised or quantitative data?

Yes. Realist evaluation is method-neutral: it draws on whatever data best test the CMO propositions. Quantitative and even experimental data are valuable for establishing outcome patterns and subgroup differences, but realism reframes them as evidence about which mechanisms fired in which contexts rather than as a single average treatment effect. The hallmark is the explanatory framework, not a prohibition on numbers.

Related methods

Sources

  1. Pawson, R., & Tilley, N. (1997). Realistic Evaluation. London: SAGE Publications.

    ISBN: 9780761950097
    Copied.
  2. Pawson, R. (2006). Evidence-Based Policy: A Realist Perspective. London: SAGE Publications.

    ISBN: 9781412910606
    Copied.