Survey Experiment
- Introduced by
- Experimental political science; synthesized by Diana Mutz
- Year
- 2011
- Type
- Randomized experiment embedded in a survey
- Family
- Experimental survey designs
What does it do?
A survey experiment embeds a randomized experiment inside a survey: respondents are randomly assigned to different versions of a question, frame, or stimulus, and their answers are compared to estimate a causal effect.
By combining the internal validity of randomization with the representative samples and rich measurement of survey research, survey experiments — especially population-based ones — let political scientists draw causal inferences about how information, framing, or message attributes shape public attitudes and behavior.
The intuition behind the method
Observational survey data can show that people who heard a certain frame hold certain attitudes, but cannot tell whether the frame caused the attitude or whether some third factor drove both. A survey experiment cuts that knot: because the version each respondent sees is decided by a coin flip, the groups are identical in expectation except for the manipulation. Any systematic difference in their answers must be caused by the manipulation. Running this on a representative sample means the estimated effect generalizes to the population the sample represents, not just to undergraduates in a lab.
History
Question-wording and framing experiments embedded in surveys date to early public-opinion research, but the modern population-based approach crystallized as random-digit-dial and later internet panels made representative samples affordable. Gaines, Kuklinski, and Quirk's 2007 Political Analysis paper sharpened the logic and cautioned about pre-treatment and durability. Diana Mutz's 2011 book formalized population-based survey experiments as a methodology, and the 2011 Cambridge Handbook of Experimental Political Science consolidated the field, which now spans framing, information, and identity-cue research across the discipline.
When to use it
Use a survey experiment when you want to estimate the causal effect of an informational, framing, or wording manipulation on attitudes or behavior, and you need the result to generalize beyond a narrow lab sample. They are ideal for studying framing, persuasion, question-wording effects, partisan cues, and information provision. They are less appropriate when the treatment cannot be delivered briefly within a survey, when real-world exposure differs fundamentally from a one-shot stimulus, or when long-run or behavioral outcomes outside the survey are the true target; field experiments then fit better.
Limitations and considerations
- A brief survey stimulus may not capture the intensity or repetition of real-world exposure, threatening external validity.
- Effects measured immediately after treatment may decay quickly, so one-shot designs can overstate durable impact.
- Pre-treatment from prior real-world exposure can attenuate or contaminate the manipulation's measured effect.
- Outcomes are usually self-reported attitudes or intentions rather than consequential behavior, limiting behavioral inference.
Source conditions do not replace independent scientific review or validation by an analysis engine.
Strengths
- Randomization delivers high internal validity, isolating the causal effect of the manipulation from confounders.
- Population-based samples improve external validity over convenience samples, letting effects generalize to defined populations.
- Flexible and inexpensive to field within existing survey infrastructure, enabling many treatments and large samples.
- Combines causal identification with the rich covariate measurement of surveys, supporting subgroup and mechanism analysis.
Applications
- Estimating how alternative issue frames shift public support for policies such as immigration or welfare.
- Testing the effect of partisan or source cues on attitude formation and credibility judgments.
- Measuring how factual information provision changes policy preferences or misperceptions.
- Studying question-wording and response-order effects that bias measurement in surveys.
- Evaluating how candidate or message attributes affect vote intention and evaluations across demographic groups.
How it works
How it works
The researcher specifies the experimental conditions — for example, two framings of a policy, the presence or absence of a partisan cue, or several information treatments — operationalizing the theoretical manipulation as concrete survey content. A control condition with no manipulation or a neutral version anchors the comparison.
Survey software assigns each respondent to one of the K conditions with known probability, usually equal. Randomization makes the treatment groups statistically equivalent on all observed and unobserved characteristics in expectation, which is what licenses a causal interpretation of differences in outcomes.
Formula
T_i \in \{1,\dots,K\},\quad \Pr(T_i = k) = 1/KAfter exposure to their assigned condition, respondents answer outcome questions — attitudes, intended behavior, knowledge, or evaluations. The observed outcome for each respondent is the potential outcome under the condition they happened to receive.
Formula
Y_i = Y_i(T_i)
The average treatment effect is estimated as the difference in mean outcomes between conditions, optionally with covariate adjustment to improve precision. Standard errors follow from the randomization, and regression can incorporate multiple treatments or interactions.
Formula
\widehat{\text{ATE}} = \bar{Y}_{\,T=k} - \bar{Y}_{\,T=k'}Analysts probe whether effects differ across subgroups (heterogeneous treatment effects) and assess external validity — whether the manipulation, sample, and context generalize to the real-world phenomenon. Gaines, Kuklinski, and Quirk warn about pre-treatment effects, durability, and the gap between experimental stimuli and real exposure.
Common pitfalls
- Confounding the manipulation with incidental differences between conditions (length, tone, salience) so the estimated effect is not the intended construct.
- Ignoring pre-treatment: respondents already exposed to the issue may show muted effects, misleadingly suggesting the frame is weak.
- Failing to check balance and attrition across arms, which can break the randomization guarantee.
- Over-generalizing a short-term attitudinal effect to durable, real-world behavioral change.
Frequently asked questions
What distinguishes a survey experiment from a lab or field experiment?
All three use randomization for causal inference, but differ in setting and sample. Lab experiments offer tight control but rely on small, often non-representative samples. Field experiments manipulate conditions in real-world settings with consequential outcomes but are costly and hard to control. Survey experiments embed the manipulation in a questionnaire administered to a (often representative) sample, balancing internal validity, generalizable samples, and low cost, at the price of brief, sometimes artificial stimuli.
Why does the sample matter if randomization already ensures internal validity?
Randomization makes the estimate internally valid for whoever is in the sample, but the magnitude and even sign of framing or information effects can vary across populations. A convenience sample of students may respond differently from the general electorate. Population-based samples let the estimated effect generalize to a defined target population, which is essential when the substantive claim concerns public opinion writ large rather than a particular subgroup.
What is the pre-treatment problem?
Pre-treatment occurs when respondents have already been exposed in the real world to information similar to the experimental treatment before the study. Such respondents may already hold the attitude the treatment would induce, so the manipulation produces little additional movement, attenuating the measured effect. This makes experimental effects on heavily covered issues look weaker than the underlying causal force, a concern Gaines and colleagues emphasized.
Related methods
Sources
Mutz, D. C. (2011). Population-Based Survey Experiments. Princeton, NJ: Princeton University Press.
ISBN: 9780691144528Copied.Gaines, B. J., Kuklinski, J. H., & Quirk, P. J. (2007). The Logic of the Survey Experiment Reexamined. Political Analysis, 15(1), 1–20.
Open DOI referenceCopied.Druckman, J. N., Green, D. P., Kuklinski, J. H., & Lupia, A. (Eds.) (2011). Cambridge Handbook of Experimental Political Science. Cambridge University Press.
ISBN: 9780521174558Copied.