List Experiment
- Introduced by
- Survey methodology; modern estimators by Kosuke Imai, Graeme Blair, Adam Glynn
- Year
- 2011
- Type
- Sensitive-question survey experiment
- Family
- Indirect questioning techniques
What does it do?
The list experiment, also called the item count technique, is a survey design that measures the prevalence of a sensitive attitude or behavior without ever requiring any respondent to directly disclose it.
Respondents are randomly split into two groups: a control group sees a list of innocuous items and reports only how many apply to them, while a treatment group sees the same list plus one sensitive item. Because respondents report only a count, no individual answer reveals their stance on the sensitive item, and the difference in average counts between the groups estimates the proportion holding the sensitive trait.
The intuition behind the method
People systematically misreport on surveys when the truthful answer is socially undesirable or risky — admitting prejudice, vote buying, or illegal behavior. The list experiment removes the incentive to lie by hiding the sensitive answer inside an aggregate. If the control group reports on average 1.8 items and the treatment group, which sees one extra sensitive item, reports 2.3, then about 0.5 — fifty percent — of respondents must have the sensitive trait, yet no single person ever said 'yes' to it. The privacy is structural: the analyst learns the population rate without learning any individual's answer.
History
Indirect counting methods to protect respondent privacy were developed in survey methodology and psychology from the late 1970s and 1980s, with the unmatched count technique applied to sensitive topics such as drug use. Political scientists adopted the design widely in the 2000s for measuring racial attitudes, support for militant groups, and electoral malpractice. Kosuke Imai's 2011 paper in the Journal of the American Statistical Association provided rigorous multivariate regression estimators, and Blair and Imai's 2012 Political Analysis paper supplied a full statistical framework with diagnostics and the widely used R package, turning the technique into a standard tool of empirical political science.
When to use it
Use a list experiment when you need an aggregate prevalence estimate of a sensitive attitude or behavior that direct questioning would distort through social-desirability bias or fear of disclosure — racial prejudice, support for political violence, vote buying, corruption, or stigmatized health behaviors. It requires a moderately large sample because the difference-in-means estimator is statistically inefficient relative to direct questions. Do not use it when you need individual-level truth for every respondent, when the sample is small, or when a suitable non-sensitive control list cannot be constructed; in those cases the randomized response technique or endorsement experiments may fit better.
Limitations and considerations
- Statistically inefficient: estimating a prevalence to a given precision needs far larger samples than direct questioning.
- Vulnerable to ceiling and floor effects when control-item prevalences are too extreme, biasing the estimate and undermining privacy.
- Relies on the no-design-effect and no-liars assumptions, which can fail if the sensitive item shifts how respondents read the whole list.
- Can yield logically impossible negative prevalence estimates in finite samples, signaling assumption violations or noise.
Source conditions do not replace independent scientific review or validation by an analysis engine.
Strengths
- Provides strong privacy protection because no respondent ever directly reveals the sensitive answer, reducing social-desirability and disclosure bias.
- The core difference-in-means estimator is unbiased under randomization and requires no distributional assumptions.
- Easily embedded in standard survey platforms and comprehensible to respondents, improving compliance relative to more complex indirect methods.
- Modern regression estimators recover multivariate relationships and predicted probabilities, not just a single prevalence number.
Applications
- Measuring the true prevalence of racial prejudice and intolerance in public-opinion surveys where direct questions are distorted.
- Estimating support for political violence or militant organizations in conflict and post-conflict settings.
- Detecting vote buying, clientelism, and electoral fraud where respondents fear admitting involvement.
- Studying stigmatized or illegal behaviors such as drug use, corruption, or tax evasion.
- Gauging hidden opposition to incumbent regimes in authoritarian contexts where candid answers carry risk.
How it works
How it works
The researcher builds a control list of three to five non-sensitive items chosen so their prevalences are neither all very high nor all very low, avoiding ceiling and floor effects that would let respondents infer that revealing the count exposes the sensitive answer. The treatment list is identical plus the single sensitive item. Items should be negatively correlated where possible so that few respondents endorse all or none.
Each respondent is independently assigned to the control or treatment condition with equal probability, typically by a coin-flip indicator embedded in the survey software. Randomization guarantees the two groups are equivalent in expectation on the baseline items, so any difference in mean counts is attributable to the added sensitive item.
Formula
Z_i \sim \text{Bernoulli}(0.5)The unbiased estimate of the sensitive-trait prevalence is simply the treatment group's mean count minus the control group's mean count. Under random assignment this difference-in-means estimator requires no modeling assumptions and its standard error follows directly from the two group variances.
Formula
\hat{\pi} = \bar{Y}_{\text{treatment}} - \bar{Y}_{\text{control}}To relate the sensitive trait to covariates, maximum-likelihood or nonlinear least-squares estimators model the count as a function of respondent characteristics in both arms, recovering how the latent sensitive response varies with predictors. Imai's two-step and EM estimators make this multivariate analysis efficient and allow predicted individual-level probabilities.
Formula
E[Y_i \mid X_i, Z_i] = f(X_i) + Z_i\, g(X_i)
Design and statistical diagnostics test the no-design-effect assumption (the sensitive item does not change responses to the control items) and the no-liars assumption (respondents answer the sensitive item truthfully). Blair and Imai's procedures detect negative estimated proportions or list inflation, flagging ceiling/floor effects or strategic misreporting.
Common pitfalls
- Choosing control items with very high or very low prevalence, which creates ceiling/floor effects and lets respondents deduce that the count reveals their sensitive answer.
- Using a sample too small to detect a plausible effect, producing wide confidence intervals that include zero or negative values.
- Failing to pilot the list, so unanticipated correlations among items inflate variance and reduce power.
- Interpreting a negative estimated proportion as a real result rather than a diagnostic signal of design failure or sampling error.
Frequently asked questions
How is a list experiment different from the randomized response technique?
Both protect privacy, but mechanically differ. The list experiment hides the sensitive answer in an aggregate count across a control and treatment list, requiring random assignment between respondents. The randomized response technique randomizes within each respondent — a coin or die determines whether they answer the sensitive question or a neutral one — so privacy comes from the analyst not knowing which question was answered. List experiments are usually simpler for respondents to understand; randomized response can be more statistically efficient.
Why can a list experiment produce a negative prevalence estimate?
The difference-in-means estimator is unbiased but noisy, so in any finite sample it can fall below zero by chance, especially with small samples or a low true prevalence. A persistently negative estimate is more worrying: it usually signals a design failure such as a floor effect, deflation of the control list, or violation of the no-design-effect assumption, and should trigger the diagnostic checks rather than be reported as a substantive finding.
How many control items should the list contain?
Most designs use three to five control items. Too few items make it easy for a respondent reporting the maximum or minimum count to be exposed, harming privacy; too many increase response variance and reduce statistical power. The items should be chosen so their combined counts rarely hit the floor or ceiling and so they are not all positively correlated, which is best confirmed in a pilot.
Related methods
Sources
Imai, K. (2011). Multivariate Regression Analysis for the Item Count Technique. Journal of the American Statistical Association, 106(494), 407–416.
Open DOI referenceCopied.Blair, G., & Imai, K. (2012). Statistical Analysis of List Experiments. Political Analysis, 20(1), 47–77.
Open DOI referenceCopied.Glynn, A. N. (2013). What Can We Learn with Statistical Truth Serum? Design and Analysis of the List Experiment. Public Opinion Quarterly, 77(S1), 159–172.
Open DOI referenceCopied.