Factorial Survey Experiment
- Introduced by
- Peter H. Rossi and collaborators
- Year
- 1982
- Type
- Multi-factor randomized vignette experiment
- Family
- Experimental survey designs
What does it do?
A factorial survey experiment, often simply called a factorial survey, asks respondents to judge short descriptions — vignettes — whose multiple features are fully crossed and randomly varied.
By factorially combining many dimensions, each at several levels, the design generates a large universe of vignettes; respondents rate a random sample of them, and regression of the ratings on the dimension levels recovers the independent causal effect of each feature on judgment. It scales the single-scenario vignette experiment up to many simultaneously manipulated attributes.
The intuition behind the method
Suppose you want to know what drives perceived fairness of a salary: the worker's experience, education, gender, and job type all plausibly matter, and they may interact. Asking abstractly is hopeless. Instead, the factorial survey builds thousands of distinct described workers by crossing every level of every dimension, then shows each respondent a handful of these profiles to rate. Because the dimensions are crossed independently and assigned at random, a regression can untangle exactly how much each feature moves the fairness rating, holding the others constant — even features respondents would never admit they weigh.
History
The factorial survey was developed by Peter Rossi and collaborators in the 1970s and codified in the 1982 edited volume Measuring Social Judgments, applying it to perceptions of social standing, fairness, and deservingness. Over the following decades sociologists adopted it widely; Wallander's 2009 review in Social Science Research surveyed twenty-five years of applications and methodological developments, including multilevel analysis of the nested ratings. Atzmüller and Steiner's 2010 paper clarified the sampling and design choices that connect the factorial survey to the broader experimental vignette tradition, and the approach is a direct ancestor of the modern conjoint experiment.
When to use it
Use a factorial survey experiment when you want to estimate how many distinct attributes of a described person or situation simultaneously and independently shape a judgment, and you suspect interactions or want to separate population-level norms from individual differences. It is ideal for studying judgment principles — fairness, deservingness, blame, hiring — across a rich attribute space. It is less suitable when the response of interest is a discrete choice between alternatives (a conjoint design fits better), when respondents cannot rate many vignettes without fatigue, or when a single attribute is the sole focus, where a simple vignette experiment suffices.
Limitations and considerations
- The vignette universe grows multiplicatively, so high-dimensional designs require careful sampling and large respondent pools.
- Rating fatigue and order effects can bias judgments when respondents evaluate many vignettes in sequence.
- Ratings are stated judgments of hypothetical cases, which may diverge from consequential real-world decisions.
- Implausible level combinations can arise from full crossing and must be constrained or screened to preserve realism.
Source conditions do not replace independent scientific review or validation by an analysis engine.
Strengths
- Estimates the independent causal effect of many attributes at once, because full factorial crossing makes the dimensions orthogonal.
- Multilevel structure separates population-level judgment principles from individual-level heterogeneity and rating styles.
- High realism: respondents react to concrete, multi-feature descriptions rather than abstract attitude items.
- Efficient use of respondents — each rates several vignettes — yielding large effective samples for the dimension effects.
Applications
- Studying principles of distributive justice and perceptions of fair earnings across worker attributes.
- Measuring welfare and immigration deservingness judgments across many recipient characteristics.
- Examining sentencing and blame attributions as functions of offender and offense features.
- Investigating hiring and promotion judgments by varying candidate qualifications and demographics.
- Assessing perceived legitimacy of policies by crossing their design features and contextual conditions.
How it works
How it works
The researcher defines several dimensions of the vignette, each with a fixed set of levels, and crosses them fully. The size of the vignette universe is the product of the number of levels across all dimensions, which can run into the thousands — far too many for any single respondent to evaluate.
Formula
|\mathcal{V}| = \prod_{j=1}^{p} L_jBecause the full universe is too large, each respondent is shown a sampled deck of vignettes, drawn randomly or via a balanced/fractional design that preserves orthogonality among dimensions. Sampling spreads the universe across the respondent pool so all level combinations are evaluated overall while keeping each respondent's task manageable.
Each respondent rates each vignette in their deck on the outcome scale — fairness, deservingness, sentence severity, hireability, and so on. This yields a nested data structure: many ratings per respondent, with vignettes' dimension levels recorded as predictors.
Formula
Y_{ik} = \text{rating by respondent } i \text{ of vignette } kRatings are modeled with multilevel (mixed-effects) regression: fixed-effect coefficients on the dimension levels give the average causal effect of each vignette feature, while a respondent random effect absorbs individual rating tendencies and the correlation among a respondent's multiple ratings. Cross-level interactions link vignette effects to respondent characteristics.
Formula
Y_{ik} = \beta_0 + \sum_{j} \beta_j\, X_{jk} + u_i + \varepsilon_{ik}The dimension coefficients are read as the independent contribution of each manipulated feature to judgment, net of all others, because the factorial crossing makes them orthogonal. Interactions among dimensions reveal conditional effects, and the design distinguishes population-level judgment principles from individual heterogeneity in how they are applied.
Common pitfalls
- Allowing logically impossible or absurd attribute combinations (e.g., a 20-year-old with 30 years of experience) that undermine realism and confuse respondents.
- Ignoring the nested data structure and fitting ordinary regression, which understates standard errors by treating correlated ratings as independent.
- Using a non-orthogonal vignette sample that confounds dimensions, defeating the purpose of factorial crossing.
- Overloading respondents with too many vignettes, inducing fatigue, satisficing, and degraded later ratings.
Frequently asked questions
How does a factorial survey differ from a conjoint experiment?
Both cross many randomized attributes, but the response task differs. In a factorial survey, respondents rate each vignette on a scale, and analysis regresses ratings on attribute levels with a multilevel model. In a conjoint experiment, respondents typically choose between or rank profiles, and analysis estimates average marginal component effects from those choices. Conjoint's forced choice mimics real selection and reduces scale-use bias; the factorial survey's ratings preserve graded judgment and intensity.
Why use a multilevel model for factorial survey data?
Each respondent rates several vignettes, so the ratings are clustered within respondents and not independent. A multilevel model adds a respondent-level random effect to absorb that clustering and individual rating tendencies, producing correct standard errors for the dimension effects. It also lets the analyst model heterogeneity — whether vignette effects depend on respondent characteristics — through cross-level interactions, separating shared judgment norms from individual variation.
How are implausible vignettes handled?
Full factorial crossing can generate combinations that are logically impossible or wildly unrealistic, which confuse respondents and add noise. Researchers handle this by imposing constraints that exclude impossible cells from the sampling universe, or by restricting level ranges so all combinations remain plausible. The trade-off is that constraints can reintroduce correlations among dimensions, so they are applied sparingly and documented.
Related methods
Sources
Wallander, L. (2009). 25 Years of Factorial Surveys in Sociology: A Review. Social Science Research, 38(3), 505–520.
Open DOI referenceCopied.Atzmüller, C., & Steiner, P. M. (2010). Experimental Vignette Studies in Survey Research. Methodology, 6(3), 128–138.
Open DOI referenceCopied.Rossi, P. H., & Nock, S. L. (Eds.) (1982). Measuring Social Judgments: The Factorial Survey Approach. Beverly Hills, CA: Sage.
ISBN: 9780803918184Copied.