Skip to content
Back to search results

Conjoint Survey Experiment

Introduced by
Jens Hainmueller, Daniel Hopkins, Teppei Yamamoto
Year
2014
Type
Multi-attribute forced-choice survey experiment with design-based causal estimands
Family
Experimental survey designs
Font size

What does it do?

A conjoint survey experiment presents respondents with profiles — of candidates, immigrants, policies, or products — described by several attributes whose levels are independently randomized, and asks respondents to choose between or rate the profiles.

Hainmueller, Hopkins, and Yamamoto's 2014 framework places this design on a rigorous causal footing, defining the average marginal component effect (AMCE) as the design-based causal effect of an attribute level, averaged over the randomization distribution of all other attributes. It lets political scientists estimate the relative causal weight of many decision factors simultaneously from realistic, multidimensional choices.

The intuition behind the method

Real choices — which candidate to vote for, which immigrant to admit — weigh many traits at once, and people cannot reliably report how much each trait matters. A conjoint experiment recreates that multidimensional choice: each profile's attributes are randomized independently, so when respondents pick a candidate, the analyst can attribute the choice to specific attribute levels. Because everything else varies at random, the effect of, say, being a teacher versus a lawyer is cleanly identified by comparing choice rates across profiles, averaged over all the other randomized traits. The AMCE answers: how much does this attribute level, on its own, change the probability of being chosen?

History

Conjoint analysis originated in mathematical psychology and marketing in the 1960s and 1970s as a tool for measuring multi-attribute preferences. Hainmueller, Hopkins, and Yamamoto's 2014 paper in Political Analysis reframed the design within the potential-outcomes framework, defining the AMCE and showing it is identified purely by attribute randomization, which catalyzed widespread adoption across political science. Their 2015 PNAS study validated conjoint estimates against real referendum behavior, and Leeper, Hobolt, and Tilley's 2020 work clarified the pitfalls of subgroup comparison, cementing best practice for causal conjoint experiments.

When to use it

Use a conjoint survey experiment when a decision of interest is inherently multidimensional and you want to estimate the independent causal effect of many attributes simultaneously under realistic trade-offs — vote choice over candidate traits, immigration preferences over applicant characteristics, or policy support over bundled features. It is well suited to disentangling which attributes drive choice and to detecting discrimination. It is less appropriate when a single attribute is the sole focus (a simple survey experiment is cleaner), when graded intensity rather than choice is the target (a rating-based factorial survey fits), or when realistic attribute combinations cannot be maintained.

Limitations and considerations

  • The AMCE depends on the baseline level and the randomization distribution of attributes, complicating cross-subgroup comparisons.
  • Independent randomization can produce implausible profiles; constraints to avoid them reintroduce attribute correlation.
  • An average effect can mask strong preference heterogeneity and non-monotonic or interactive attribute effects.
  • Many repeated tasks risk respondent fatigue, satisficing, and carryover effects that degrade later choices.

Source conditions do not replace independent scientific review or validation by an analysis engine.

Strengths

  • Estimates the independent causal effect of many attributes at once under realistic, multidimensional trade-offs.
  • Design-based identification: the AMCE is nonparametrically identified by attribute randomization with minimal modeling assumptions.
  • Forced choice reduces social-desirability bias and scale-use heterogeneity relative to direct attitude items.
  • Validated against real-world behavior, with conjoint estimates tracking actual choices in several benchmark studies.

Applications

  • Estimating which candidate attributes (gender, party, occupation, policy positions) drive vote choice.
  • Measuring public preferences over immigrant characteristics and detecting ethnic or skill-based discrimination.
  • Assessing support for multidimensional policy packages by randomizing their component features.
  • Studying voter responses to coalition or party platform bundles in multiparty systems.
  • Examining hiring and admissions preferences by randomizing applicant attributes in described profiles.

How it works

How it works

1

Each profile attribute is assigned a level drawn independently from a known distribution, so attributes are statistically independent of one another by design. This independence is the engine of identification: it ensures that any effect attributed to one attribute is not confounded by correlated variation in the others.

Formula
\Pr(T_{jk} = \ell) = p_{j\ell},\quad \text{attributes independent}
2

Respondents view two or more fully described profiles side by side, typically across several repeated tasks, mimicking a realistic comparison. Attribute order can be randomized across respondents to detect and average over ordering effects, and constraints can forbid implausible combinations at a documented cost to independence.

3

The primary outcome is the respondent's forced choice of which profile they prefer (a binary indicator per profile), optionally supplemented by a rating. Repeated tasks per respondent generate many profile-level observations, increasing efficiency.

Formula
Y_{i} = \mathbb{1}\{\text{profile chosen}\}
4

The average marginal component effect is the change in the probability that a profile is chosen when one attribute moves from a baseline level to another, averaged over the randomization distribution of all remaining attributes and over the respondent population. It is estimated simply by regressing the choice indicator on attribute-level dummies with respondent-clustered standard errors.

Formula
\text{AMCE}_{j} = E\big[Y(t_{j}) - Y(t_{j}^{0}) \mid \text{other attributes randomized}\big]
5

Marginal means and subgroup analyses describe preferences within respondent groups; Leeper, Hobolt, and Tilley show that comparing AMCEs across subgroups can mislead because the AMCE depends on the chosen baseline, and recommend marginal means for descriptive subgroup comparison. Diagnostics also check attribute-order and profile-order effects and carryover across tasks.

Common pitfalls

  • Comparing AMCEs across respondent subgroups as if they were preferences, when the AMCE is baseline-dependent; use marginal means instead.
  • Imposing attribute constraints to avoid implausible profiles without acknowledging the resulting loss of independence and bias.
  • Ignoring profile-order and attribute-order effects, which can bias estimates if not randomized and averaged over.
  • Interpreting a near-zero AMCE as 'no preference' when offsetting heterogeneous effects across respondents cancel out.

Frequently asked questions

How does this differ from classic marketing conjoint analysis?

Classic conjoint analysis, rooted in marketing, focuses on estimating individual-level part-worth utilities and predicting market shares, often with hierarchical Bayesian choice models. The political-science conjoint survey experiment, following Hainmueller and colleagues, emphasizes design-based causal identification: it defines the AMCE as a population causal estimand identified by attribute randomization and estimated with simple regression, prioritizing transparent causal interpretation over individual utility recovery. The two share the multi-attribute profile structure but differ in estimand and inferential goals.

What exactly does the AMCE measure?

The average marginal component effect is the change in the probability that a profile is selected when one attribute is moved from a baseline level to a target level, averaged over the randomization distribution of all other attributes and over respondents. It is a causal quantity under the design, but it is relative to the chosen baseline and to how the other attributes are randomized, which is why it should not be naively compared across subgroups with different choice patterns.

Why are marginal means preferred for subgroup comparisons?

Because the AMCE is defined relative to a baseline level, two subgroups can have identical preference orderings yet different AMCEs simply due to differing baseline favorability, producing spurious differences. Marginal means — the average probability that profiles with a given attribute level are chosen — are not baseline-dependent, so Leeper, Hobolt, and Tilley recommend them for descriptively comparing preferences across subgroups, reserving AMCEs for the overall causal effects.

Related methods

Sources

  1. Hainmueller, J., Hopkins, D. J., & Yamamoto, T. (2014). Causal Inference in Conjoint Analysis: Understanding Multidimensional Choices via Stated Preference Experiments. Political Analysis, 22(1), 1–30.

    Open DOI reference
    Copied.
  2. Hainmueller, J., Hangartner, D., & Yamamoto, T. (2015). Validating Vignette and Conjoint Survey Experiments against Real-World Behavior. Proceedings of the National Academy of Sciences, 112(8), 2395–2400.

    Open DOI reference
    Copied.
  3. Leeper, T. J., Hobolt, S. B., & Tilley, J. (2020). Measuring Subgroup Preferences in Conjoint Experiments. Political Analysis, 28(2), 207–221.

    Open DOI reference
    Copied.