Skip to content
Back to search results

Expert Survey

Introduced by
Comparative party-positioning research (Castles & Mair; Chapel Hill team)
Type
Survey of subject-matter experts to measure latent positions
Family
Expert judgment / measurement
Font size

What does it do?

An expert survey measures latent political quantities — most often parties' positions on policy dimensions — by asking a panel of country and subject-matter experts to place the objects of interest on structured numerical scales.

Averaging many experts' judgments yields position estimates, while the spread across experts provides a built-in measure of uncertainty and reliability. The Chapel Hill Expert Survey is the leading example, producing comparable measures of European parties' positions on ideology, European integration, and many specific issues over time.

The intuition behind the method

Some things political scientists want to measure — exactly how Eurosceptic a party is, or how far left it sits on economics — are not written down in any single document and are hard to extract from behavior. But people who study a country's politics closely carry that knowledge. An expert survey taps it systematically: instead of one scholar's impression, it asks dozens of experts to rate each party on the same scales, then pools their ratings. The average is the best collective estimate, and the agreement among experts tells you how trustworthy that estimate is — much like averaging many measurements to cancel individual error.

History

Expert placement of parties dates to Castles and Mair's 1984 left–right survey and Laver and Hunt's expert study of party policy. The approach matured with the Chapel Hill Expert Survey, which from 1999 systematically measured European parties' positions on ideology and European integration. Steenbergen and Marks' 2007 paper developed methods for evaluating expert-judgment quality, and Hooghe and colleagues' 2010 study assessed the reliability and validity of the Chapel Hill waves. Bakker and colleagues' 2015 trend file consolidated comparable measures over 1999–2010, making expert surveys a core data source for comparative politics.

When to use it

Use an expert survey when you need comparable measures of latent positions — party ideology, issue stances, regime characteristics, institutional quality — that are not directly observable and are hard to extract from documents or behavior, and when knowledgeable experts exist. They are ideal for cross-national party-positioning research and for benchmarking other methods. They are less appropriate when expert pools are thin or biased, when the construct is rapidly changing between waves, or when experts' shared conventional wisdom may be systematically wrong; in such cases manifesto coding, text scaling, or mass surveys provide complementary evidence.

Limitations and considerations

  • Estimates reflect experts' perceptions, which can lag reality, share conventional wisdom, or carry collective bias.
  • Quality depends on the size, expertise, and independence of the expert pool, which varies across countries.
  • Cross-national scale interpretation may differ, so a '5' in one country need not mean the same as in another.
  • Snapshots at survey waves can miss rapid position change between waves, unlike continuously coded sources.

Source conditions do not replace independent scientific review or validation by an analysis engine.

Strengths

  • Captures latent positions that are hard to measure from documents or behavior by tapping experts' synthesized knowledge.
  • Averaging many experts cancels idiosyncratic error and yields reliable, comparable cross-national estimates.
  • The dispersion across experts provides a direct, interpretable measure of uncertainty and construct contestedness.
  • Flexible: can measure many specific issues and dimensions in one instrument, including newly salient ones.

Applications

  • Measuring parties' left–right and European-integration positions across countries and over time.
  • Locating parties on specific issue dimensions such as immigration, the environment, or civil liberties.
  • Validating and benchmarking manifesto codings and automated text-scaling estimates of party positions.
  • Measuring institutional and regime characteristics (e.g., judicial independence, corruption) via expert panels.
  • Constructing cross-national indices for studies of representation, polarization, and party competition.

How it works

How it works

1

The researcher defines the constructs to be measured and writes structured placement items — typically continuous or fixed-point scales with clearly anchored endpoints (e.g., 0 'extreme left' to 10 'extreme right'). Careful wording and anchoring are essential so experts in different countries interpret the scales consistently.

2

A panel of experts with deep knowledge of each country's party system is recruited, aiming for several experts per party to average out idiosyncratic judgment. Coverage, expertise, and independence of experts shape the quality of the resulting estimates.

3

Each expert places every relevant party (or other object) on each scale, giving a numerical judgment. Experts may also rate their own certainty or the salience of an issue to a party, providing auxiliary information for weighting and interpretation.

Formula
x_{ej} \in [\text{min}, \text{max}]
4

For each party and scale, the expert placements are aggregated, usually by taking the mean (sometimes a weighted or trimmed mean) across the experts who rated it. This pooled value is the position estimate, and the number of experts determines the precision of the average.

Formula
\hat{x}_j = \frac{1}{E_j}\sum_{e=1}^{E_j} x_{ej}
5

The dispersion of expert ratings is used to assess reliability — high agreement (e.g., a high intraclass correlation, low variance) indicates a well-measured position, while wide disagreement flags contested or poorly defined constructs. Estimates are validated by cross-checking against manifesto codings, voter surveys, and earlier expert waves.

Formula
\text{ICC},\ \operatorname{Var}(x_{ej}),\ \text{cross-validation}

Common pitfalls

  • Relying on too few experts per object, so idiosyncratic judgments are not averaged out and estimates are noisy.
  • Ignoring inter-expert disagreement and reporting a mean as if it were precisely measured.
  • Assuming scale endpoints are interpreted identically across countries without anchoring or differential-item-functioning checks.
  • Treating expert perceptions as ground truth rather than one validated measure to be triangulated with others.

Frequently asked questions

How do expert surveys compare to manifesto coding for measuring party positions?

Both estimate party positions but from different sources. Manifesto coding analyzes parties' own written platforms, giving a behavioral, document-based measure tied to specific elections. Expert surveys aggregate the perceptions of knowledgeable scholars, capturing a party's overall reputation and recent behavior beyond any single document. Expert surveys often correlate well with manifesto and voter data and are frequently used to validate them; using both triangulates the latent positions and exposes where sources disagree.

How is the reliability of an expert survey assessed?

Reliability is judged primarily from agreement among experts rating the same object. Low variance or high intraclass correlation across experts indicates the position is measured consistently, while wide disagreement signals a contested or ambiguous construct. Analysts also check validity by correlating estimates with independent measures — manifesto data, mass surveys, prior waves — and by examining whether scale endpoints function equivalently across countries, as Steenbergen and Marks and the Chapel Hill reliability studies demonstrate.

Why use multiple experts per party instead of one authority?

Any single expert's placement carries idiosyncratic error and personal bias. Averaging several independent experts cancels much of this random error, producing a more precise and defensible estimate, and the spread of their judgments quantifies the remaining uncertainty. With only one expert there is no way to separate signal from individual error or to assess reliability, which is why expert surveys deliberately solicit several experts per object.

Related methods

Sources

  1. Bakker, R., de Vries, C., Edwards, E., Hooghe, L., Jolly, S., Marks, G., Polk, J., Rovny, J., Steenbergen, M., & Vachudova, M. A. (2015). Measuring Party Positions in Europe: The Chapel Hill Expert Survey Trend File, 1999–2010. Party Politics, 21(1), 143–152.

    Open DOI reference
    Copied.
  2. Steenbergen, M. R., & Marks, G. (2007). Evaluating Expert Judgments. European Journal of Political Research, 46(3), 347–366.

    Open DOI reference
    Copied.
  3. Hooghe, L., Bakker, R., Brigevich, A., de Vries, C., Edwards, E., Marks, G., Rovny, J., Steenbergen, M., & Vachudova, M. (2010). Reliability and Validity of the 2002 and 2006 Chapel Hill Expert Surveys on Party Positioning. European Journal of Political Research, 49(5), 687–703.

    Open DOI reference
    Copied.