Intercoder Reliability
- Introduced by
- Foundational coefficients by Cohen (1960) and Krippendorff (1970)
- Year
- 1960
- Type
- Procedure and coefficients for certifying coding agreement
- Family
- Reliability assessment
What does it do?
Intercoder reliability is the degree to which independent coders, applying the same coding scheme to the same content, arrive at the same coding decisions.
In content analysis it is the central guarantee that findings reflect the messages rather than the idiosyncrasies of who happened to code them, and reporting a chance-corrected reliability coefficient is a near-universal requirement for publication in communication research.
The intuition behind the method
A content analysis is only as trustworthy as its coding: if two trained coders read the same article and disagree on whether it is 'negative,' the resulting numbers say more about the coders than about the news. Intercoder reliability puts this to the test by having multiple coders independently code a shared subset of the material and measuring how often, and how meaningfully, they agree. The twist is that raw agreement is misleading — if 90% of items belong to one category, coders agree 90% of the time just by both picking it. So reliability is measured with chance-corrected coefficients that ask how much agreement exceeds what guessing alone would produce.
History
Concern with coder agreement is as old as quantitative content analysis, but it was formalized through a family of coefficients: Scott's pi (1955), Cohen's kappa (1960), and Krippendorff's alpha (1970). Long-running methodological debates — over which coefficient to use, what thresholds are acceptable, and how to handle the prevalence paradox — shaped the field. Hayes and Krippendorff's 2007 article in Communication Methods and Measures consolidated practice by providing a general algorithm and freely available macros for alpha, making rigorous reliability assessment routine in communication research.
When to use it
Assess intercoder reliability whenever human coders apply a coding scheme that involves any judgment and you intend to draw inferences from the coded data — which is essentially every content analysis and every manually annotated data set. It is required to demonstrate that a study's measurements are reproducible and not coder-dependent. It assumes the reliability subsample is representative of the full corpus and that the chosen coefficient matches the data's measurement level and coder count. It is unnecessary only when coding is fully deterministic (e.g., automated keyword counts with no judgment), though even then validating the rule against human judgment is wise.
Limitations and considerations
- Reliability is necessary but not sufficient for validity — coders can agree consistently on a flawed or trivial category.
- Chance-corrected coefficients can be paradoxically low when one category dominates, even with high raw agreement.
- Estimates depend on the size and representativeness of the reliability subsample; small samples give unstable values.
- Different coefficients (kappa, pi, alpha) can give different numbers, and choosing among them involves judgment.
Source conditions do not replace independent scientific review or validation by an analysis engine.
Strengths
- Certifies that coded data are reproducible and not artifacts of particular coders, the bedrock of valid content analysis.
- Chance-corrected coefficients prevent the inflated confidence that raw percent agreement gives for skewed categories.
- Forces codebook refinement: low reliability surfaces ambiguous categories before they corrupt the full data set.
- Widely standardized, so reported coefficients are comparable across studies and reviewers know how to judge them.
Applications
- Certifying the coding of news, advertising, and entertainment content before publishing content-analytic findings.
- Validating human annotations used to train and benchmark machine-learning text classifiers.
- Assessing agreement among raters in survey open-end coding and qualitative thematic analysis.
- Quality-controlling large coding teams by monitoring reliability across coders and over time.
- Establishing the credibility of cross-national or multi-team coding projects with shared codebooks.
How it works
How it works
Coders are trained on the codebook until they share a common understanding of the categories and decision rules, usually through practice coding and reconciliation of disagreements on material outside the reliability sample.
A representative subsample of the corpus (often 10–20% of units, with a sensible minimum) is coded independently by all coders, without consultation and blind to one another's decisions, to produce the data on which agreement is assessed.
An appropriate coefficient is selected for the design. Cohen's kappa, defined as observed agreement p_o minus chance agreement p_e over 1 minus p_e, suits two coders on nominal categories; percent agreement alone is rejected because it ignores chance.
Formula
\kappa = \frac{p_o - p_e}{1 - p_e}For more than two coders, ordinal/interval variables, or missing data, Krippendorff's alpha — one minus observed disagreement D_o over expected disagreement D_e — is the general standard, computed per variable on the reliability subsample.
Formula
\alpha = 1 - \frac{D_o}{D_e}The coefficient for each variable is reported and compared to accepted thresholds (commonly alpha at or above .80 for confident use, .667–.80 for tentative conclusions); variables falling short are revised, recoded, or dropped before the full corpus is analyzed.
Common pitfalls
- Reporting only percent agreement, which overstates reliability whenever category distributions are skewed.
- Letting coders confer during reliability coding, which manufactures agreement and invalidates the estimate.
- Computing a single overall coefficient across all variables, hiding which specific variables are unreliable.
- Estimating reliability on the same data used to train and refine coders, conflating practice with the actual test.
Frequently asked questions
Which intercoder reliability coefficient should I report?
For two coders on nominal categories, Cohen's kappa or Scott's pi are common; for any number of coders, ordinal/interval data, or missing values, Krippendorff's alpha is the general standard and is increasingly expected in communication journals. Percent agreement alone is not acceptable because it ignores chance agreement. When in doubt, report alpha per variable, and report percent agreement only as a supplement, never as the sole evidence.
How much of my data needs to be double-coded?
There is no universal rule, but coding 10–20% of the units with all coders is a widely used guideline, subject to a sensible minimum number of units per variable so that rare categories are represented. The subsample must be randomly drawn so it represents the full corpus. Larger or rarer-category-heavy studies need a larger reliability sample to estimate the coefficient with acceptable precision.
What reliability level is acceptable?
A common convention, from Krippendorff, is to treat coefficients at or above .80 as reliable, accept .667–.80 only for tentative conclusions, and reject variables below .667. These are guidelines, not strict laws; high-stakes coding warrants stricter cutoffs, and the appropriate threshold depends on the consequences of coding error in the specific study. Whatever the level, report the coefficient per variable so readers can judge.
Related methods
Sources
Hayes, A. F., & Krippendorff, K. (2007). Answering the call for a standard reliability measure for coding data. Communication Methods and Measures, 1(1), 77–89.
Open DOI referenceCopied.Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1), 37–46.
Open DOI referenceCopied.Krippendorff, K. (2004). Content Analysis: An Introduction to Its Methodology (2nd ed.). Thousand Oaks, CA: Sage.
ISBN: 9780761915454Copied.