Historical Nominal Record Linkage
- Introduced by
- Ivan Fellegi and Alan Sunter (probabilistic theory); James Feigenbaum, Ran Abramitzky, Leah Boustan (historical ML methods)
- Year
- 2016
- Type
- measurement-linkage
- Family
- record-linkage
What does it do?
Historical nominal record linkage is the task of recognising when records in different sources, two censuses, a census and a draft register, a baptism and a marriage, refer to the same person, even though no shared identifier exists and names are misspelled, ages misreported, and places renamed.
Linkage is the engine behind longitudinal historical micro-data: it builds the life-course panels that underpin studies of migration, mobility, mortality, and the long-run effects of early-life conditions. Three families of methods dominate. Deterministic linkage applies hand-crafted rules; the probabilistic Fellegi-Sunter framework weights field agreements and disagreements by their discriminating power; and supervised machine learning, trained on hand-linked examples, learns to classify candidate pairs. Modern historical practice, led by Abramitzky, Boustan, Feigenbaum, and collaborators, emphasises transparent, replicable algorithms and, crucially, explicit measurement of linkage error, since false matches and missed links can bias every downstream estimate.
The intuition behind the method
Picture two enormous lists of names taken decades apart, and the job of deciding which 'John Smith, age 34, born Ohio' on the first list is the same man as a 'Jno. Smyth, age 44, born Ohio' on the second, knowing that ages drift, clerks mishear names, and birthplaces get recoded. Comparing every name to every other is hopeless at scale, so you first block, grouping records that could plausibly match (same name sound, same birth decade) to shrink the search. Then you score each candidate pair by how well its fields agree, rewarding agreement on rare, discriminating values and discounting agreement on common ones, since matching on a frequent name means little. High-scoring pairs are linked, low-scoring rejected, and an uncertain middle band is flagged. The decisive discipline is honesty about error: because no rule is perfect, you measure how often you link wrong people together and how often you miss true links, and you carry that uncertainty into everything you build on the links.
History
The statistical theory of record linkage was formalised by Newcombe and then by Fellegi and Sunter, whose 1969 model remains foundational. Historical applications long relied on painstaking manual linkage, but digitised full-count censuses, especially the U.S. censuses via IPUMS, transformed the field. Feigenbaum's 2016 work introduced supervised machine learning trained on hand-linked samples for historical census linking, and Abramitzky, Boustan, and collaborators systematised the approach, publishing comparisons of methods and, influentially, urging explicit reporting of linkage error and its consequences for inference. Their 'A Method of Linking' and the LIFE-M and Census Linking Project resources have made transparent, replicable historical linkage a methodological standard across economic history and historical demography.
When to use it
Use nominal record linkage whenever a research question requires following the same individuals across separate historical sources, to build life-course or intergenerational panels for studies of migration, social mobility, mortality, fertility, or the long-run effects of early-life exposures. Probabilistic and machine-learning methods are preferred at scale and when transparent error measurement is needed; simple deterministic rules suffice only for small, clean, high-quality sources. Always pair linkage with evaluation against a labelled sample and with analysis of selection into the linked set. Avoid relying on linked data without quantifying linkage error, since false matches and selective non-linkage can severely bias every downstream estimate.
Limitations and considerations
- Common names and sparse identifiers make many individuals unlinkable
- Blocking can discard true matches split across blocks
- Linked samples are selective, over-representing stable, distinctively named people
- Requires labelled training data for probabilistic or ML calibration
Source conditions do not replace independent scientific review or validation by an analysis engine.
Strengths
- Builds longitudinal panels from otherwise cross-sectional historical sources
- Fellegi-Sunter weighting principled in rewarding rare, discriminating agreements
- Machine-learning classifiers exploit complex patterns and scale to millions of records
- Modern practice quantifies linkage error explicitly, enabling bias correction
Applications
- Building intergenerational mobility panels from linked full-count censuses
- Tracking internal and international migration across census waves
- Measuring long-run effects of early-life conditions via linked life histories
- Constructing genealogies and family reconstitutions from registers
- Linking soldiers, immigrants, or pension applicants to census records
How it works
How it works
Linkage begins by standardising the identifying fields so they can be compared across sources: names are cleaned of spelling and abbreviation variants (often via phonetic codes like Soundex or NYSIIS), ages are converted to inferred birth years, and place names are mapped to consistent geographic codes. Careful preprocessing is decisive, because most linkage error originates in inconsistent fields, and the quality of standardisation sets a ceiling on what any matching algorithm can achieve.
Formula
\text{standardize}(name, age, place) \to \text{comparable fields}Because comparing every record to every other is computationally infeasible at the scale of millions of entries, the data are blocked: partitioned into groups, by name sound, birth decade, and sex, within which candidate pairs are formed. Blocking trades completeness for tractability, since true matches split across blocks are lost. Choosing blocking keys that are stable yet discriminating, and sometimes using multiple passes, is a key design decision balancing recall against computational cost.
Formula
B = \text{partition by (Soundex(name), birth decade, sex)}Within each block, candidate record pairs are compared field by field, producing an agreement pattern: exact match, approximate match (by string distance), or disagreement on name, age, and place. These comparison vectors summarise the evidence that two records describe the same person. The granularity of comparison, exact versus fuzzy, and the choice of string metrics directly shape how tolerant the linkage is to the transcription errors endemic in historical sources.
Formula
\gamma_{ab} = (\text{agreement pattern on name, age, place})The Fellegi-Sunter model converts each agreement pattern into a match weight by summing, over fields, the log ratio of the probability of that agreement among true matches (m) to its probability among non-matches (u). Agreement on rare values earns high weight; agreement on common values little. These probabilities can be estimated from training data or via the EM algorithm. The total weight expresses how strongly the evidence favours the records being the same individual.
Formula
w_{ab} = \sum_f \log\frac{m_f}{u_f}\ \text{(Fellegi-Sunter weight)}Pairs are classified by comparing the weight to thresholds: above an upper threshold they are accepted as links, below a lower threshold rejected, and in between sent to clerical review or resolved by additional rules. Supervised machine learning can replace or augment this step, using a classifier trained on hand-linked examples to predict match probability. Threshold choice, or the classifier's decision rule, trades false matches against missed links and should reflect the downstream analysis's tolerance for each.
Formula
\text{link if } w_{ab} > T_u;\ \text{review if } T_l < w_{ab} \le T_uCrucially, linkage quality is measured against a hand-linked ground-truth sample. Positive predictive value (the share of accepted links that are correct), true positive rate (the share of true links recovered), and the overall match rate are reported. These metrics reveal both error types and any systematic bias, for instance, links skewing toward common names or non-migrants, so that downstream estimates can be corrected or bounded rather than naively trusted.
Formula
\text{PPV},\ \text{TPR},\ \text{match rate on a labeled training set}Common pitfalls
- Reporting a match rate without precision and recall on a ground-truth sample
- Ignoring that linked individuals differ systematically from the unlinked
- Over-tight thresholds that bias toward common names or non-migrants
- Treating linked data as error-free in downstream regressions
Frequently asked questions
Deterministic, probabilistic, or machine learning, which is best?
It depends on data quality and scale. Deterministic rules are transparent and adequate for small, clean sources but brittle to error. The probabilistic Fellegi-Sunter model handles noise principledly and is well understood. Supervised machine learning typically achieves the best precision-recall trade-off at scale but needs hand-linked training data. Modern historical practice often combines them and, regardless of choice, insists on measuring linkage error explicitly.
Why does linkage error matter for results?
False matches link different people, injecting noise and bias, while selective non-linkage means the linked sample is unrepresentative, typically over-representing people with stable, distinctive names who did not migrate. Both distort downstream estimates of mobility or migration. Reporting precision and recall on a labelled sample, and analysing who gets linked, lets researchers bound or correct these biases instead of presenting linked-data results as if exact.
What is blocking and why is it necessary?
Blocking partitions records into groups, by name sound, birth decade, sex, within which candidate pairs are compared, avoiding the impossible task of comparing every record to every other across millions of entries. It makes linkage computationally feasible but risks discarding true matches that fall into different blocks. Designers therefore choose blocking keys that are both stable across sources and discriminating, sometimes running several blocking passes to recover otherwise lost links.
Related methods
Sources
Abramitzky, R., Boustan, L., Eriksson, K., Feigenbaum, J., & Perez, S. (2021). Automated Linking of Historical Data. Journal of Economic Literature, 59(3), 865-918.
Open DOI referenceCopied.Feigenbaum, J. J. (2016). Automated Census Record Linking: A Machine Learning Approach. Working paper, Boston University.
Open publication referenceCopied.