EnsembleLink: Accurate Record Linkage Without Training Data
This addresses the challenge of accurate record linkage for empirical social science researchers, offering a method that eliminates the need for training data, which is a significant improvement over existing approaches.
The paper tackles the problem of record linkage without requiring labeled training data, presenting EnsembleLink, which uses pre-trained language models to achieve high accuracy, matching or exceeding methods that need extensive labeling across various benchmarks.
Record linkage, the process of matching records that refer to the same entity across datasets, is essential to empirical social science but remains methodologically underdeveloped. Researchers treat it as a preprocessing step, applying ad hoc rules without quantifying the uncertainty that linkage errors introduce into downstream analyses. Existing methods either achieve low accuracy or require substantial labeled training data. I present EnsembleLink, a method that achieves high accuracy without any training labels. EnsembleLink leverages pre-trained language models that have learned semantic relationships (e.g., that "South Ozone Park" is a neighborhood in "New York City" or that "Lutte ouvriere" refers to the Trotskyist "Workers' Struggle" party) from large text corpora. On benchmarks spanning city names, person names, organizations, multilingual political parties, and bibliographic records, EnsembleLink matches or exceeds methods requiring extensive labeling. The method runs locally on open-source models, requiring no external API calls, and completes typical linkage tasks in minutes.