Performance Bounds for Pairwise Entity Resolution
This work addresses a key bottleneck in data integration for researchers and practitioners, though it is incremental as it builds on existing pairwise entity resolution methods.
The paper tackles the problem of scaling entity resolution algorithms to large datasets by proving performance bounds between small validation sets and arbitrarily sized datasets, enabling optimization using limited labeled data.
One significant challenge to scaling entity resolution algorithms to massive datasets is understanding how performance changes after moving beyond the realm of small, manually labeled reference datasets. Unlike traditional machine learning tasks, when an entity resolution algorithm performs well on small hold-out datasets, there is no guarantee this performance holds on larger hold-out datasets. We prove simple bounding properties between the performance of a match function on a small validation set and the performance of a pairwise entity resolution algorithm on arbitrarily sized datasets. Thus, our approach enables optimization of pairwise entity resolution algorithms for large datasets, using a small set of labeled data.