Supervised Negative Binomial Classifier for Probabilistic Record Linkage
This addresses the problem of linking records across databases for data integration tasks, though it is incremental in its approach.
The paper tackles probabilistic record linkage by proposing a novel graphical model classifier using a mixture of Poisson distributions with latent variables and gamma priors, achieving a method that handles sparse and streaming data effectively.
Motivated by the need of the linking records across various databases, we propose a novel graphical model based classifier that uses a mixture of Poisson distributions with latent variables. The idea is to derive insight into each pair of hypothesis records that match by inferring its underlying latent rate of error using Bayesian Modeling techniques. The novel approach of using gamma priors for learning the latent variables along with supervised labels is unique and allows for active learning. The naive assumption is made deliberately as to the independence of the fields to propose a generalized theory for this class of problems and not to undermine the hierarchical dependencies that could be present in different scenarios. This classifier is able to work with sparse and streaming data. The application to record linkage is able to meet several challenges of sparsity, data streams and varying nature of the data-sets.