PClean: Bayesian Data Cleaning at Scale with Domain-Specific Probabilistic Programming
It addresses the problem of scalable and automated data cleaning for users dealing with large, error-prone datasets, representing a novel method for a known bottleneck.
The paper tackles automating Bayesian data cleaning by introducing PClean, a domain-specific probabilistic programming language, which enables faster and more accurate inference than generic PPLs, matches state-of-the-art systems in accuracy and runtime, and scales to millions of records.
Data cleaning is naturally framed as probabilistic inference in a generative model of ground-truth data and likely errors, but the diversity of real-world error patterns and the hardness of inference make Bayesian approaches difficult to automate. We present PClean, a probabilistic programming language (PPL) for leveraging dataset-specific knowledge to automate Bayesian cleaning. Compared to general-purpose PPLs, PClean tackles a restricted problem domain, enabling three modeling and inference innovations: (1) a non-parametric model of relational database instances, which users' programs customize; (2) a novel sequential Monte Carlo inference algorithm that exploits the structure of PClean's model class; and (3) a compiler that generates near-optimal SMC proposals and blocked-Gibbs rejuvenation kernels based on the user's model and data. We show empirically that short (< 50-line) PClean programs can: be faster and more accurate than generic PPL inference on data-cleaning benchmarks; match state-of-the-art data-cleaning systems in terms of accuracy and runtime (unlike generic PPL inference in the same runtime); and scale to real-world datasets with millions of records.