Learning Over Dirty Data Without Cleaning
This addresses the challenge for data scientists and analysts who must clean data before learning, potentially saving time and effort, though it appears incremental as it builds on existing relational learning with constraints.
The paper tackles the problem of learning from dirty databases containing errors like integrity violations and duplicates, which typically require time-consuming cleaning. The result is DLearn, a relational learning system that learns directly over dirty data without preprocessing, achieving accurate models efficiently on large real-world databases.
Real-world datasets are dirty and contain many errors. Examples of these issues are violations of integrity constraints, duplicates, and inconsistencies in representing data values and entities. Learning over dirty databases may result in inaccurate models. Users have to spend a great deal of time and effort to repair data errors and create a clean database for learning. Moreover, as the information required to repair these errors is not often available, there may be numerous possible clean versions for a dirty database. We propose DLearn, a novel relational learning system that learns directly over dirty databases effectively and efficiently without any preprocessing. DLearn leverages database constraints to learn accurate relational models over inconsistent and heterogeneous data. Its learned models represent patterns over all possible clean instances of the data in a usable form. Our empirical study indicates that DLearn learns accurate models over large real-world databases efficiently.