Auto-Validate: Unsupervised Data Validation Using Data-Domain Patterns Inferred from Data Lakes
This work addresses data quality issues in complex production pipelines for BI reporting and ML modeling, though it appears incremental as it builds on existing validation efforts by improving accuracy for specific data types.
The paper tackles the problem of high false-positive rates in data validation for string-valued machine-generated data by developing Auto-Validate, which infers data-domain patterns from data lakes to minimize false positives while catching data quality issues. Evaluations on production data show it is substantially more effective than existing methods, with part of the technology shipping in Microsoft Azure Purview.
Complex data pipelines are increasingly common in diverse applications such as BI reporting and ML modeling. These pipelines often recur regularly (e.g., daily or weekly), as BI reports need to be refreshed, and ML models need to be retrained. However, it is widely reported that in complex production pipelines, upstream data feeds can change in unexpected ways, causing downstream applications to break silently that are expensive to resolve. Data validation has thus become an important topic, as evidenced by notable recent efforts from Google and Amazon, where the objective is to catch data quality issues early as they arise in the pipelines. Our experience on production data suggests, however, that on string-valued data, these existing approaches yield high false-positive rates and frequently require human intervention. In this work, we develop a corpus-driven approach to auto-validate \emph{machine-generated data} by inferring suitable data-validation "patterns" that accurately describe the underlying data domain, which minimizes false positives while maximizing data quality issues caught. Evaluations using production data from real data lakes suggest that Auto-Validate is substantially more effective than existing methods. Part of this technology ships as an Auto-Tag feature in Microsoft Azure Purview.