Auto-Validate by-History: Auto-Program Data Quality Constraints to Validate Recurring Data Pipelines
This addresses the high human cost of manually monitoring data quality for data engineers in enterprises with large-scale pipeline operations, though it appears incremental as it builds on existing automation efforts.
The paper tackles the problem of data quality issues in recurring data pipelines by proposing Auto-Validate-by-History (AVH), which automatically detects these issues using historical statistics, with evaluations on 2000 production pipelines at Microsoft showing effectiveness and efficiency.
Data pipelines are widely employed in modern enterprises to power a variety of Machine-Learning (ML) and Business-Intelligence (BI) applications. Crucially, these pipelines are \emph{recurring} (e.g., daily or hourly) in production settings to keep data updated so that ML models can be re-trained regularly, and BI dashboards refreshed frequently. However, data quality (DQ) issues can often creep into recurring pipelines because of upstream schema and data drift over time. As modern enterprises operate thousands of recurring pipelines, today data engineers have to spend substantial efforts to \emph{manually} monitor and resolve DQ issues, as part of their DataOps and MLOps practices. Given the high human cost of managing large-scale pipeline operations, it is imperative that we can \emph{automate} as much as possible. In this work, we propose Auto-Validate-by-History (AVH) that can automatically detect DQ issues in recurring pipelines, leveraging rich statistics from historical executions. We formalize this as an optimization problem, and develop constant-factor approximation algorithms with provable precision guarantees. Extensive evaluations using 2000 production data pipelines at Microsoft demonstrate the effectiveness and efficiency of AVH.