Illuminating Patterns of Divergence: DataDios SmartDiff for Large-Scale Data Difference Analysis
This addresses data quality issues for data engineers and analysts, offering a significant improvement over existing tools but is incremental in its approach.
The paper tackles the problem of reliable data differencing in data engineering workflows by introducing SmartDiff, a unified system that achieves over 95% precision and recall, runs 30-40% faster, uses 30-50% less memory, and reduces root-cause analysis time from 10 hours to 12 minutes.
Data engineering workflows require reliable differencing across files, databases, and query outputs, yet existing tools falter under schema drift, heterogeneous types, and limited explainability. SmartDiff is a unified system that combines schema-aware mapping, type-specific comparators, and parallel execution. It aligns evolving schemas, compares structured and semi-structured data (strings, numbers, dates, JSON/XML), and clusters results with labels that explain how and why differences occur. On multi-million-row datasets, SmartDiff achieves over 95 percent precision and recall, runs 30 to 40 percent faster, and uses 30 to 50 percent less memory than baselines; in user studies, it reduces root-cause analysis time from 10 hours to 12 minutes. An LLM-assisted labeling pipeline produces deterministic, schema-valid multilabel explanations using retrieval augmentation and constrained decoding; ablations show further gains in label accuracy and time to diagnosis over rules-only baselines. These results indicate SmartDiff's utility for migration validation, regression testing, compliance auditing, and continuous data quality monitoring. Index Terms: data differencing, schema evolution, data quality, parallel processing, clustering, explainable validation, big data