DIM-SUM: Dynamic IMputation for Smart Utility Management
This addresses the challenge for practitioners in infrastructure monitoring who deal with datasets with large, complex missing data, though it appears incremental as it builds on existing imputation methods with a new preprocessing framework.
The paper tackles the problem of training robust imputation models for real-world infrastructure monitoring where data has complex, heterogeneous missing patterns, and demonstrates that DIM-SUM outperforms traditional methods by achieving similar accuracy with lower processing time and less training data, and averages 2x higher accuracy with less inference time compared to a large pre-trained model.
Time series imputation models have traditionally been developed using complete datasets with artificial masking patterns to simulate missing values. However, in real-world infrastructure monitoring, practitioners often encounter datasets where large amounts of data are missing and follow complex, heterogeneous patterns. We introduce DIM-SUM, a preprocessing framework for training robust imputation models that bridges the gap between artificially masked training data and real missing patterns. DIM-SUM combines pattern clustering and adaptive masking strategies with theoretical learning guarantees to handle diverse missing patterns actually observed in the data. Through extensive experiments on over 2 billion readings from California water districts, electricity datasets, and benchmarks, we demonstrate that DIM-SUM outperforms traditional methods by reaching similar accuracy with lower processing time and significantly less training data. When compared against a large pre-trained model, DIM-SUM averages 2x higher accuracy with significantly less inference time.