Online Missing Value Imputation and Change Point Detection with the Gaussian Copula
This work addresses the need for efficient and accurate imputation in streaming data applications, such as real-time analytics, but is incremental as it builds on existing copula-based methods.
The authors tackled the problem of online missing value imputation for mixed data types by developing a new algorithm using the Gaussian copula, which improved accuracy by adapting to changing distributions and sped up processing by up to an order of magnitude on large datasets, while also enabling change point detection in multivariate dependence structures with missing values.
Missing value imputation is crucial for real-world data science workflows. Imputation is harder in the online setting, as it requires the imputation method itself to be able to evolve over time. For practical applications, imputation algorithms should produce imputations that match the true data distribution, handle data of mixed types, including ordinal, boolean, and continuous variables, and scale to large datasets. In this work we develop a new online imputation algorithm for mixed data using the Gaussian copula. The online Gaussian copula model meets all the desiderata: its imputations match the data distribution even for mixed data, improve over its offline counterpart on the accuracy when the streaming data has a changing distribution, and on the speed (up to an order of magnitude) especially on large scale datasets. By fitting the copula model to online data, we also provide a new method to detect change points in the multivariate dependence structure with missing values. Experimental results on synthetic and real world data validate the performance of the proposed methods.