Matrix Completion with Quantified Uncertainty through Low Rank Gaussian Copula
This addresses the lack of uncertainty estimation in existing imputation methods for tabular data, which is crucial for applications requiring reliable predictions, though it is incremental by augmenting a standard probabilistic model.
The paper tackles the problem of missing value imputation in large-scale datasets by proposing a probabilistic framework that quantifies uncertainty for each imputation, achieving state-of-the-art accuracy across various data types and showing that uncertainty measures predict imputation error well.
Modern large scale datasets are often plagued with missing entries. For tabular data with missing values, a flurry of imputation algorithms solve for a complete matrix which minimizes some penalized reconstruction error. However, almost none of them can estimate the uncertainty of its imputations. This paper proposes a probabilistic and scalable framework for missing value imputation with quantified uncertainty. Our model, the Low Rank Gaussian Copula, augments a standard probabilistic model, Probabilistic Principal Component Analysis, with marginal transformations for each column that allow the model to better match the distribution of the data. It naturally handles Boolean, ordinal, and real-valued observations and quantifies the uncertainty in each imputation. The time required to fit the model scales linearly with the number of rows and the number of columns in the dataset. Empirical results show the method yields state-of-the-art imputation accuracy across a wide range of data types, including those with high rank. Our uncertainty measure predicts imputation error well: entries with lower uncertainty do have lower imputation error (on average). Moreover, for real-valued data, the resulting confidence intervals are well-calibrated.