When Pattern-by-Pattern Works: Theoretical and Empirical Insights for Logistic Models with Missing Values
This work addresses the problem of making accurate predictions in logistic regression when data has missing values, which is incremental but provides practical guidance for statisticians and data scientists.
The paper tackles prediction in logistic models with missing inputs by proving that a Pattern-by-Pattern (PbP) strategy approximates Bayes probabilities under various missing data scenarios, and empirically shows that PbP is best for large samples with Gaussian mixtures while MICE.RF.Y performs better with nonlinear features.
Predicting a response with partially missing inputs remains a challenging task even in parametric models, since parameter estimation in itself is not sufficient to predict on partially observed inputs. Several works study prediction in linear models. In this paper, we focus on logistic models, which present their own difficulties. From a theoretical perspective, we prove that a Pattern-by-Pattern strategy (PbP), which learns one logistic model per missingness pattern, accurately approximates Bayes probabilities in various missing data scenarios (MCAR, MAR and MNAR). Empirically, we thoroughly compare various methods (constant and iterative imputations, complete case analysis, PbP, and an EM algorithm) across classification, probability estimation, calibration, and parameter inference. Our analysis provides a comprehensive view on the logistic regression with missing values. It reveals that mean imputation can be used as baseline for low sample sizes, and improved performance is obtained via nonlinear multiple iterative imputation techniques with the labels (MICE.RF.Y). For large sample sizes, PbP is the best method for Gaussian mixtures, and we recommend MICE.RF.Y in presence of nonlinear features.