Model Evaluation in the Dark: Robust Classifier Metrics with Missing Labels
This addresses a specific oversight in model evaluation for practitioners dealing with incomplete datasets, though it is incremental as it builds on existing imputation techniques.
The paper tackles the problem of missing labels during classifier evaluation, which can bias metrics like precision and recall, especially with Missing Not At Random data, by proposing a multiple imputation method that provides point estimates and predictive distributions, showing empirically correct distributions and theoretical convergence bounds.
Missing data in supervised learning is well-studied, but the specific issue of missing labels during model evaluation has been overlooked. Ignoring samples with missing values, a common solution, can introduce bias, especially when data is Missing Not At Random (MNAR). We propose a multiple imputation technique for evaluating classifiers using metrics such as precision, recall, and ROC-AUC. This method not only offers point estimates but also a predictive distribution for these quantities when labels are missing. We empirically show that the predictive distribution's location and shape are generally correct, even in the MNAR regime. Moreover, we establish that this distribution is approximately Gaussian and provide finite-sample convergence bounds. Additionally, a robustness proof is presented, confirming the validity of the approximation under a realistic error model.