Regretful Decisions under Label Noise
This addresses the issue of model reliability and adoption in high-stakes decisions like healthcare, where label noise can lead to unfair or harmful outcomes for individuals, representing an incremental improvement in handling label noise.
The paper tackles the problem of machine learning models making unforeseen mistakes at the individual level due to label noise in datasets, and presents an approach to estimate the likelihood of such mistakes, supported by empirical studies in clinical prediction tasks.
Machine learning models are routinely used to support decisions that affect individuals -- be it to screen a patient for a serious illness or to gauge their response to treatment. In these tasks, we are limited to learning models from datasets with noisy labels. In this paper, we study the instance-level impact of learning under label noise. We introduce a notion of regret for this regime, which measures the number of unforeseen mistakes due to noisy labels. We show that standard approaches to learning under label noise can return models that perform well at a population-level while subjecting individuals to a lottery of mistakes. We present a versatile approach to estimate the likelihood of mistakes at the individual-level from a noisy dataset by training models over plausible realizations of datasets without label noise. This is supported by a comprehensive empirical study of label noise in clinical prediction tasks. Our results reveal how failure to anticipate mistakes can compromise model reliability and adoption -- we demonstrate how we can address these challenges by anticipating and avoiding regretful decisions.