Bayesian analysis of the prevalence bias: learning and predicting from imbalanced data
This addresses sampling bias issues in machine learning, particularly for medical or imbalanced data applications, offering a principled solution to enhance real-world deployment, though it is incremental as it builds on existing Bayesian and loss correction methods.
The paper tackles the problem of prevalence bias in datasets, where the sampling rate of a pathology differs from its real-world prevalence, by developing a Bayesian risk minimization framework that yields a bias-corrected loss function and predictive rules, resulting in improved model performance without specifying concrete numbers.
Datasets are rarely a realistic approximation of the target population. Say, prevalence is misrepresented, image quality is above clinical standards, etc. This mismatch is known as sampling bias. Sampling biases are a major hindrance for machine learning models. They cause significant gaps between model performance in the lab and in the real world. Our work is a solution to prevalence bias. Prevalence bias is the discrepancy between the prevalence of a pathology and its sampling rate in the training dataset, introduced upon collecting data or due to the practioner rebalancing the training batches. This paper lays the theoretical and computational framework for training models, and for prediction, in the presence of prevalence bias. Concretely a bias-corrected loss function, as well as bias-corrected predictive rules, are derived under the principles of Bayesian risk minimization. The loss exhibits a direct connection to the information gain. It offers a principled alternative to heuristic training losses and complements test-time procedures based on selecting an operating point from summary curves. It integrates seamlessly in the current paradigm of (deep) learning using stochastic backpropagation and naturally with Bayesian models.