Calibrating for Class Weights by Modeling Machine Learning
This addresses calibration issues in imbalanced or cost-sensitive learning scenarios, but is incremental as it builds on existing models.
The paper tackles the incompatibility between calibration and class weighting in machine learning, proposing a model-based method to recover likelihoods from miscalibrated algorithms, and validates it on a pneumonia detection task.
A much studied issue is the extent to which the confidence scores provided by machine learning algorithms are calibrated to ground truth probabilities. Our starting point is that calibration is seemingly incompatible with class weighting, a technique often employed when one class is less common (class imbalance) or with the hope of achieving some external objective (cost-sensitive learning). We provide a model-based explanation for this incompatibility and use our anthropomorphic model to generate a simple method of recovering likelihoods from an algorithm that is miscalibrated due to class weighting. We validate this approach in the binary pneumonia detection task of Rajpurkar, Irvin, Zhu, et al. (2017).