Aligning Evaluation with Clinical Priorities: Calibration, Label Shift, and Error Costs
This work addresses the misalignment of evaluation metrics with clinical needs for decision support systems, offering a practical solution for healthcare applications.
The authors tackled the problem that standard evaluation metrics like accuracy and AUC-ROC do not align with clinical priorities such as calibration, robustness to distribution shifts, and asymmetric error costs. They proposed a principled evaluation framework based on adjusted cross-entropy to select calibrated thresholded classifiers, resulting in a simple, sensitive method for clinical deployment.
Machine learning-based decision support systems are increasingly deployed in clinical settings, where probabilistic scoring functions are used to inform and prioritize patient management decisions. However, widely used scoring rules, such as accuracy and AUC-ROC, fail to adequately reflect key clinical priorities, including calibration, robustness to distributional shifts, and sensitivity to asymmetric error costs. In this work, we propose a principled yet practical evaluation framework for selecting calibrated thresholded classifiers that explicitly accounts for the uncertainty in class prevalences and domain-specific cost asymmetries often found in clinical settings. Building on the theory of proper scoring rules, particularly the Schervish representation, we derive an adjusted variant of cross-entropy (log score) that averages cost-weighted performance over clinically relevant ranges of class balance. The resulting evaluation is simple to apply, sensitive to clinical deployment conditions, and designed to prioritize models that are both calibrated and robust to real-world variations.