On Calibration in Multi-Distribution Learning
This highlights a critical limitation for designing fair and robust predictors in MDL, which is incremental as it builds on existing MDL frameworks.
The paper studied calibration properties in multi-distribution learning (MDL), finding that while the Bayes optimal rule minimizes worst-case loss, it can cause non-uniform calibration errors across distributions and reveals a calibration-refinement trade-off.
Modern challenges of robustness, fairness, and decision-making in machine learning have led to the formulation of multi-distribution learning (MDL) frameworks in which a predictor is optimized across multiple distributions. We study the calibration properties of MDL to better understand how the predictor performs uniformly across the multiple distributions. Through classical results on decomposing proper scoring losses, we first derive the Bayes optimal rule for MDL, demonstrating that it maximizes the generalized entropy of the associated loss function. Our analysis reveals that while this approach ensures minimal worst-case loss, it can lead to non-uniform calibration errors across the multiple distributions and there is an inherent calibration-refinement trade-off, even at Bayes optimality. Our results highlight a critical limitation: despite the promise of MDL, one must use caution when designing predictors tailored to multiple distributions so as to minimize disparity.