Scoring Rules and Calibration for Imprecise Probabilities
This work addresses a theoretical gap for researchers and practitioners in fields like machine learning and statistics who deal with uncertain or imprecise probability estimates, though it is incremental as it builds on existing precise probability theory.
The paper tackles the lack of theoretical foundations for evaluating imprecise probabilistic forecasts, such as intervals like 20-30% probability, by generalizing proper scoring rules and calibration to this case, linking it to distributional robustness and revealing pitfalls in loss function choices in machine learning practice.
What does it mean to say that, for example, the probability for rain tomorrow is between 20% and 30%? The theory for the evaluation of precise probabilistic forecasts is well-developed and is grounded in the key concepts of proper scoring rules and calibration. For the case of imprecise probabilistic forecasts (sets of probabilities), such theory is still lacking. In this work, we therefore generalize proper scoring rules and calibration to the imprecise case. We develop these concepts as relative to data models and decision problems. As a consequence, the imprecision is embedded in a clear context. We establish a close link to the paradigm of (group) distributional robustness and in doing so provide new insights for it. We argue that proper scoring rules and calibration serve two distinct goals, which are aligned in the precise case, but intriguingly are not necessarily aligned in the imprecise case. The concept of decision-theoretic entropy plays a key role for both goals. Finally, we demonstrate the theoretical insights in machine learning practice, in particular we illustrate subtle pitfalls relating to the choice of loss function in distributional robustness.