How to show a probabilistic model is better
This work addresses the need for standardized evaluation methods in empirical machine learning, though it is incremental as it adapts existing statistical concepts.
The paper tackles the problem of comparing probabilistic models in machine learning by presenting a simple theoretical framework and practical procedures based on proper scoring rules, aiming to facilitate wider adoption for performance evaluation.
We present a simple theoretical framework, and corresponding practical procedures, for comparing probabilistic models on real data in a traditional machine learning setting. This framework is based on the theory of proper scoring rules, but requires only basic algebra and probability theory to understand and verify. The theoretical concepts presented are well-studied, primarily in the statistics literature. The goal of this paper is to advocate their wider adoption for performance evaluation in empirical machine learning.