A Unified Evaluation Framework for Epistemic Predictions
This work addresses the need for standardized evaluation in uncertainty-aware machine learning, though it appears incremental as it builds on existing methods without introducing a new paradigm.
The authors tackled the problem of evaluating uncertainty-aware classifiers by proposing a unified evaluation framework that allows users to tailor the trade-off between accuracy and precision, with experiments on datasets like CIFAR-10 and MNIST showing the metric behaves as desired.
Predictions of uncertainty-aware models are diverse, ranging from single point estimates (often averaged over prediction samples) to predictive distributions, to set-valued or credal-set representations. We propose a novel unified evaluation framework for uncertainty-aware classifiers, applicable to a wide range of model classes, which allows users to tailor the trade-off between accuracy and precision of predictions via a suitably designed performance metric. This makes possible the selection of the most suitable model for a particular real-world application as a function of the desired trade-off. Our experiments, concerning Bayesian, ensemble, evidential, deterministic, credal and belief function classifiers on the CIFAR-10, MNIST and CIFAR-100 datasets, show that the metric behaves as desired.