MACEst: The reliable and trustworthy Model Agnostic Confidence Estimator
This addresses the need for reliable confidence estimates in any machine learning model to be truly useful, representing a novel approach rather than an incremental improvement.
The paper tackles the problem of unreliable confidence estimates in machine learning models, especially under high epistemic uncertainty, by introducing MACEst, a model-agnostic estimator that provides trustworthy confidence estimates by accounting for both aleatoric and epistemic uncertainty locally.
Reliable Confidence Estimates are hugely important for any machine learning model to be truly useful. In this paper, we argue that any confidence estimates based upon standard machine learning point prediction algorithms are fundamentally flawed and under situations with a large amount of epistemic uncertainty are likely to be untrustworthy. To address these issues, we present MACEst, a Model Agnostic Confidence Estimator, which provides reliable and trustworthy confidence estimates. The algorithm differs from current methods by estimating confidence independently as a local quantity which explicitly accounts for both aleatoric and epistemic uncertainty. This approach differs from standard calibration methods that use a global point prediction model as a starting point for the confidence estimate.