Credal Ensemble Distillation for Uncertainty Quantification
This addresses the practical deployment challenge of uncertainty quantification methods for machine learning practitioners by offering a more efficient alternative.
The paper tackles the high computational and memory costs of deep ensembles for uncertainty quantification by proposing credal ensemble distillation (CED), which compresses an ensemble into a single model that predicts class-wise probability intervals, achieving superior or comparable uncertainty estimation on benchmarks while reducing inference overhead.
Deep ensembles (DE) have emerged as a powerful approach for quantifying predictive uncertainty and distinguishing its aleatoric and epistemic components, thereby enhancing model robustness and reliability. However, their high computational and memory costs during inference pose significant challenges for wide practical deployment. To overcome this issue, we propose credal ensemble distillation (CED), a novel framework that compresses a DE into a single model, CREDIT, for classification tasks. Instead of a single softmax probability distribution, CREDIT predicts class-wise probability intervals that define a credal set, a convex set of probability distributions, for uncertainty quantification. Empirical results on out-of-distribution detection benchmarks demonstrate that CED achieves superior or comparable uncertainty estimation compared to several existing baselines, while substantially reducing inference overhead compared to DE.