Training, Architecture, and Prior for Deterministic Uncertainty Methods
It addresses uncertainty estimation for reliable ML models, but the findings are incremental, focusing on optimizing existing DUMs rather than introducing new paradigms.
This work investigates design choices in Deterministic Uncertainty Methods (DUMs) for efficient uncertainty estimation, showing that decoupled training schemes and core architecture expressiveness improve performance, while prior definitions have minimal impact.
Accurate and efficient uncertainty estimation is crucial to build reliable Machine Learning (ML) models capable to provide calibrated uncertainty estimates, generalize and detect Out-Of-Distribution (OOD) datasets. To this end, Deterministic Uncertainty Methods (DUMs) is a promising model family capable to perform uncertainty estimation in a single forward pass. This work investigates important design choices in DUMs: (1) we show that training schemes decoupling the core architecture and the uncertainty head schemes can significantly improve uncertainty performances. (2) we demonstrate that the core architecture expressiveness is crucial for uncertainty performance and that additional architecture constraints to avoid feature collapse can deteriorate the trade-off between OOD generalization and detection. (3) Contrary to other Bayesian models, we show that the prior defined by DUMs do not have a strong effect on the final performances.