Efficient Post-Hoc Uncertainty Calibration via Variance-Based Smoothing
This work addresses the problem of efficient uncertainty calibration for practitioners in machine learning, offering an incremental improvement by integrating prior information into existing methods.
The paper tackles the computational cost of uncertainty estimation in deep neural networks by proposing a variance-based scaling framework that uses variance between sub-predictions as a proxy for uncertainty, achieving competitive uncertainty estimates with lower computational demands and enabling post-hoc calibration.
Since state-of-the-art uncertainty estimation methods are often computationally demanding, we investigate whether incorporating prior information can improve uncertainty estimates in conventional deep neural networks. Our focus is on machine learning tasks where meaningful predictions can be made from sub-parts of the input. For example, in speaker classification, the speech waveform can be divided into sequential patches, each containing information about the same speaker. We observe that the variance between sub-predictions serves as a reliable proxy for uncertainty in such settings. Our proposed variance-based scaling framework produces competitive uncertainty estimates in classification while being less computationally demanding and allowing for integration as a post-hoc calibration tool. This approach also leads to a simple extension of deep ensembles, improving the expressiveness of their predicted distributions.