A Rate-Distortion View of Uncertainty Quantification
This addresses uncertainty quantification for deep learning models, particularly in safety-critical applications, though it builds on existing information bottleneck approaches.
The paper tackles the problem of deep neural networks lacking uncertainty estimates for inputs far from training data by introducing Distance Aware Bottleneck (DAB), which learns a compressed codebook to measure distance and provide uncertainty estimates, achieving better out-of-distribution detection and misclassification prediction than prior methods.
In supervised learning, understanding an input's proximity to the training data can help a model decide whether it has sufficient evidence for reaching a reliable prediction. While powerful probabilistic models such as Gaussian Processes naturally have this property, deep neural networks often lack it. In this paper, we introduce Distance Aware Bottleneck (DAB), i.e., a new method for enriching deep neural networks with this property. Building on prior information bottleneck approaches, our method learns a codebook that stores a compressed representation of all inputs seen during training. The distance of a new example from this codebook can serve as an uncertainty estimate for the example. The resulting model is simple to train and provides deterministic uncertainty estimates by a single forward pass. Finally, our method achieves better out-of-distribution (OOD) detection and misclassification prediction than prior methods, including expensive ensemble methods, deep kernel Gaussian Processes, and approaches based on the standard information bottleneck.