Deep Deterministic Uncertainty: A Simple Baseline
This provides a computationally efficient solution for uncertainty estimation in machine learning, particularly for applications like out-of-distribution detection and active learning, though it is incremental as it builds on existing regularization techniques.
The paper tackles the problem of reliable uncertainty quantification in deterministic single-forward-pass models by showing that a simple baseline method, using a well-regularized feature space with Gaussian Discriminant Analysis, outperforms more complex approaches like DUQ and SNGP and matches state-of-the-art Deep Ensembles on out-of-distribution benchmarks while being computationally cheaper.
Reliable uncertainty from deterministic single-forward pass models is sought after because conventional methods of uncertainty quantification are computationally expensive. We take two complex single-forward-pass uncertainty approaches, DUQ and SNGP, and examine whether they mainly rely on a well-regularized feature space. Crucially, without using their more complex methods for estimating uncertainty, a single softmax neural net with such a feature-space, achieved via residual connections and spectral normalization, *outperforms* DUQ and SNGP's epistemic uncertainty predictions using simple Gaussian Discriminant Analysis *post-training* as a separate feature-space density estimator -- without fine-tuning on OoD data, feature ensembling, or input pre-procressing. This conceptually simple *Deep Deterministic Uncertainty (DDU)* baseline can also be used to disentangle aleatoric and epistemic uncertainty and performs as well as Deep Ensembles, the state-of-the art for uncertainty prediction, on several OoD benchmarks (CIFAR-10/100 vs SVHN/Tiny-ImageNet, ImageNet vs ImageNet-O) as well as in active learning settings across different model architectures, yet is *computationally cheaper*.