A Simple Framework for Uncertainty in Contrastive Learning
This addresses the problem of uncertainty quantification for contrastive learning models, which is incremental as it builds on existing contrastive approaches.
The paper tackles the lack of uncertainty estimation in contrastive learning by introducing a method to assign uncertainty to pretrained representations, showing improvements of up to 14% in anomaly detection across 11 tasks and competitive performance in out-of-distribution classification.
Contrastive approaches to representation learning have recently shown great promise. In contrast to generative approaches, these contrastive models learn a deterministic encoder with no notion of uncertainty or confidence. In this paper, we introduce a simple approach based on "contrasting distributions" that learns to assign uncertainty for pretrained contrastive representations. In particular, we train a deep network from a representation to a distribution in representation space, whose variance can be used as a measure of confidence. In our experiments, we show that this deep uncertainty model can be used (1) to visually interpret model behavior, (2) to detect new noise in the input to deployed models, (3) to detect anomalies, where we outperform 10 baseline methods across 11 tasks with improvements of up to 14% absolute, and (4) to classify out-of-distribution examples where our fully unsupervised model is competitive with supervised methods.