URL: A Representation Learning Benchmark for Transferable Uncertainty Estimates
This work addresses the need for reliable machine learning with uncertainty quantification in representation learning, but it is incremental as it focuses on benchmarking existing methods rather than introducing a new solution.
The authors tackled the problem of developing pretrained models that provide transferable uncertainty estimates by proposing the Uncertainty-aware Representation Learning (URL) benchmark, which evaluates eleven uncertainty quantifiers on eight downstream datasets and finds that certain approaches outperform others, though achieving transferable uncertainty remains a challenge.
Representation learning has significantly driven the field to develop pretrained models that can act as a valuable starting point when transferring to new datasets. With the rising demand for reliable machine learning and uncertainty quantification, there is a need for pretrained models that not only provide embeddings but also transferable uncertainty estimates. To guide the development of such models, we propose the Uncertainty-aware Representation Learning (URL) benchmark. Besides the transferability of the representations, it also measures the zero-shot transferability of the uncertainty estimate using a novel metric. We apply URL to evaluate eleven uncertainty quantifiers that are pretrained on ImageNet and transferred to eight downstream datasets. We find that approaches that focus on the uncertainty of the representation itself or estimate the prediction risk directly outperform those that are based on the probabilities of upstream classes. Yet, achieving transferable uncertainty quantification remains an open challenge. Our findings indicate that it is not necessarily in conflict with traditional representation learning goals. Code is provided under https://github.com/mkirchhof/url .