An Empirical Study Into What Matters for Calibrating Vision-Language Models
This work addresses the need for reliable uncertainty estimation in VLMs for risk-sensitive applications, though it is incremental as it applies existing calibration methods to a new model type.
The study investigated the calibration of Vision-Language Models (VLMs) for uncertainty estimation across different domains, datasets, and training strategies, finding that temperature scaling consistently improves calibration even under distribution shifts and with minimal data.
Vision-Language Models (VLMs) have emerged as the dominant approach for zero-shot recognition, adept at handling diverse scenarios and significant distribution changes. However, their deployment in risk-sensitive areas requires a deeper understanding of their uncertainty estimation capabilities, a relatively uncharted area. In this study, we explore the calibration properties of VLMs across different architectures, datasets, and training strategies. In particular, we analyze the uncertainty estimation performance of VLMs when calibrated in one domain, label set or hierarchy level, and tested in a different one. Our findings reveal that while VLMs are not inherently calibrated for uncertainty, temperature scaling significantly and consistently improves calibration, even across shifts in distribution and changes in label set. Moreover, VLMs can be calibrated with a very small set of examples. Through detailed experimentation, we highlight the potential applications and importance of our insights, aiming for more reliable and effective use of VLMs in critical, real-world scenarios.