Epistemic Uncertainty Quantification for Pre-trained VLMs via Riemannian Flow Matching
This addresses the lack of uncertainty quantification in VLMs, enabling improved reliability for tasks like out-of-distribution detection and data curation, though it is incremental as it builds on existing flow matching techniques.
The paper tackled the problem of quantifying epistemic uncertainty in deterministic Vision-Language Models (VLMs) by proposing REPVLM, which uses Riemannian Flow Matching to compute probability densities on hyperspherical embeddings, achieving near-perfect correlation between uncertainty and prediction error.
Vision-Language Models (VLMs) are typically deterministic in nature and lack intrinsic mechanisms to quantify epistemic uncertainty, which reflects the model's lack of knowledge or ignorance of its own representations. We theoretically motivate negative log-density of an embedding as a proxy for the epistemic uncertainty, where low-density regions signify model ignorance. The proposed method REPVLM computes the probability density on the hyperspherical manifold of the VLM embeddings using Riemannian Flow Matching. We empirically demonstrate that REPVLM achieves near-perfect correlation between uncertainty and prediction error, significantly outperforming existing baselines. Beyond classification, we also demonstrate that the model also provides a scalable metric for out-of-distribution detection and automated data curation.