OVFact: Measuring and Improving Open-Vocabulary Factuality for Long Caption Models
This addresses the challenge of evaluating and improving factuality in long captions for vision-language models, which is incremental as it builds on existing metrics but introduces novel applications like data filtering.
The paper tackles the problem of generating factual long captions in vision-language models by introducing OV-Fact, a reference-free method that improves agreement with human judgments by 15% and enables data filtering, leading to models trained on filtered subsets achieving up to 30% higher factuality precision without loss in descriptiveness.
Large vision-language models (VLMs) often struggle to generate long and factual captions. However, traditional measures for hallucination and factuality are not well suited for evaluating longer, more diverse captions and in settings where ground-truth human-annotated captions are unavailable. We introduce OV-Fact, a novel method for measuring caption factuality of long captions that leverages open-vocabulary visual grounding and tool-based verification without depending on human annotations. Our method improves agreement with human judgments and captures both caption descriptiveness (recall) and factual precision in the same metric. Furthermore, unlike previous metrics, our reference-free method design enables new applications towards factuality-based data filtering. We observe models trained on an OVFact-filtered (2.5-5x less) subset of a large-scale, noisy (VLM-generated) pretraining set meaningfully improve factuality precision without sacrificing caption descriptiveness across a range of downstream long caption benchmarks.