Bridging the Visual Gap: Fine-Tuning Multimodal Models with Knowledge-Adapted Captions
This addresses a critical balance issue in fine-tuning multimodal models for researchers and practitioners, though it is incremental as it builds on existing data-centric approaches.
The paper tackles the problem of small-scale vision-language models hallucinating content when fine-tuned with detailed captions, and introduces Knowledge Adapted (KnowAda) fine-tuning to reduce hallucinations while preserving descriptiveness, showing it outperforms baselines in metrics and human evaluations.
Recent research increasingly focuses on training vision-language models (VLMs) with long, detailed image captions. However, small-scale VLMs often struggle to balance the richness of these captions with the risk of hallucinating content during fine-tuning. In this paper, we explore how well VLMs adapt to such captions. To quantify caption quality, we propose Decomposed NLI (DNLI), an evaluation framework that breaks down generated captions into individual propositions, assessing each in isolation. This fine-grained analysis reveals a critical balance between capturing descriptive details and preventing hallucinations. Our findings show that simply reducing caption complexity or employing standard data curation techniques does not effectively resolve this issue. To tackle this challenge, we introduce Knowledge Adapted (KnowAda) fine-tuning, a data-centric approach that automatically adapts training data with the model's existing knowledge and visual understanding. KnowAda minimizes hallucinations while preserving high descriptiveness. We validate this approach across several small-scale VLMs (up to 7B parameters) and dense caption datasets, demonstrating that KnowAda effectively balances hallucination reduction and descriptiveness. Our results show that KnowAda outperforms various baselines in both automatic metrics and human evaluations. We will release our code and models.