VERITAS: Leveraging Vision Priors and Expert Fusion to Improve Multimodal Data
This work addresses data quality issues for researchers and practitioners using large multimodal models, though it is incremental as it builds on existing methods like vision priors and statistical fusion.
The paper tackles the problem of factual errors and hallucinations in supervised fine-tuning data for large multimodal models by proposing VERITAS, a pipeline that integrates vision priors and expert fusion to enhance data quality, resulting in models fine-tuned with VERITAS-processed data outperforming those using raw data across six benchmarks, with a lightweight critic model achieving comparable capability to state-of-the-art LMMs while being more efficient.
The quality of supervised fine-tuning (SFT) data is crucial for the performance of large multimodal models (LMMs), yet current data enhancement methods often suffer from factual errors and hallucinations due to inadequate visual perception. To address this challenge, we propose VERITAS, a pipeline that systematically integrates vision priors and multiple state-of-the-art LMMs with statistical methods to enhance SFT data quality. VERITAS leverages visual recognition models (RAM++) and OCR systems (PP-OCRv4) to extract structured vision priors, which are combined with images, questions, and answers. Three LMMs (GPT-4o, Gemini-2.5-Pro, Doubao-1.5-pro) evaluate the original answers, providing critique rationales and scores that are statistically fused into a high-confidence consensus score serving as ground truth. Using this consensus, we train a lightweight critic model via Group Relative Policy Optimization (GRPO), enhancing reasoning capabilities efficiently. Each LMM then refines the original answers based on the critiques, generating new candidate answers; we select the highest-scoring one as the final refined answer. Experiments across six multimodal benchmarks demonstrate that models fine-tuned with data processed by VERITAS consistently outperform those using raw data, particularly in text-rich and fine-grained reasoning tasks. Our critic model exhibits enhanced capability comparable to state-of-the-art LMMs while being significantly more efficient. We release our pipeline, datasets, and model checkpoints to advance research in multimodal data optimization.