Fine-Grained Verifiers: Preference Modeling as Next-token Prediction in Vision-Language Alignment
This addresses the challenge of scalable and effective alignment in VLLMs for applications requiring precise vision-language integration, representing an incremental advancement with a novel token-level approach.
The paper tackles the problem of modality misalignment in vision-language large models (VLLMs), which causes issues like hallucinations, by proposing FiSAO, a self-alignment method that uses the model's own visual encoder as a fine-grained verifier to provide token-level feedback, resulting in significant improvements in alignment without needing additional data and surpassing traditional preference tuning methods.
The recent advancements in large language models (LLMs) and pre-trained vision models have accelerated the development of vision-language large models (VLLMs), enhancing the interaction between visual and linguistic modalities. Despite their notable success across various domains, VLLMs face challenges in modality alignment, which can lead to issues like hallucinations and unsafe content generation. Current alignment techniques often rely on coarse feedback and external datasets, limiting scalability and performance. In this paper, we propose FiSAO (Fine-Grained Self-Alignment Optimization), a novel self-alignment method that utilizes the model's own visual encoder as a fine-grained verifier to improve vision-language alignment without the need for additional data. By leveraging token-level feedback from the vision encoder, FiSAO significantly improves vision-language alignment, even surpassing traditional preference tuning methods that require additional data. Through both theoretical analysis and experimental validation, we demonstrate that FiSAO effectively addresses the misalignment problem in VLLMs, marking the first instance of token-level rewards being applied to such models.