Multi-modal Preference Alignment Remedies Degradation of Visual Instruction Tuning on Language Models
This addresses performance degradation in MLLMs for multi-turn queries, offering an incremental improvement over existing alignment methods with minimal impact on visual knowledge.
The paper tackles the degradation of multi-modal large language models (MLLMs) after visual instruction tuning by proposing a distillation-based alignment method using a small, fine-grained VQA preference dataset. With Direct Preference Optimization (DPO), it surpasses baseline language models, achieving a 6.73 score on MT-Bench and boosting visual instruction performance by up to 6% on benchmarks.
Multi-modal large language models (MLLMs) are expected to support multi-turn queries of interchanging image and text modalities in production. However, the current MLLMs trained with visual-question-answering (VQA) datasets could suffer from degradation, as VQA datasets lack the diversity and complexity of the original text instruction datasets with which the underlying language model was trained. To address this degradation, we first collect a lightweight, 5k-sample VQA preference dataset where answers were annotated by Gemini for five quality metrics in a granular fashion and investigate standard Supervised Fine-tuning, rejection sampling, Direct Preference Optimization (DPO) and SteerLM algorithms. Our findings indicate that with DPO, we can surpass the instruction-following capabilities of the language model, achieving a 6.73 score on MT-Bench, compared to Vicuna's 6.57 and LLaVA's 5.99. This enhancement in textual instruction-following capability correlates with boosted visual instruction performance (+4.9\% on MM-Vet, +6\% on LLaVA-Bench), with minimal alignment tax on visual knowledge benchmarks compared to the previous RLHF approach. In conclusion, we propose a distillation-based multi-modal alignment model with fine-grained annotations on a small dataset that restores and boosts MLLM's language capability after visual instruction tuning.