ASPO: Adaptive Sentence-Level Preference Optimization for Fine-Grained Multimodal Reasoning
This work addresses the issue of suboptimal solutions in multimodal alignment for researchers and practitioners, though it is incremental as it builds on existing DPO methods.
The paper tackled the problem of fine-grained multimodal reasoning by addressing the lack of fine-grained supervision in Direct Preference Optimization (DPO), proposing Adaptive Sentence-level Preference Optimization (ASPO) to evaluate individual sentences, which significantly improved multimodal model performance.
Direct Preference Optimization (DPO) has gained significant attention for its simplicity and computational efficiency in aligning large language models (LLMs). Recent advancements have extended DPO to multimodal scenarios, achieving strong performance. However, traditional DPO relies on binary preference optimization, rewarding or penalizing entire responses without considering fine-grained segment correctness, leading to suboptimal solutions. The root of this issue lies in the absence of fine-grained supervision during the optimization process. To address this, we propose Adaptive Sentence-level Preference Optimization (ASPO), which evaluates individual sentences for more precise preference optimization. By dynamically calculating adaptive rewards at the sentence level based on model predictions, ASPO enhances response content assessment without additional models or parameters. This significantly improves the alignment of multimodal features. Extensive experiments show that ASPO substantially enhances the overall performance of multimodal models.