Importance Sampling for Multi-Negative Multimodal Direct Preference Optimization
This addresses optimization bias and hallucinations in vision-language models for researchers and practitioners in multimodal AI, representing an incremental advancement over existing DPO methods.
The paper tackled the problem of oversimplified pairwise comparisons in multimodal Direct Preference Optimization (DPO) by proposing MISP-DPO, which incorporates multiple, semantically diverse negative images via the Plackett-Luce model and importance sampling, resulting in improved multimodal alignment across five benchmarks.
Direct Preference Optimization (DPO) has recently been extended from text-only models to vision-language models. However, existing methods rely on oversimplified pairwise comparisons, generating a single negative image via basic perturbations or similarity-based retrieval, which fail to capture the complex nature of multimodal preferences, inducing optimization bias and hallucinations. To address this issue, we propose MISP-DPO, the first framework to incorporate multiple, semantically diverse negative images in multimodal DPO via the Plackett-Luce model. Our method embeds prompts and candidate images in CLIP (Contrastive Language-Image Pretraining) space and applies a sparse autoencoder to uncover semantic deviations into interpretable factors. Negative samples are selected based on reconstruction difficulty, semantic deviation from the positive, and mutual diversity, yielding broader and more informative supervision. To handle multi-negative comparisons, we adopt a Plackett-Luce objective and introduce an importance sampling strategy that improves training efficiency. Experiments across five diverse benchmarks demonstrate that MISP-DPO consistently improves multimodal alignment over prior methods, validating the effectiveness of semantic-aware, multi-negative sampling in preference-based learning.