Analyzing and Improving Fine-grained Preference Optimization in Medical LVLMs
For developers of medical LVLMs, this work addresses critical alignment issues that hinder clinical reliability, offering a more precise optimization approach than existing DPO variants.
The paper identifies three key limitations of existing preference optimization methods for medical LVLMs (token-level reward neglect, off-policy distribution shift, and lack of visual grounding) and proposes a fine-grained alignment framework combining bidirectional token-wise KL regularization and visual-contrastive grounding. The method achieves improved clinical correctness and visual grounding across medical imaging tasks and clinical text generation benchmarks.
Large Vision-Language Models (LVLMs) have achieved strong performance across medical imaging tasks, yet they remain prone to factual inconsistencies, poor visual grounding, and misalignment with clinically meaningful feedback. Existing post-training alignment approaches, including Direct Preference Optimization (DPO) and its variants, face three critical limitations in the medical domain: (1) sequence-level reward signals treat clinically critical tokens identically to generic filler text; (2) reliance on static supervised fine-tuning references as preferred responses introduces an off-policy distribution shift, steering optimization toward stylistic artifacts over clinical correctness; and (3) alignment objectives lack explicit visual grounding constraints, leaving models insensitive to subtle yet diagnostically decisive pathological features. Our method leverages a bidirectional token-wise KL regularizer alongside a visual-contrastive grounding objective that pairs clean and lesion-corrupted images to penalize responses generated without adequate visual evidence. Together, these components form a fine-grained, on-policy alignment framework that constructs preference pairs by minimally editing model-generated outputs, correcting only clinically erroneous spans while preserving the original linguistic style. Extensive experiments across medical imaging tasks and clinical text generation benchmarks validate the effectiveness of our approach.