Revisiting Visual Understanding in Multimodal Reasoning through a Lens of Image Perturbation
This addresses the visual processing bottleneck in multimodal reasoning for AI researchers, though it appears incremental as it enhances existing methods without fundamental changes.
The paper tackles the problem that multimodal large language models (MLLMs) may generate accurate visual descriptions but fail to effectively integrate them during reasoning, finding that language-only models with image captions can outperform MLLMs. The result is that a simple visual perturbation framework improves mathematical reasoning performance with gains comparable to algorithmic changes, achieving competitive performance among open-source 7B RL-tuned models.
Despite the rapid progress of multimodal large language models (MLLMs), they have largely overlooked the importance of visual processing. In a simple yet revealing experiment, we interestingly find that language-only models, when provided with image captions, can achieve comparable or even better performance than MLLMs that consume raw visual inputs. This suggests that current MLLMs may generate accurate visual descriptions but fail to effectively integrate them during reasoning. Motivated by this, we propose a simple visual perturbation framework that enhances perceptual robustness without requiring algorithmic modifications or additional training data. Our approach introduces three targeted perturbations: distractor concatenation, dominance-preserving mixup, and random rotation, that can be easily integrated into existing post-training pipelines including SFT, DPO, and GRPO. Through extensive experiments across multiple datasets, we demonstrate consistent improvements in mathematical reasoning performance, with gains comparable to those achieved through algorithmic changes. Additionally, we achieve competitive performance among open-source 7B RL-tuned models by training Qwen2.5-VL-7B with visual perturbation. Through comprehensive ablation studies, we analyze the effectiveness of different perturbation strategies, revealing that each perturbation type contributes uniquely to different aspects of visual reasoning. Our findings highlight the critical role of visual perturbation in multimodal mathematical reasoning: better reasoning begins with better seeing. Our code is available at https://github.com/YutingLi0606/Vision-Matters.