Deep But Reliable: Advancing Multi-turn Reasoning for Thinking with Images
This addresses a key limitation in multimodal reasoning for AI systems, though it is incremental as it builds on existing Chain-of-Thought methods.
The paper tackles the problem of large Vision-Language Models struggling to self-correct during multi-turn reasoning with images, proposing DRIM to enable reliable reasoning through a pipeline of data construction, supervised fine-tuning, and redundancy-penalized reinforcement learning, achieving superior performance on visual understanding benchmarks.
Recent advances in large Vision-Language Models (VLMs) have exhibited strong reasoning capabilities on complex visual tasks by thinking with images in their Chain-of-Thought (CoT), which is achieved by actively invoking tools to analyze visual inputs rather than merely perceiving them. However, existing models often struggle to reflect on and correct themselves when attempting incorrect reasoning trajectories. To address this limitation, we propose DRIM, a model that enables deep but reliable multi-turn reasoning when thinking with images in its multimodal CoT. Our pipeline comprises three stages: data construction, cold-start SFT and RL. Based on a high-resolution image dataset, we construct high-difficulty and verifiable visual question-answer pairs, where solving each task requires multi-turn tool calls to reach the correct answer. In the SFT stage, we collect tool trajectories as cold-start data, guiding a multi-turn reasoning pattern. In the RL stage, we introduce redundancy-penalized policy optimization, which incentivizes the model to develop a self-reflective reasoning pattern. The basic idea is to impose judgment on reasoning trajectories and penalize those that produce incorrect answers without sufficient multi-scale exploration. Extensive experiments demonstrate that DRIM achieves superior performance on visual understanding benchmarks.