VTool-R1: VLMs Learn to Think with Images via Reinforcement Learning on Multimodal Tool Use
This work addresses the challenge of improving multimodal reasoning capabilities in VLMs for applications like visual question answering, though it is incremental as it builds on existing reinforcement learning finetuning methods by incorporating visual tools.
The paper tackles the problem of enabling vision-language models (VLMs) to perform true multimodal reasoning by generating chains of thought that interleave text and visual steps, rather than relying on text-only reasoning conditioned on static images. The result is VTool-R1, a framework that trains VLMs using reinforcement learning with visual editing tools, which enhances reasoning performance on structured visual question answering tasks over charts and tables.
Reinforcement Learning Finetuning (RFT) has significantly advanced the reasoning capabilities of large language models (LLMs) by enabling long chains of thought, self-correction, and effective tool use. While recent works attempt to extend RFT to vision-language models (VLMs), these efforts largely produce text-only reasoning conditioned on static image inputs, falling short of true multimodal reasoning in the response. In contrast, test-time methods like Visual Sketchpad incorporate visual steps but lack training mechanisms. We introduce VTool-R1, the first framework that trains VLMs to generate multimodal chains of thought by interleaving text and intermediate visual reasoning steps. VTool-R1 integrates Python-based visual editing tools into the RFT process, enabling VLMs to learn when and how to generate visual reasoning steps that benefit final reasoning. Trained with outcome-based rewards tied to task accuracy, our approach elicits strategic visual tool use for reasoning without relying on process-based supervision. Experiments on structured visual question answering over charts and tables show that VTool-R1 enhances reasoning performance by teaching VLMs to "think with images" and generate multimodal chain of thoughts with tools.