Describe-then-Reason: Improving Multimodal Mathematical Reasoning through Visual Comprehension Training
This work addresses a specific bottleneck in multimodal AI for mathematical reasoning, offering an incremental improvement over existing fine-tuning methods.
The paper tackles the problem of poor visual comprehension in open-source multimodal large language models for complex mathematical reasoning by proposing a two-step training pipeline called VCAR, which first trains on visual description generation and then on rationale generation, resulting in substantial performance improvements on benchmarks, especially for visually demanding problems.
Open-source multimodal large language models (MLLMs) excel in various tasks involving textual and visual inputs but still struggle with complex multimodal mathematical reasoning, lagging behind proprietary models like GPT-4V(ision) and Gemini-Pro. Although fine-tuning with intermediate steps (i.e., rationales) elicits some mathematical reasoning skills, the resulting models still fall short in visual comprehension due to inadequate visual-centric supervision, which leads to inaccurate interpretation of math figures. To address this issue, we propose a two-step training pipeline VCAR, which emphasizes the Visual Comprehension training in Addition to mathematical Reasoning learning. It first improves the visual comprehension ability of MLLMs through the visual description generation task, followed by another training step on generating rationales with the assistance of descriptions. Experimental results on two popular benchmarks demonstrate that VCAR substantially outperforms baseline methods solely relying on rationale supervision, especially on problems with high visual demands.