The Role of Visual Modality in Multimodal Mathematical Reasoning: Challenges and Insights
This work addresses the challenge of improving visual modality integration in AI for mathematical reasoning, which is incremental as it builds on existing datasets and benchmarks.
The study tackled the problem of visual information being underutilized in multimodal mathematical reasoning, finding that existing models perform similarly with or without images due to textual dominance, and introduced the HC-M3D dataset to require image reliance, revealing models' failure to detect subtle visual differences.
Recent research has increasingly focused on multimodal mathematical reasoning, particularly emphasizing the creation of relevant datasets and benchmarks. Despite this, the role of visual information in reasoning has been underexplored. Our findings show that existing multimodal mathematical models minimally leverage visual information, and model performance remains largely unaffected by changes to or removal of images in the dataset. We attribute this to the dominance of textual information and answer options that inadvertently guide the model to correct answers. To improve evaluation methods, we introduce the HC-M3D dataset, specifically designed to require image reliance for problem-solving and to challenge models with similar, yet distinct, images that change the correct answer. In testing leading models, their failure to detect these subtle visual differences suggests limitations in current visual perception capabilities. Additionally, we observe that the common approach of improving general VQA capabilities by combining various types of image encoders does not contribute to math reasoning performance. This finding also presents a challenge to enhancing visual reliance during math reasoning. Our benchmark and code would be available at \href{https://github.com/Yufang-Liu/visual_modality_role}{https://github.com/Yufang-Liu/visual\_modality\_role}.