MaRVL-QA: A Benchmark for Mathematical Reasoning over Visual Landscapes
This provides a challenging new benchmark for the research community to measure progress and expose limitations in MLLMs, addressing the need for deeper reasoning beyond semantic description, though it is incremental as it focuses on evaluation rather than solving the reasoning problem directly.
The authors tackled the problem of evaluating mathematical and spatial reasoning in Multimodal Large Language Models (MLLMs) by introducing MaRVL-QA, a benchmark with tasks like Topological Counting and Transformation Recognition, and found that state-of-the-art MLLMs struggle significantly, often using superficial heuristics.
A key frontier for Multimodal Large Language Models (MLLMs) is the ability to perform deep mathematical and spatial reasoning directly from images, moving beyond their established success in semantic description. Mathematical surface plots provide a rigorous testbed for this capability, as they isolate the task of reasoning from the semantic noise common in natural images. To measure progress on this frontier, we introduce MaRVL-QA (Mathematical Reasoning over Visual Landscapes), a new benchmark designed to quantitatively evaluate these core reasoning skills. The benchmark comprises two novel tasks: Topological Counting, identifying and enumerating features like local maxima; and Transformation Recognition, recognizing applied geometric transformations. Generated from a curated library of functions with rigorous ambiguity filtering, our evaluation on MaRVL-QA reveals that even state-of-the-art MLLMs struggle significantly, often resorting to superficial heuristics instead of robust spatial reasoning. MaRVL-QA provides a challenging new tool for the research community to measure progress, expose model limitations, and guide the development of MLLMs with more profound reasoning abilities.