MM-MATH: Advancing Multimodal Math Evaluation with Process Evaluation and Fine-grained Classification
This addresses the problem of limited evaluation methods for multimodal math reasoning in AI models, though it is incremental as it builds on existing benchmarks by adding process analysis.
The paper tackles the evaluation of multimodal math reasoning in large multimodal models by introducing MM-MATH, a benchmark with 5,929 open-ended problems that includes process evaluation to analyze solution steps and error types, revealing that the best model achieves only 31% accuracy compared to 82% for humans.
To advance the evaluation of multimodal math reasoning in large multimodal models (LMMs), this paper introduces a novel benchmark, MM-MATH. MM-MATH consists of 5,929 open-ended middle school math problems with visual contexts, with fine-grained classification across difficulty, grade level, and knowledge points. Unlike existing benchmarks relying on binary answer comparison, MM-MATH incorporates both outcome and process evaluations. Process evaluation employs LMM-as-a-judge to automatically analyze solution steps, identifying and categorizing errors into specific error types. Extensive evaluation of ten models on MM-MATH reveals significant challenges for existing LMMs, highlighting their limited utilization of visual information and struggles with higher-difficulty problems. The best-performing model achieves only 31% accuracy on MM-MATH, compared to 82% for humans. This highlights the challenging nature of our benchmark for existing models and the significant gap between the multimodal reasoning capabilities of current models and humans. Our process evaluation reveals that diagram misinterpretation is the most common error, accounting for more than half of the total error cases, underscoring the need for improved image comprehension in multimodal reasoning.