PyraMathBench: Evaluating and Improving Mathematical Capability in Large Language Models
For researchers developing LLMs, this benchmark and training method address the interpretability gap in numerical reasoning failures, though the approach is incremental.
PyraMathBench is a hierarchical benchmark with 32,505 questions for evaluating LLMs' numerical reasoning, revealing that performance is limited by poor numerical computation and abstract handling. The proposed SOLVE and IRPO methods improve Qwen-2.5's score by 5.0.
Despite the pivotal role of numerical reasoning as the cornerstone of mathematical capabilities in large language models (LLMs) across applications, few benchmarks evaluate LLMs by integrating numerical processing and mathematical reasoning, hindering the interpretability of failures in math tasks. We introduce PyraMathBench, a comprehensive hierarchical benchmark with 32,505 questions derived from 7,404 math word problems, spanning 4 key cognitive aspects, 14 subcategories, and 2 modalities. Experiments reveal that LLMs' performance is severely compromised by inadequate numerical computation and weak handling of abstract numerical questions. To address this, we propose the Smart Optimization & Learning-based VErsatile module (SOLVE) and Interactive Relative Policy Optimization (IRPO), which enhance LLMs' numerical-mathematical synergy via efficient tool calls (fuzzy matching and low-quality call rejection). Comparative experiments show Qwen-2.5 achieves a 5.0 score improvement with SOLVE and IRPO training.