StepMathAgent: A Step-Wise Agent for Evaluating Mathematical Processes through Tree-of-Error
This addresses inaccurate and uninterpretable evaluation outcomes for mathematical processes in LLMs, offering a domain-specific improvement.
The authors tackled the problem of evaluating mathematical capabilities in large language models by proposing StepMathAgent, a step-wise agent based on Tree-of-Error, which outperformed all state-of-the-art methods on the StepMathBench benchmark of 1,000 instances.
Evaluating mathematical capabilities is critical for assessing the overall performance of large language models (LLMs). However, existing evaluation methods often focus solely on final answers, resulting in highly inaccurate and uninterpretable evaluation outcomes, as well as their failure to assess proof or open-ended problems. To address these issues, we propose a novel mathematical process evaluation agent based on Tree-of-Error, called StepMathAgent. This agent incorporates four internal core operations: logical step segmentation, step scoring, score aggregation and error tree generation, along with four external extension modules: difficulty calibration, simplicity evaluation, completeness validation and format assessment. Furthermore, we introduce StepMathBench, a benchmark comprising 1,000 step-divided process evaluation instances, derived from 200 high-quality math problems grouped by problem type, subject category and difficulty level. Experiments on StepMathBench show that our proposed StepMathAgent outperforms all state-of-the-art methods, demonstrating human-aligned evaluation preferences and broad applicability to various scenarios. Our data and code are available at https://github.com/SHU-XUN/StepMathAgent.