Evaluating and Improving Tool-Augmented Computation-Intensive Math Reasoning
This work addresses the need for better evaluation and enhancement of LLMs in math reasoning, particularly for Chinese-language tasks, but it is incremental as it builds on existing chain-of-thought and tool-augmentation methods.
The authors tackled the problem of evaluating and improving large language models (LLMs) on computation-intensive math reasoning by constructing a new Chinese dataset (CARP) with 4,886 problems and annotations, and proposing a deliberation approach (DELI) that iteratively refines reasoning steps, which outperformed baselines on CARP and six other datasets.
Chain-of-thought prompting~(CoT) and tool augmentation have been validated in recent work as effective practices for improving large language models~(LLMs) to perform step-by-step reasoning on complex math-related tasks. However, most existing math reasoning datasets may be not able to fully evaluate and analyze the ability of LLMs in manipulating tools and performing reasoning, as they may only require very few invocations of tools or miss annotations for evaluating intermediate reasoning steps. To address the issue, we construct \textbf{CARP}, a new Chinese dataset consisting of 4,886 computation-intensive algebra problems with formulated annotations on intermediate steps. In CARP, we test four LLMs with CoT prompting, and find that they are all prone to make mistakes at the early steps of the solution, leading to wrong answers. Based on this finding, we propose a new approach that can deliberate the reasoning steps with tool interfaces, namely \textbf{DELI}. In DELI, we first initialize a step-by-step solution based on retrieved exemplars, then iterate two deliberation procedures that check and refine the intermediate steps of the generated solution, from the perspectives of tool manipulation and natural language reasoning, until obtaining converged solutions or reaching the maximum turn. Experimental results on CARP and six other datasets show that the proposed DELI mostly outperforms competitive baselines, and can further boost the performance of existing CoT methods. Our data and code are available in \url{https://github.com/RUCAIBox/CARP}.