Enhancing Mathematical Problem Solving in LLMs through Execution-Driven Reasoning Augmentation
This addresses the need for reliably revisable reasoning in AI for applications like education and science, though it appears incremental as it builds on existing multi-agent LLM systems.
The paper tackled the problem of unreliable reasoning processes in LLMs for mathematical problem solving by introducing Iteratively Improved Program Construction (IIPC), which iteratively refines programmatic reasoning chains using execution feedback and Chain-of-thought, resulting in surpassing competing approaches on most reasoning benchmarks across multiple base LLMs.
Mathematical problem solving is a fundamental benchmark for assessing the reasoning capabilities of artificial intelligence and a gateway to applications in education, science, and engineering where reliable symbolic reasoning is essential. Although recent advances in multi-agent LLM-based systems have enhanced their mathematical reasoning capabilities, they still lack a reliably revisable representation of the reasoning process. Existing agents either operate in rigid sequential pipelines that cannot correct earlier steps or rely on heuristic self-evaluation that can fail to identify and fix errors. In addition, programmatic context can distract language models and degrade accuracy. To address these gaps, we introduce Iteratively Improved Program Construction (IIPC), a reasoning method that iteratively refines programmatic reasoning chains and combines execution feedback with the native Chain-of-thought abilities of the base LLM to maintain high-level contextual focus. IIPC surpasses competing approaches in the majority of reasoning benchmarks on multiple base LLMs. All code and implementations are released as open source.