Design of Chain-of-Thought in Math Problem Solving
This work addresses the challenge of improving reasoning in math problem solving for AI systems, offering incremental guidelines for CoT design based on programming language and coding style.
The paper tackled the problem of designing effective Chain-of-Thought (CoT) methods for math problem solving by comparing natural language CoT with various program CoTs, finding that program CoTs often have superior effectiveness, with the best combination beating GPT-3.5-turbo by a significant margin.
Chain-of-Thought (CoT) plays a crucial role in reasoning for math problem solving. We conduct a comprehensive examination of methods for designing CoT, comparing conventional natural language CoT with various program CoTs, including the self-describing program, the comment-describing program, and the non-describing program. Furthermore, we investigate the impact of programming language on program CoTs, comparing Python and Wolfram Language. Through extensive experiments on GSM8K, MATHQA, and SVAMP, we find that program CoTs often have superior effectiveness in math problem solving. Notably, the best performing combination with 30B parameters beats GPT-3.5-turbo by a significant margin. The results show that self-describing program offers greater diversity and thus can generally achieve higher performance. We also find that Python is a better choice of language than Wolfram for program CoTs. The experimental results provide a valuable guideline for future CoT designs that take into account both programming language and coding style for further advancements. Our datasets and code are publicly available.