CoinMath: Harnessing the Power of Coding Instruction for Math LLMs
It addresses the challenge of improving math problem-solving in LLMs for researchers and developers, but is incremental as it builds on existing code-based methods.
This study tackled the problem of optimizing coding instruction data to enhance mathematical reasoning in Large Language Models, finding that code-based rationales with concise comments, descriptive naming, and hardcoded solutions are beneficial, and proposing CoinMath, which significantly outperforms the baseline MAmmoTH model.
Large Language Models (LLMs) have shown strong performance in solving mathematical problems, with code-based solutions proving particularly effective. However, the best practice to leverage coding instruction data to enhance mathematical reasoning remains underexplored. This study investigates three key questions: (1) How do different coding styles of mathematical code-based rationales impact LLMs' learning performance? (2) Can general-domain coding instructions improve performance? (3) How does integrating textual rationales with code-based ones during training enhance mathematical reasoning abilities? Our findings reveal that code-based rationales with concise comments, descriptive naming, and hardcoded solutions are beneficial, while improvements from general-domain coding instructions and textual rationales are relatively minor. Based on these insights, we propose CoinMath, a learning strategy designed to enhance mathematical reasoning by diversifying the coding styles of code-based rationales. CoinMath generates a variety of code-based rationales incorporating concise comments, descriptive naming conventions, and hardcoded solutions. Experimental results demonstrate that CoinMath significantly outperforms its baseline model, MAmmoTH, one of the SOTA math LLMs.