AceMath: Advancing Frontier Math Reasoning with Post-Training and Reward Modeling
This work addresses the challenge of reliable math problem-solving and evaluation for AI systems, representing an incremental advancement with strong domain-specific improvements.
The paper tackles the problem of complex math reasoning by introducing AceMath, a suite of models that includes an instruction-tuned model and a reward model, which outperform state-of-the-art models like Qwen2.5-Math-72B-Instruct, GPT-4o, and Claude-3.5 Sonnet, achieving the highest average rm@8 score across math reasoning benchmarks.
In this paper, we introduce AceMath, a suite of frontier math models that excel in solving complex math problems, along with highly effective reward models capable of evaluating generated solutions and reliably identifying the correct ones. To develop the instruction-tuned math models, we propose a supervised fine-tuning (SFT) process that first achieves competitive performance across general domains, followed by targeted fine-tuning for the math domain using a carefully curated set of prompts and synthetically generated responses. The resulting model, AceMath-72B-Instruct greatly outperforms Qwen2.5-Math-72B-Instruct, GPT-4o and Claude-3.5 Sonnet. To develop math-specialized reward model, we first construct AceMath-RewardBench, a comprehensive and robust benchmark for evaluating math reward models across diverse problems and difficulty levels. After that, we present a systematic approach to build our math reward models. The resulting model, AceMath-72B-RM, consistently outperforms state-of-the-art reward models. Furthermore, when combining AceMath-72B-Instruct with AceMath-72B-RM, we achieve the highest average rm@8 score across the math reasoning benchmarks. We release model weights, training data, and evaluation benchmarks at: https://research.nvidia.com/labs/adlr/acemath