Mars-PO: Multi-Agent Reasoning System Preference Optimization
This addresses the problem of errors and inconsistencies in multi-step mathematical reasoning for LLM users, representing a strong specific gain rather than a foundational advancement.
The paper tackles the challenge of improving mathematical reasoning in large language models by proposing Mars-PO, a multi-agent framework that combines outputs to create preference pairs for training, resulting in an accuracy increase from 50.38% to 57.82% on the MATH benchmark for Llama3.1-8B-Instruct.
Mathematical reasoning is a fundamental capability for large language models (LLMs), yet achieving high performance in this domain remains a significant challenge. The auto-regressive generation process often makes LLMs susceptible to errors, hallucinations, and inconsistencies, particularly during multi-step reasoning. In this paper, we propose Mars-PO, a novel framework to improve the mathematical reasoning capabilities of LLMs through a multi-agent system. It combines high-quality outputs from multiple agents into a hybrid positive sample set and pairs them with agent-specific negative samples to construct robust preference pairs for training. By aligning agents with shared positive samples while addressing individual weaknesses, Mars-PO achieves substantial performance improvements on mathematical reasoning benchmarks. For example, it increases the accuracy on the MATH benchmark of the state-of-the-art instruction-tuned LLM, Llama3.1-8B-Instruct, from 50.38% to 57.82%. Experimental results further demonstrate that our method consistently outperforms other baselines, such as supervised fine-tuning, vanilla DPO, and its enhanced versions, highlighting the effectiveness of our approach.