Prepare Reasoning Language Models for Multi-Agent Debate with Self-Debate Reinforcement Learning
This addresses the challenge of enhancing LLM reasoning for collaborative problem-solving, though it is incremental as it builds on existing RLVR and MAD methods.
The paper tackles the problem of training large language models (LLMs) to better benefit from multi-agent debate (MAD) by proposing Self-Debate Reinforcement Learning (SDRL), which improves overall MAD performance and strengthens single-model reasoning across multiple benchmarks.
The reasoning abilities of large language models (LLMs) have been substantially improved by reinforcement learning with verifiable rewards (RLVR). At test time, collaborative reasoning through Multi-Agent Debate (MAD) has emerged as a promising approach for enhancing LLM performance. However, current RLVR methods typically train LLMs to solve problems in isolation, without explicitly preparing them to synthesize and benefit from different rationales that arise during debate. In this work, we propose Self-Debate Reinforcement Learning (SDRL), a training framework that equips a single LLM with strong standalone problem-solving ability and the capability to learn from diverse reasoning trajectories in MAD. Given a prompt, SDRL first samples multiple candidate solutions, then constructs a debate context with diverse reasoning paths and generates second-turn responses conditioned on this context. Finally, SDRL jointly optimizes both the initial and debate-conditioned responses, yielding a model that is effective as both a standalone solver and a debate participant. Experiments across multiple base models and reasoning benchmarks show that SDRL improves overall MAD performance while simultaneously strengthening single model reasoning.