Towards Understanding Self-play for LLM Reasoning
This work addresses the problem of understanding self-play mechanisms for improving LLM reasoning, which is incremental as it clarifies differences from other methods.
The study analyzed the training dynamics of self-play for large language model reasoning, comparing it to reinforcement learning with verifiable rewards and supervised fine-tuning, and found that self-play shows strong in-domain and out-of-domain gains but has inherent limitations.
Recent advances in large language model (LLM) reasoning, led by reinforcement learning with verifiable rewards (RLVR), have inspired self-play post-training, where models improve by generating and solving their own problems. While self-play has shown strong in-domain and out-of-domain gains, the mechanisms behind these improvements remain poorly understood. In this work, we analyze the training dynamics of self-play through the lens of the Absolute Zero Reasoner, comparing it against RLVR and supervised fine-tuning (SFT). Our study examines parameter update sparsity, entropy dynamics of token distributions, and alternative proposer reward functions. We further connect these dynamics to reasoning performance using pass@k evaluations. Together, our findings clarify how self-play differs from other post-training strategies, highlight its inherent limitations, and point toward future directions for improving LLM math reasoning through self-play.