Language Self-Play For Data-Free Training
This addresses the bottleneck of needing ever more training data for LLMs, offering a potential solution for researchers and developers in AI.
The paper tackles the problem of data dependency in training large language models by proposing a reinforcement learning approach using self-play, which enables models to improve without additional data. Experiments with Llama-3.2-3B-Instruct show that this method enhances performance on instruction-following benchmarks more effectively than data-driven baselines.
Large language models (LLMs) have advanced rapidly in recent years, driven by scale, abundant high-quality training data, and reinforcement learning. Yet this progress faces a fundamental bottleneck: the need for ever more data from which models can continue to learn. In this work, we propose a reinforcement learning approach that removes this dependency by enabling models to improve without additional data. Our method leverages a game-theoretic framework of self-play, where a model's capabilities are cast as performance in a competitive game and stronger policies emerge by having the model play against itself - a process we call Language Self-Play (LSP). Experiments with Llama-3.2-3B-Instruct on instruction-following benchmarks show that pretrained models can not only enhance their performance on challenging tasks through self-play alone, but can also do so more effectively than data-driven baselines.