Bootstrapping Post-training Signals for Open-ended Tasks via Rubric-based Self-play on Pre-training Text
This addresses the need for cost-effective post-training for LLMs on realistic open-ended tasks, though it is incremental as it builds on existing self-play methods.
The paper tackles the problem of extending self-play training to open-ended tasks by proposing POP, a framework that synthesizes evaluation rubrics and input-output pairs using the same LLM, resulting in performance improvements on models like Qwen-2.5-7B across tasks such as healthcare QA and creative writing.
Self-play has recently emerged as a promising paradigm to train Large Language Models (LLMs). In self-play, the target LLM creates the task input (e.g., ask a question), which it then addresses itself by producing a task output (e.g., give an answer). A reward model evaluates the output, and the rewards are then used to train the LLM, typically via Reinforcement Learning (RL). Self-play incurs minimal supervision costs, and this is especially helpful for post-training LLMs, which require high-quality input-output pairs that traditionally have to be written by humans or expensive proprietary models. However, existing work explores self-play only for verifiable tasks such as math and coding. Instead, we seek to extend it to more realistic open-ended tasks. In particular, we propose POP, a self-play framework that uses the same LLM to synthesize evaluation rubrics, along with input-output pairs, for each example. The rubric is then used to evaluate outputs and train the model. We further ground the framework on a content-rich pretraining corpus to (1) ensure a generation-verification gap and reduce reward hacking, and (2) prevent mode collapse. On Qwen-2.5-7B, POP increases performance of both pretrained and instruction-tuned models, across different tasks ranging from long-form Healthcare QA to creative writing and instruction following.