SCOPE: Self-Play via Co-Evolving Policies for Open-Ended Tasks
This work addresses the problem of training language models on open-ended tasks without external supervision, which is a significant challenge for researchers and practitioners in natural language processing.
This paper introduces SCOPE, a data-free self-play framework for training language models on open-ended tasks by co-evolving a task-generating Challenger and a task-answering Solver. SCOPE improves open-ended performance by up to +10.4 points across eight benchmarks and also boosts held-out short-form QA by up to +13.8 points on seven benchmarks, outperforming GRPO_data which uses ~9K curated prompts.
Self-play can train language models without external supervision. However, existing methods require rule-checkable answers, leaving open-ended tasks dependent on curated prompts or frontier-model judges. We introduce SCOPE, a data-free self-play framework for open-ended tasks that co-evolves two policies: a Challenger that generates document-grounded tasks, and a Solver that answers them through multi-turn retrieval. A frozen copy of the initial model serves as the self-judge, which writes task-specific rubrics from the source document and grades Solver responses against them. Across three 7-8B instruction-tuned models (Qwen2.5, Qwen3, OLMo-3), SCOPE improves open-ended performance by up to +10.4 points on eight benchmarks and matches or exceeds GRPO_data trained on ~9K curated prompts. Although trained only on open-ended tasks, SCOPE also improves held-out short-form QA by up to +13.8 points on seven held-out benchmarks, surpassing GRPO_data on all three models. Ablations show that co-evolving the Challenger is necessary to keep tasks near the Solver's frontier, that gains arise from improvements in both retrieval and synthesis with the relative contribution varying by task, and that rubric generation quality is the bottleneck for self-judging.