Learning Generative Selection for Best-of-N
This work addresses the bottleneck of efficient test-time scaling for LLM reasoning, offering a scalable method to enhance selection in small models, though it is incremental as it builds on existing generative selection methods.
The paper tackled the problem of improving Best-of-N selection quality for LLM reasoning by enabling small models to acquire strong generative selection capabilities through targeted reinforcement learning, achieving consistent performance gains across math and code benchmarks that often approach or exceed larger models.
Scaling test-time compute via parallel sampling can substantially improve LLM reasoning, but is often limited by Best-of-N selection quality. Generative selection methods, such as GenSelect, address this bottleneck, yet strong selection performance remains largely limited to large models. We show that small reasoning models can acquire strong GenSelect capabilities through targeted reinforcement learning. To this end, we synthesize selection tasks from large-scale math and code instruction datasets by filtering to instances with both correct and incorrect candidate solutions, and train 1.7B-parameter models with DAPO to reward correct selections. Across math (AIME24, AIME25, HMMT25) and code (LiveCodeBench) reasoning benchmarks, our models consistently outperform prompting and majority-voting baselines, often approaching or exceeding much larger models. Moreover, these gains generalize to selecting outputs from stronger models despite training only on outputs from weaker models. Overall, our results establish reinforcement learning as a scalable way to unlock strong generative selection in small models, enabling efficient test-time scaling.