Auditable-choice reframing unlocks RL-based verification for open-ended tasks
This work addresses a bottleneck in using RL-based verification for creative and open-ended tasks like writing and instruction following, offering a method to transfer reasoning improvements to domains without standard answers.
The paper tackles the problem of applying reinforcement learning with verifiable rewards (RLVR) to open-ended tasks lacking ground-truth solutions by introducing Verifiable Multiple-Choice Reformulation (VMR), which restructures such tasks into verifiable multiple-choice formats, achieving an average gain of 5.99 points over baselines across eight benchmarks.
Reinforcement Learning with Verifiable Rewards (RLVR) has demonstrated great potential in enhancing the reasoning capabilities of large language models (LLMs), achieving remarkable progress in domains such as mathematics and programming where standard answers are available. However, for open-ended tasks lacking ground-truth solutions (e.g., creative writing and instruction following), existing studies typically regard them as non-reasoning scenarios, thereby overlooking the latent value of reasoning capabilities. This raises a key question: Can strengthening reasoning improve performance in open-ended tasks? To address this, we explore the transfer of the RLVR paradigm to the open domain. Yet, since RLVR fundamentally relies on verifiers that presuppose the existence of standard answers, it cannot be directly applied to open-ended tasks. To overcome this challenge, we introduce Verifiable Multiple-Choice Reformulation (VMR), a novel training strategy that restructures open-ended data into verifiable multiple-choice formats, enabling effective training even in the absence of explicit ground truth. Experimental results on multiple benchmarks validate the effectiveness of our method in improving LLM performance on open-ended tasks. Notably, across eight open-ended benchmarks, our VMR-based training delivers an average gain of 5.99 points over the baseline. Code will be released upon acceptance to facilitate reproducibility.