DARL: Encouraging Diverse Answers for General Reasoning without Verifiers
This addresses a limitation in reinforcement learning for language models, improving diversity and accuracy in general reasoning tasks, though it is incremental as it builds on existing methods like RLPR.
The paper tackles the problem of reinforcement learning methods for language models overfitting to reference answers, limiting output diversity, especially in open-ended tasks. The proposed DARL framework encourages diverse answers within a controlled deviation range, achieving average gains of 1.3 points on reasoning benchmarks and 9.5 points on general benchmarks.
Reinforcement Learning with Verifiable Rewards (RLVR) has demonstrated promising gains in enhancing the reasoning capabilities of large language models. However, its dependence on domain-specific verifiers significantly restricts its applicability to open and general domains. Recent efforts such as RLPR have extended RLVR to general domains, enabling training on broader datasets and achieving improvements over RLVR. However, a notable limitation of these methods is their tendency to overfit to reference answers, which constrains the model's ability to generate diverse outputs. This limitation is particularly pronounced in open-ended tasks such as writing, where multiple plausible answers exist. To address this, we propose DARL, a simple yet effective reinforcement learning framework that encourages the generation of diverse answers within a controlled deviation range from the reference while preserving alignment with it. Our framework is fully compatible with existing general reinforcement learning methods and can be seamlessly integrated without additional verifiers. Extensive experiments on thirteen benchmarks demonstrate consistent improvements in reasoning performance. Notably, DARL surpasses RLPR, achieving average gains of 1.3 points on six reasoning benchmarks and 9.5 points on seven general benchmarks, highlighting its effectiveness in improving both reasoning accuracy and output diversity.