Knowledge-to-Verification: Exploring RLVR for LLMs in Knowledge-Intensive Domains
This work addresses the challenge of applying RLVR to knowledge-intensive domains for LLM reasoning enhancement, but the results are incremental as they show improvement without major degradation rather than a breakthrough.
The paper proposes K2V, a framework that extends reinforcement learning with verifiable rewards (RLVR) to knowledge-intensive domains by automating verifiable data synthesis and enabling reasoning process verification. Experiments show K2V enhances LLM reasoning in these domains without significantly harming general capabilities.
Reinforcement learning with verifiable rewards (RLVR) has demonstrated promising potential to enhance the reasoning capabilities of large language models (LLMs) in domains such as mathematics and coding. However, its applications on knowledge-intensive domains have not been effectively explored due to the scarcity of high-quality verifiable data. Furthermore, current RLVR focuses solely on the correctness of final answers, leading to the limitations of flawed reasoning and sparse reward signals. In this work, we propose Knowledge-to-Verification (K2V), a framework that extends RLVR to knowledge-intensive domains through automated verifiable data synthesis, while enabling verification of the LLM's reasoning process. Extensive experiments demonstrate that K2V enhances the reasoning of LLM in knowledge-intensive domains without significantly compromising the model's general capabilities. This study also suggests that integrating automated data synthesis with reasoning verification is a promising direction to enhance model capabilities in these broader domains. Code is available at https://github.com/SeedScientist/K2V.