From Trial-and-Error to Improvement: A Systematic Analysis of LLM Exploration Mechanisms in RLVR
This work addresses the underexplored issue of how LLMs explore in RLVR, offering incremental insights for researchers in AI and machine learning.
The paper tackles the problem of understanding exploration mechanisms in reinforcement learning with verifiable rewards (RLVR) for large language models, analyzing aspects like exploration space shaping and entropy-performance exchange to provide a foundational framework.
Reinforcement learning with verifiable rewards (RLVR) has emerged as a powerful paradigm for enhancing the reasoning capabilities of large language models (LLMs). Unlike traditional RL approaches, RLVR leverages rule-based feedback to guide LLMs in generating and refining complex reasoning chains -- a process critically dependent on effective exploration strategies. While prior work has demonstrated RLVR's empirical success, the fundamental mechanisms governing LLMs' exploration behaviors remain underexplored. This technical report presents a systematic investigation of exploration capacities in RLVR, covering four main aspects: (1) exploration space shaping, where we develop quantitative metrics to characterize LLMs' capability boundaries; (2) entropy-performance exchange, analyzed across training stages, individual instances, and token-level patterns; and (3) RL performance optimization, examining methods to effectively translate exploration gains into measurable improvements. By unifying previously identified insights with new empirical evidence, this work aims to provide a foundational framework for advancing RLVR systems.