Context Bootstrapped Reinforcement Learning
This addresses exploration challenges in RLVR for tasks requiring novel reasoning patterns, though it appears incremental as it augments existing RLVR training rather than introducing a fundamentally new paradigm.
The paper tackles the exploration inefficiency problem in Reinforcement Learning from Verifiable Rewards (RLVR) by proposing Context Bootstrapped Reinforcement Learning (CBRL), which stochastically prepends few-shot demonstrations to training prompts with an annealing curriculum. The results show CBRL consistently improves success rates and exploration efficiency across two model families and five Reasoning Gym tasks, and demonstrates practical applicability on a domain-specific programming language.
Reinforcement Learning from Verifiable Rewards (RLVR) suffers from exploration inefficiency, where models struggle to generate successful rollouts, resulting in minimal learning signal. This challenge is particularly severe for tasks that require the acquisition of novel reasoning patterns or domain-specific knowledge. To address this, we propose Context Bootstrapped Reinforcement Learning (CBRL), which augments RLVR training by stochastically prepending few-shot demonstrations to training prompts. The injection probability follows a curriculum that starts high to bootstrap early exploration, then anneals to zero so the model must ultimately succeed without assistance. This forces the policy to internalize reasoning patterns from the demonstrations rather than relying on them at test time. We validate CBRL across two model families and five Reasoning Gym tasks. Our results demonstrate that CBRL consistently improves success rate, provides better exploration efficiency, and is algorithm-agnostic. We further demonstrate CBRL's practical applicability on Q, a domain-specific programming language that diverges significantly from mainstream language conventions.