TTVS: Boosting Self-Exploring Reinforcement Learning via Test-time Variational Synthesis
This addresses the challenge of expensive or unavailable supervision in specialized domains for AI researchers and practitioners, offering a novel test-time adaptation approach.
The paper tackles the problem of limited supervision in reinforcement learning for Large Reasoning Models by introducing Test-Time Variational Synthesis (TTVS), which dynamically augments training from unlabeled test queries, resulting in superior performance that surpasses both test-time adaptation and supervised state-of-the-art methods across eight model architectures.
Despite significant advances in Large Reasoning Models (LRMs) driven by reinforcement learning with verifiable rewards (RLVR), this paradigm is fundamentally limited in specialized or novel domains where such supervision is prohibitively expensive or unavailable, posing a key challenge for test-time adaptation. While existing test-time methods offer a potential solution, they are constrained by learning from static query sets, risking overfitting to textual patterns. To address this gap, we introduce Test-Time Variational Synthesis (TTVS), a novel framework that enables LRMs to self-evolve by dynamically augmenting the training stream from unlabeled test queries. TTVS comprises two synergistic modules: (1) Online Variational Synthesis, which transforms static test queries into a dynamic stream of diverse, semantically-equivalent variations, enforcing the model to learn underlying problem logic rather than superficial patterns; (2) Test-time Hybrid Exploration, which balances accuracy-driven exploitation with consistency-driven exploration across synthetic variants. Extensive experiments show TTVS yields superior performance across eight model architectures. Notably, using only unlabeled test-time data, TTVS not only surpasses other test-time adaptation methods but also outperforms state-of-the-art supervised RL-based techniques trained on vast, high-quality labeled data.