Tailored Primitive Initialization is the Secret Key to Reinforcement Learning
This work addresses sample efficiency and stability issues in RL for LLMs, which is an incremental improvement for researchers and practitioners in AI reasoning tasks.
The paper tackles the challenge of low sampling efficiency and initialization dependence in reinforcement learning for large language models by proposing Tailor, a finetuning pipeline that discovers and curates reasoning primitives, resulting in higher downstream RL performance on mathematical and logical reasoning benchmarks.
Reinforcement learning (RL) has emerged as a powerful paradigm for enhancing the reasoning capabilities of large language models (LLMs). While RL has demonstrated substantial performance gains, it still faces key challenges, including low sampling efficiency and a strong dependence on model initialization: some models achieve rapid improvements with minimal RL steps, while others require significant training data to make progress. In this work, we investigate these challenges through the lens of reasoning token coverage and argue that initializing LLMs with diverse, high-quality reasoning primitives is essential for achieving stable and sample-efficient RL training. We propose Tailor, a finetuning pipeline that automatically discovers and curates novel reasoning primitives, thereby expanding the coverage of reasoning-state distributions before RL. Extensive experiments on mathematical and logical reasoning benchmarks demonstrate that Tailor generates more diverse and higher-quality warm-start data, resulting in higher downstream RL performance.