ToRL: Scaling Tool-Integrated RL
This addresses the challenge of enabling LLMs to effectively leverage tools for complex tasks like mathematical reasoning, representing a novel method rather than an incremental improvement.
The paper tackles the problem of training large language models to autonomously use computational tools via reinforcement learning, resulting in ToRL-7B achieving 43.3% accuracy on AIME 24, surpassing reinforcement learning without tool integration by 14% and the best existing Tool-Integrated Reasoning model by 17%.
We introduce ToRL (Tool-Integrated Reinforcement Learning), a framework for training large language models (LLMs) to autonomously use computational tools via reinforcement learning. Unlike supervised fine-tuning, ToRL allows models to explore and discover optimal strategies for tool use. Experiments with Qwen2.5-Math models show significant improvements: ToRL-7B reaches 43.3\% accuracy on AIME~24, surpassing reinforcement learning without tool integration by 14\% and the best existing Tool-Integrated Reasoning (TIR) model by 17\%. Further analysis reveals emergent behaviors such as strategic tool invocation, self-regulation of ineffective code, and dynamic adaptation between computational and analytical reasoning, all arising purely through reward-driven learning.