In-Context Reinforcement Learning for Tool Use in Large Language Models
This work addresses the challenge of efficiently training large language models to use external tools, which is crucial for improving their performance on complex tasks without extensive labeled data.
This paper introduces In-Context Reinforcement Learning (ICRL), an RL-only framework for enabling large language models to use external tools, eliminating the need for supervised fine-tuning. ICRL leverages few-shot prompting during RL rollouts and gradually reduces the number of in-context examples, achieving state-of-the-art performance on various reasoning and tool-use benchmarks.
While large language models (LLMs) exhibit strong reasoning abilities, their performance on complex tasks is often constrained by the limitations of their internal knowledge. A compelling approach to overcome this challenge is to augment these models with external tools -- such as Python interpreters for mathematical computations or search engines for retrieving factual information. However, enabling models to use these tools effectively remains a significant challenge. Existing methods typically rely on cold-start pipelines that begin with supervised fine-tuning (SFT), followed by reinforcement learning (RL). These approaches often require substantial amounts of labeled data for SFT, which is expensive to annotate or synthesize. In this work, we propose In-Context Reinforcement Learning (ICRL), an RL-only framework that eliminates the need for SFT by leveraging few-shot prompting during the rollout stage of RL. Specifically, ICRL introduces in-context examples within the rollout prompts to teach the model how to invoke external tools. Furthermore, as training progresses, the number of in-context examples is gradually reduced, eventually reaching a zero-shot setting where the model learns to call tools independently. We conduct extensive experiments across a range of reasoning and tool-use benchmarks. Results show that ICRL achieves state-of-the-art performance, demonstrating its effectiveness as a scalable, data-efficient alternative to traditional SFT-based pipelines.