AgentFly: Extensible and Scalable Reinforcement Learning for LM Agents
This work addresses the need for systematic reinforcement learning integration in LM agents, which is incremental as it builds on existing RL methods with token-level masking and decorator-based interfaces.
The paper tackles the underexplored combination of language model agents and reinforcement learning by introducing AgentFly, a scalable and extensible framework that supports multi-turn interactions and high-throughput training, demonstrating successful agent training across multiple tasks.
Language model (LM) agents have gained significant attention for their ability to autonomously complete tasks through interactions with environments, tools, and APIs. LM agents are primarily built with prompt engineering or supervised finetuning. At the same time, reinforcement learning (RL) has been explored to enhance LM's capabilities, such as reasoning and factuality. However, the combination of the LM agents and reinforcement learning (Agent-RL) remains underexplored and lacks systematic study. To this end, we built AgentFly, a scalable and extensible Agent-RL framework designed to empower LM agents with a variety of RL algorithms. Our framework supports multi-turn interactions by adapting traditional RL methods with token-level masking. It features a decorator-based interface for defining tools and reward functions, enabling seamless extension and ease of use. To support high-throughput training, we implement asynchronous execution of tool calls and reward computations, and design a centralized resource management system for scalable environment coordination. We also provide a suite of prebuilt tools and environments, demonstrating the framework's effectiveness through successful agent training across multiple tasks.