AutoForge: Automated Environment Synthesis for Agentic Reinforcement Learning
This work addresses the challenge of scalable and stable environment synthesis for language-based agents in reinforcement learning, representing an incremental advancement.
The paper tackles the problem of limited and unstable simulated environments for agentic reinforcement learning by proposing an automated synthesis pipeline and an environment-level RL algorithm, resulting in improved training efficiency and stability validated on benchmarks like tau-bench and VitaBench.
Conducting reinforcement learning (RL) in simulated environments offers a cost-effective and highly scalable way to enhance language-based agents. However, previous work has been limited to semi-automated environment synthesis or tasks lacking sufficient difficulty, offering little breadth or depth. In addition, the instability of simulated users integrated into these environments, along with the heterogeneity across simulated environments, poses further challenges for agentic RL. In this work, we propose: (1) a unified pipeline for automated and scalable synthesis of simulated environments associated with high-difficulty but easily verifiable tasks; and (2) an environment level RL algorithm that not only effectively mitigates user instability but also performs advantage estimation at the environment level, thereby improving training efficiency and stability. Comprehensive evaluations on agentic benchmarks, including tau-bench, tau2-Bench, and VitaBench, validate the effectiveness of our proposed method. Further in-depth analyses underscore its out-of-domain generalization.