Generalizable End-to-End Tool-Use RL with Synthetic CodeGym
This addresses the brittleness of LLM agents with new tools and workflows, offering a scalable training approach, though it is incremental as it builds on existing RL and coding-based methods.
The paper tackles the problem of poor generalization in tool-augmented LLM agents by introducing CodeGym, a framework that synthesizes interactive tool-use environments from coding problems, resulting in models like Qwen2.5-32B-Instruct achieving an 8.7-point accuracy gain on an out-of-distribution benchmark.
Tool-augmented large language models (LLMs), hereafter LLM agents, leverage external tools to solve diverse tasks and interface with the real world. However, current training practices largely rely on supervised fine-tuning (SFT) over static trajectories or reinforcement learning (RL) on narrow tasks, and generalize poorly beyond development settings, leading to brittleness with new tools and unseen workflows. Because code execution reflects many structures of real-world workflows, coding problems provide a natural basis for building agent training environments. Motivated by this, we introduce CodeGym, a scalable framework that synthesizes diverse, verifiable, and controllable multi-turn tool-use environments for agent RL, enabling LLM agents to explore and master various workflows actively. CodeGym rewrites static coding problems into interactive environments by extracting atomic functions or logic into callable tools, yielding verifiable tasks that span various tool-execution workflows. Models of varying sizes and chain-of-thought configurations, trained in CodeGym, exhibit consistent out-of-distribution generalizability; for example, Qwen2.5-32B-Instruct achieves an absolute accuracy gain of 8.7 points on the OOD benchmark $τ$-Bench. These results highlight CodeGym as a step toward scalable general-purpose RL environments that align with real-world agent workflows.