CLI-Gym: Scalable CLI Task Generation via Agentic Environment Inversion
This addresses the need for scalable task generation to enhance agents' capabilities in interacting with command line interfaces, representing a novel pipeline rather than an incremental improvement.
The paper tackled the problem of generating scalable environment-intensive tasks for agentic coding by proposing CLI-Gym, which inverts healthy environment histories to create buggy states and tasks, resulting in a collection of 1,655 tasks and a fine-tuned model achieving a +21.1% improvement to 46.1% on Terminal-Bench.
Agentic coding requires agents to effectively interact with runtime environments, e.g., command line interfaces (CLI), so as to complete tasks like resolving dependency issues, fixing system problems, etc. But it remains underexplored how such environment-intensive tasks can be obtained at scale to enhance agents' capabilities. To address this, based on an analogy between the Dockerfile and the agentic task, we propose to employ agents to simulate and explore environment histories, guided by execution feedback. By tracing histories of a healthy environment, its state can be inverted to an earlier one with runtime failures, from which a task can be derived by packing the buggy state and the corresponding error messages. With our method, named CLI-Gym, a total of 1,655 environment-intensive tasks are derived, being the largest collection of its kind. Moreover, with curated successful trajectories, our fine-tuned model, named LiberCoder, achieves substantial absolute improvements of +21.1% (to 46.1%) on Terminal-Bench, outperforming various strong baselines. To our knowledge, this is the first public pipeline for scalable derivation of environment-intensive tasks.