GEM: A Gym for Agentic LLMs
This work addresses the problem of facilitating agentic LLM research by providing a standardized toolkit, though it is incremental as it builds on existing RL frameworks like OpenAI-Gym.
The authors tackled the need for standardized environments for experience-based learning in agentic LLMs by introducing GEM, an open-source simulator that provides a framework with diverse environments and tools, and they demonstrated its utility with baselines and benchmarking across 24 environments using methods like REINFORCE with Return Batch Normalization.
The training paradigm for large language models (LLMs) is moving from static datasets to experience-based learning, where agents acquire skills via interacting with complex environments. To facilitate this transition we introduce GEM (General Experience Maker), an open-source environment simulator designed for the age of LLMs. Analogous to OpenAI-Gym for traditional reinforcement learning (RL), GEM provides a standardized framework for the environment-agent interface, including asynchronous vectorized execution for high throughput, and flexible wrappers for easy extensibility. GEM also features a diverse suite of environments, robust integrated tools, and single-file example scripts demonstrating using GEM with five popular RL training frameworks. Along with this, we also provide a set of baselines across 24 environments using REINFORCE with Return Batch Normalization (ReBN), which -- unlike GRPO -- is compatible with the full RL setting of dense per-turn rewards and offers better credit assignment. We further conduct apple-to-apple benchmarking of PPO, GRPO and REINFORCE in both single- and multi-turn settings using GEM to shed light on the algorithmic designs. Lastly, GEM also functions as a convenient evaluation toolkit besides a training environment. We hope this framework can help accelerate future agentic LLM research.