General Modular Harness for LLM Agents in Multi-Turn Gaming Environments
This work addresses the challenge of enabling LLM agents to handle diverse gaming environments without domain-specific engineering, representing an incremental advancement in general-purpose agent design.
The authors tackled the problem of creating general-purpose LLM agents for multi-turn gaming environments by introducing a modular harness design with perception, memory, and reasoning components, which consistently improved gameplay performance over un-harnessed baselines and revealed distinct module contributions (e.g., memory dominates in long-horizon puzzles).
We introduce a modular harness design for LLM agents that composes of perception, memory, and reasoning components, enabling a single LLM or VLM backbone to tackle a wide spectrum of multi turn gaming environments without domain-specific engineering. Using classic and modern game suites as low-barrier, high-diversity testbeds, our framework provides a unified workflow for analyzing how each module affects performance across dynamic interactive settings. Extensive experiments demonstrate that the harness lifts gameplay performance consistently over un-harnessed baselines and reveals distinct contribution patterns, for example, memory dominates in long-horizon puzzles while perception is critical in vision noisy arcades. These findings highlight the effectiveness of our modular harness design in advancing general-purpose agent, given the familiarity and ubiquity of games in everyday human experience.