lmgame-Bench: How Good are LLMs at Playing Games?
This provides a reliable benchmark for assessing LLM capabilities in gaming and planning tasks, though it is incremental as it builds on existing evaluation frameworks.
The paper tackles the problem of evaluating large language models (LLMs) in video games by identifying challenges like brittle vision perception and prompt sensitivity, and introduces lmgame-Bench, a suite of games with a unified API and scaffolds that stabilizes evaluation and shows strong model separation across 13 models.
Playing video games requires perception, memory, and planning, exactly the faculties modern large language model (LLM) agents are expected to master. We study the major challenges in using popular video games to evaluate modern LLMs and find that directly dropping LLMs into games cannot make an effective evaluation, for three reasons -- brittle vision perception, prompt sensitivity, and potential data contamination. We introduce lmgame-Bench to turn games into reliable evaluations. lmgame-Bench features a suite of platformer, puzzle, and narrative games delivered through a unified Gym-style API and paired with lightweight perception and memory scaffolds, and is designed to stabilize prompt variance and remove contamination. Across 13 leading models, we show lmgame-Bench is challenging while still separating models well. Correlation analysis shows that every game probes a unique blend of capabilities often tested in isolation elsewhere. More interestingly, performing reinforcement learning on a single game from lmgame-Bench transfers both to unseen games and to external planning tasks. Our evaluation code is available at https://github.com/lmgame-org/GamingAgent/lmgame-bench.