GVGAI-LLM: Evaluating Large Language Model Agents with Infinite Games
This provides a reproducible testbed for advancing research on language model capabilities, particularly for agentic behavior and contextual reasoning, though it is incremental as it builds on existing frameworks.
The authors tackled the problem of evaluating large language models' reasoning and problem-solving capabilities by introducing GVGAI-LLM, a video game benchmark based on the General Video Game AI framework, which revealed persistent limitations in spatial reasoning and basic planning through zero-shot evaluations.
We introduce GVGAI-LLM, a video game benchmark for evaluating the reasoning and problem-solving capabilities of large language models (LLMs). Built on the General Video Game AI framework, it features a diverse collection of arcade-style games designed to test a model's ability to handle tasks that differ from most existing LLM benchmarks. The benchmark leverages a game description language that enables rapid creation of new games and levels, helping to prevent overfitting over time. Each game scene is represented by a compact set of ASCII characters, allowing for efficient processing by language models. GVGAI-LLM defines interpretable metrics, including the meaningful step ratio, step efficiency, and overall score, to assess model behavior. Through zero-shot evaluations across a broad set of games and levels with diverse challenges and skill depth, we reveal persistent limitations of LLMs in spatial reasoning and basic planning. Current models consistently exhibit spatial and logical errors, motivating structured prompting and spatial grounding techniques. While these interventions lead to partial improvements, the benchmark remains very far from solved. GVGAI-LLM provides a reproducible testbed for advancing research on language model capabilities, with a particular emphasis on agentic behavior and contextual reasoning.