RPGBENCH: Evaluating Large Language Models as Role-Playing Game Engines
This addresses the need for standardized evaluation of LLMs in interactive storytelling for researchers and developers, though it is incremental as it builds on existing LLM-as-a-judge frameworks.
The authors tackled the problem of evaluating large language models (LLMs) as text-based role-playing game engines by introducing RPGBench, a benchmark with tasks for game creation and simulation, and found that state-of-the-art LLMs can produce engaging stories but often struggle with consistent game mechanics in complex scenarios.
We present RPGBench, the first benchmark designed to evaluate large language models (LLMs) as text-based role-playing game (RPG) engines. RPGBench comprises two core tasks: Game Creation (GC) and Game Simulation (GS). In GC, an LLM must craft a valid and playable RPG world using a structured event-state representation, ensuring logical coherence and proper termination conditions. In GS, the LLM simulates interactive gameplay across multiple rounds while consistently updating states and enforcing game rules. To comprehensively assess performance, RPGBench integrates objective and subjective evaluation methodologies. Objective measures verify adherence to event mechanics and check variable updates without requiring human intervention. Subjective measures, such as content interestingness, action quality, and role-playing capability, are evaluated via an LLM-as-a-judge framework, where a strong LLM grades each candidate's outputs. Empirical results demonstrate that state-of-the-art LLMs can produce engaging stories but often struggle to implement consistent, verifiable game mechanics, particularly in long or complex scenarios. By combining structured, rule-based assessments with LLM-based judgments, RPGBench provides a new standard for evaluating how well LLMs can balance creativity, coherence, and complexity in text-based RPGs, opening avenues for more immersive and controllable interactive storytelling.