ChessArena: A Chess Testbed for Evaluating Strategic Reasoning Capabilities of Large Language Models
This addresses the problem of assessing genuine reasoning versus pattern recognition in LLMs for researchers, though it is incremental as it focuses on a specific domain testbed.
The paper introduced ChessArena, a chess testbed to evaluate strategic reasoning in large language models (LLMs), revealing that no model could beat a human amateur-level chess engine and some failed against a random player, while a fine-tuned Qwen3-8B model showed improved performance.
Recent large language models (LLMs) have shown strong reasoning capabilities. However, a critical question remains: do these models possess genuine reasoning skills particularly complex strategic reasoning or are they primarily excelling at sophisticated pattern recognition within their training data? To address this question, this paper presents a chess testbed, ChessArena, to evaluate the strategic reasoning capabilities of LLMs. Chess requires complex strategic reasoning capabilities including long-term planning, strict rule comprehension, and multi-turn conversation memorization. Specifically, ChessArena is a competitive framework where LLMs play against each other, under four different play modes. The testbed is equipped with a ranking algorithm and a leaderboard. The testbed can also evaluate fine-grained capabilities including basic understanding, move selection, and puzzle solving. Over 13 LLMs with different modes are evaluated in ChessArena, playing over 800 games. The results reveal significant shortcomings in current LLMs: no model can beat Maia-1100 (a chess engine at human amateur level), while some even failed to defeat a random player that selects moves arbitrarily. We also present a strong baseline to the testbed: our fine-tuned Qwen3-8B substantially improved performance, approaching much larger state-of-the-art reasoning models.