Explore the Reasoning Capability of LLMs in the Chess Testbed
This work addresses the challenge of enhancing reasoning in AI for chess, but it is incremental as it builds on existing methods with new data.
The paper tackled the problem of improving large language models' reasoning in complex tasks like chess by integrating annotated strategy and tactic data, resulting in a finetuned model that outperformed GPT, Claude, and Gemini in selecting better chess moves.
Reasoning is a central capability of human intelligence. In recent years, with the advent of large-scale datasets, pretrained large language models have emerged with new capabilities, including reasoning. However, these models still struggle with long-term, complex reasoning tasks, such as playing chess. Based on the observation that expert chess players employ a dual approach combining long-term strategic play with short-term tactical play along with language explanation, we propose improving the reasoning capability of large language models in chess by integrating annotated strategy and tactic. Specifically, we collect a dataset named MATE, which consists of 1 million chess positions with candidate moves annotated by chess experts for strategy and tactics. We finetune the LLaMA-3-8B model and compare it against state-of-the-art commercial language models in the task of selecting better chess moves. Our experiments show that our models perform better than GPT, Claude, and Gemini models. We find that language explanations can enhance the reasoning capability of large language models.