What to Ask Next? Probing the Imaginative Reasoning of LLMs with TurtleSoup Puzzles
This work addresses the need for better benchmarks to assess LLMs' exploratory reasoning, which is incremental as it builds on existing evaluation methods by introducing a new framework.
The paper tackles the problem of evaluating Large Language Models' capacity for imaginative reasoning in information-sparse environments by introducing a new benchmark, agent, and evaluation protocol based on Turtle Soup puzzles. The results show clear capability limits and a significant performance gap compared to humans, with experiments conducted on 800 bilingual puzzles.
We investigate the capacity of Large Language Models (LLMs) for imaginative reasoning--the proactive construction, testing, and revision of hypotheses in information-sparse environments. Existing benchmarks, often static or focused on social deduction, fail to capture the dynamic, exploratory nature of this reasoning process. To address this gap, we introduce a comprehensive research framework based on the classic "Turtle Soup" game, integrating a benchmark, an agent, and an evaluation protocol. We present TurtleSoup-Bench, the first large-scale, bilingual, interactive benchmark for imaginative reasoning, comprising 800 turtle soup puzzles sourced from both the Internet and expert authors. We also propose Mosaic-Agent, a novel agent designed to assess LLMs' performance in this setting. To evaluate reasoning quality, we develop a multi-dimensional protocol measuring logical consistency, detail completion, and conclusion alignment. Experiments with leading LLMs reveal clear capability limits, common failure patterns, and a significant performance gap compared to humans. Our work offers new insights into LLMs' imaginative reasoning and establishes a foundation for future research on exploratory agent behavior.