Multi-Turn Puzzles: Evaluating Interactive Reasoning and Strategic Dialogue in LLMs
This work addresses the need for LLMs to handle real-world interactive scenarios, though it is incremental as it focuses on benchmarking rather than developing new methods.
The authors tackled the problem of LLMs struggling with interactive tasks by introducing a novel benchmark for evaluating multi-turn reasoning and dialogue, revealing significant performance gaps and common error types like poor instruction following and reasoning failures.
Large language models (LLMs) excel at solving problems with clear and complete statements, but often struggle with nuanced environments or interactive tasks which are common in most real-world scenarios. This highlights the critical need for developing LLMs that can effectively engage in logically consistent multi-turn dialogue, seek information and reason with incomplete data. To this end, we introduce a novel benchmark comprising a suite of multi-turn tasks each designed to test specific reasoning, interactive dialogue, and information-seeking abilities. These tasks have deterministic scoring mechanisms, thus eliminating the need for human intervention. Evaluating frontier models on our benchmark reveals significant headroom. Our analysis shows that most errors emerge from poor instruction following, reasoning failures, and poor planning. This benchmark provides valuable insights into the strengths and weaknesses of current LLMs in handling complex, interactive scenarios and offers a robust platform for future research aimed at improving these critical capabilities.