MTR-Bench: A Comprehensive Benchmark for Multi-Turn Reasoning Evaluation
This addresses the problem of evaluating interactive reasoning for AI researchers, though it is incremental as it builds on existing benchmarking efforts.
The authors tackled the lack of comprehensive evaluation for multi-turn reasoning in LLMs by introducing MTR-Bench, a benchmark with 4 classes, 40 tasks, and 3600 instances, and found that even state-of-the-art models perform poorly on these interactive tasks.
Recent advances in Large Language Models (LLMs) have shown promising results in complex reasoning tasks. However, current evaluations predominantly focus on single-turn reasoning scenarios, leaving interactive tasks largely unexplored. We attribute it to the absence of comprehensive datasets and scalable automatic evaluation protocols. To fill these gaps, we present MTR-Bench for LLMs' Multi-Turn Reasoning evaluation. Comprising 4 classes, 40 tasks, and 3600 instances, MTR-Bench covers diverse reasoning capabilities, fine-grained difficulty granularity, and necessitates multi-turn interactions with the environments. Moreover, MTR-Bench features fully-automated framework spanning both dataset constructions and model evaluations, which enables scalable assessment without human interventions. Extensive experiments reveal that even the cutting-edge reasoning models fall short of multi-turn, interactive reasoning tasks. And the further analysis upon these results brings valuable insights for future research in interactive AI systems.