ItinBench: Benchmarking Planning Across Multiple Cognitive Dimensions with Large Language Models
This work addresses the need for more comprehensive evaluation testbeds for LLMs to better reflect real-world challenges, though it is incremental as it builds on existing travel planning benchmarks by adding spatial reasoning.
The paper tackles the problem of evaluating large language models (LLMs) across multiple cognitive dimensions by introducing ItinBench, a benchmark that integrates spatial reasoning (route optimization) with verbal reasoning in trip itinerary planning, and finds that LLMs struggle to maintain consistent performance when handling these tasks simultaneously.
Large language models (LLMs) with advanced cognitive capabilities are emerging as agents for various reasoning and planning tasks. Traditional evaluations often focus on specific reasoning or planning questions within controlled environments. Recent studies have explored travel planning as a medium to integrate various verbal reasoning tasks into real-world contexts. However, reasoning tasks extend beyond verbal reasoning alone, and a comprehensive evaluation of LLMs requires a testbed that incorporates tasks from multiple cognitive domains. To address this gap, we introduce ItinBench, a benchmark that features one task of spatial reasoning, i.e., route optimization, into trip itinerary planning while keeping the traditional verbal reasoning tasks. ItinBench evaluates various LLMs across diverse tasks simultaneously, including Llama 3.1 8B, Mistral Large, Gemini 1.5 Pro, and GPT family. Our findings reveal that LLMs struggle to maintain high and consistent performance when concurrently handling multiple cognitive dimensions. By incorporating tasks from distinct human-level cognitive domains, ItinBench provides new insights into building more comprehensive reasoning testbeds that better reflect real-world challenges. The code and dataset: https://ethanwtl.github.io/IBweb/