TripScore: Benchmarking and rewarding real-world travel planning with fine-grained evaluation
This work addresses the need for better evaluation of travel planning in LLMs, though it is incremental as it builds on existing benchmarks by adding more detailed criteria.
The authors tackled the problem of evaluating travel planning by LLMs by introducing a benchmark with fine-grained criteria unified into a single reward, achieving 60.75% agreement with expert annotations and outperforming LLM-as-judge baselines.
Travel planning is a valuable yet complex task that poses significant challenges even for advanced large language models (LLMs). While recent benchmarks have advanced in evaluating LLMs' planning capabilities, they often fall short in evaluating feasibility, reliability, and engagement of travel plans. We introduce a comprehensive benchmark for travel planning that unifies fine-grained criteria into a single reward, enabling direct comparison of plan quality and seamless integration with reinforcement learning (RL). Our evaluator achieves moderate agreement with travel-expert annotations (60.75%) and outperforms multiple LLM-as-judge baselines. We further release a large-scale dataset of 4,870 queries including 219 real-world, free-form requests for generalization to authentic user intent. Using this benchmark, we conduct extensive experiments across diverse methods and LLMs, including test-time computation, neuro-symbolic approaches, supervised fine-tuning, and RL via GRPO. Across base models, RL generally improves itinerary feasibility over prompt-only and supervised baselines, yielding higher unified reward scores.