COMPASS: A Multi-Turn Benchmark for Tool-Mediated Planning & Preference Optimization
This work addresses the need for practical benchmarks to measure LLM agents' tool-mediated planning and preference optimization in real-world scenarios like travel planning, though it is incremental as it builds on existing evaluation frameworks.
The authors tackled the problem of evaluating LLM agents on realistic multi-turn planning tasks by introducing COMPASS, a benchmark for travel planning that revealed critical gaps in agents' ability to optimize user preferences and coordinate multi-service tasks, with open-source models showing significant performance collapses.
Real-world large language model (LLM) agents must master strategic tool use and user preference optimization through multi-turn interactions to assist users with complex planning tasks. We introduce COMPASS (Constrained Optimization through Multi-turn Planning and Strategic Solutions), a benchmark that evaluates agents on realistic travel-planning scenarios. We cast travel planning as a constrained preference optimization problem, where agents must satisfy hard constraints while simultaneously optimizing soft user preferences. To support this, we build a realistic travel database covering transportation, accommodation, and ticketing for 20 U.S. National Parks, along with a comprehensive tool ecosystem that mirrors commercial booking platforms. Evaluating state-of-the-art models, we uncover two critical gaps: (i) an acceptable-optimal gap, where agents reliably meet constraints but fail to optimize preferences, and (ii) a plan-coordination gap, where performance collapses on multi-service (flight and hotel) coordination tasks, especially for open-source models. By grounding reasoning and planning in a practical, user-facing domain, COMPASS provides a benchmark that directly measures an agent's ability to optimize user preferences in realistic tasks, bridging theoretical advances with real-world impact.