PLANET: A Collection of Benchmarks for Evaluating LLMs' Planning Capabilities
This work provides a systematic review to help researchers and practitioners compare planning algorithms and select benchmarks, but it is incremental as it synthesizes existing benchmarks without introducing new methods or data.
The paper addresses the lack of comprehensive understanding in planning benchmarks for LLMs by examining and categorizing existing testbeds, recommending appropriate benchmarks for algorithms and guiding future development.
Planning is central to agents and agentic AI. The ability to plan, e.g., creating travel itineraries within a budget, holds immense potential in both scientific and commercial contexts. Moreover, optimal plans tend to require fewer resources compared to ad-hoc methods. To date, a comprehensive understanding of existing planning benchmarks appears to be lacking. Without it, comparing planning algorithms' performance across domains or selecting suitable algorithms for new scenarios remains challenging. In this paper, we examine a range of planning benchmarks to identify commonly used testbeds for algorithm development and highlight potential gaps. These benchmarks are categorized into embodied environments, web navigation, scheduling, games and puzzles, and everyday task automation. Our study recommends the most appropriate benchmarks for various algorithms and offers insights to guide future benchmark development.