Plancraft: an evaluation dataset for planning with LLM agents
This work addresses the need for better evaluation of planning capabilities in LLM agents, though it is incremental as it builds on existing agent architectures and datasets.
The authors introduced Plancraft, a multi-modal evaluation dataset for planning with LLM agents based on Minecraft crafting, and found that LLMs and VLMs struggle with the planning problems it presents, offering suggestions for improvement.
We present Plancraft, a multi-modal evaluation dataset for LLM agents. Plancraft has both a text-only and multi-modal interface, based on the Minecraft crafting GUI. We include the Minecraft Wiki to evaluate tool use and Retrieval Augmented Generation (RAG), as well as a handcrafted planner and Oracle Retriever, to ablate the different components of a modern agent architecture. To evaluate decision-making, Plancraft also includes a subset of examples that are intentionally unsolvable, providing a realistic challenge that requires the agent not only to complete tasks but also to decide whether they are solvable at all. We benchmark both open-source and closed-source LLMs and compare their performance and efficiency to a handcrafted planner. Overall, we find that LLMs and VLMs struggle with the planning problems that Plancraft introduces, and offer suggestions on how to improve their capabilities.