A Systematic Study of Large Language Models for Task and Motion Planning With PDDLStream
This work addresses the challenge of understanding LLM planning capabilities for robotics, but it is incremental as it shows current limitations rather than breakthroughs.
The study systematically evaluated 16 algorithms using Gemini 2.5 Flash to integrate large language models into task and motion planning, finding that these LLM-based planners had lower success rates and higher planning times compared to engineered counterparts across 4,950 problems.
Using large language models (LLMs) to solve complex robotics problems requires understanding their planning capabilities. Yet while we know that LLMs can plan on some problems, the extent to which these planning capabilities cover the space of robotics tasks is unclear. One promising direction is to integrate the semantic knowledge of LLMs with the formal reasoning of task and motion planning (TAMP). However, the myriad of choices for how to integrate LLMs within TAMP complicates the design of such systems. We develop 16 algorithms that use Gemini 2.5 Flash to substitute key TAMP components. Our zero-shot experiments across 4,950 problems and three domains reveal that the Gemini-based planners exhibit lower success rates and higher planning times than their engineered counterparts. We show that providing geometric details increases the number of task-planning errors compared to pure PDDL descriptions, and that (faster) non-reasoning LLM variants outperform (slower) reasoning variants in most cases, since the TAMP system can direct the LLM to correct its mistakes.