Generalized Planning in PDDL Domains with Pretrained Large Language Models
This addresses the challenge of automated planning for AI systems, offering a novel approach that could reduce manual effort, though it is incremental as it builds on existing LLM capabilities.
The paper tackles the problem of using large language models (LLMs) as generalized planners in PDDL domains, finding that GPT-4 can synthesize Python programs from training tasks to efficiently generate plans for other tasks, with automated debugging being crucial and two training tasks often sufficient for strong generalization.
Recent work has considered whether large language models (LLMs) can function as planners: given a task, generate a plan. We investigate whether LLMs can serve as generalized planners: given a domain and training tasks, generate a program that efficiently produces plans for other tasks in the domain. In particular, we consider PDDL domains and use GPT-4 to synthesize Python programs. We also consider (1) Chain-of-Thought (CoT) summarization, where the LLM is prompted to summarize the domain and propose a strategy in words before synthesizing the program; and (2) automated debugging, where the program is validated with respect to the training tasks, and in case of errors, the LLM is re-prompted with four types of feedback. We evaluate this approach in seven PDDL domains and compare it to four ablations and four baselines. Overall, we find that GPT-4 is a surprisingly powerful generalized planner. We also conclude that automated debugging is very important, that CoT summarization has non-uniform impact, that GPT-4 is far superior to GPT-3.5, and that just two training tasks are often sufficient for strong generalization.