Look Further Ahead: Testing the Limits of GPT-4 in Path Planning
This work addresses the challenge of planning for AI systems, but it is incremental as it tests existing methods on new benchmarks without achieving optimal results.
The paper tackled the problem of long-horizon planning in large language models by evaluating GPT-4 on path planning tasks, finding that prompts framed as Python code and task decomposition improved effectiveness but did not achieve optimal paths or generalize well over extended horizons.
Large Language Models (LLMs) have shown impressive capabilities across a wide variety of tasks. However, they still face challenges with long-horizon planning. To study this, we propose path planning tasks as a platform to evaluate LLMs' ability to navigate long trajectories under geometric constraints. Our proposed benchmark systematically tests path-planning skills in complex settings. Using this, we examined GPT-4's planning abilities using various task representations and prompting approaches. We found that framing prompts as Python code and decomposing long trajectory tasks improve GPT-4's path planning effectiveness. However, while these approaches show some promise toward improving the planning ability of the model, they do not obtain optimal paths and fail at generalizing over extended horizons.