Scaling LLM Planning: NL2FLOW for Parametric Problem Generation and Rigorous Evaluation
This work addresses the problem of scalable evaluation for LLM planning researchers, though it is incremental as it focuses on generating data and benchmarking existing models.
The authors tackled the scarcity of scalable evaluation data for LLM planning by introducing NL2Flow, an automated pipeline for generating and evaluating workflow planning problems, resulting in the best-performing model achieving 86% success in valid plans and 69% in optimal plans on a dataset of 2296 problems.
Robust workflow composition is critical for effective agent performance, yet progress in Large Language Model (LLM) planning and reasoning is hindered by a scarcity of scalable evaluation data. This work introduces NL2Flow, a fully automated pipeline for generating and evaluating workflow planning problems. NL2Flow generates problems parametrically in a structured intermediate representation, translating them into both natural language and formal PDDL. I evaluate several open-source, instruct-tuned LLMs on a dataset of 2296 low-difficulty problems generated by NL2Flow. Results demonstrate that the best-performing model achieved 86% success in generating valid plans and 69% in generating optimal plans (for solvable problems). Regression analysis shows that the influence of problem characteristics on plan generation is contingent on both model and prompt design. Importantly, translating natural language problems into a structured JSON representation prior to symbolic planning significantly improved success rates, suggesting a benefit from neuro-symbolic integration. These findings underscore the importance of understanding error sources within LLM reasoning as systems scale to more complex tasks. As LLM reasoning scales to increasingly complex problems, understanding the shifting bottlenecks and sources of error within these systems will be crucial.