Does The Way You Plan Matter? An Empirical Study of Planning Representations for LLM Web Agents
For researchers building LLM-based web agents, this work provides empirical evidence that plan representation matters, addressing a previously unexplored factor in agent failures.
The paper investigates how different natural language plan representations affect the performance of LLM-based web agents, finding that plan formulation and the underlying LLM significantly influence task success and robustness, with checklist plans achieving up to 15% higher success rates.
Despite recent advances, LLM-based web agents still struggle with limited exploration, omission of critical steps, and sensitivity to task constraints. Prior work suggests that many of these failures stem from weaknesses in planning, yet the impact of alternative natural language plan representation remains unexplored. To address this, we introduce PlanAhead, a static planner-executor framework that evaluates the impact of plan representation in agent performance. We first automatically categorize WebArena tasks into 3 difficulty levels, enabling consistent difficulty grading without human annotation. Then we systematically evaluate 4 different plan representations on the tasks categorized as hard: sequential subgoals, narrative, pseudocode, and checklist; across different families of multimodal LLM powered agents (OpenAI, Alibaba, and Google). To account for stochastic variability, we introduce two novel evaluation metrics: Achievement Rate (AR) and Solved-Task Consistency (STC). Our results show that both, the plan formulation and the underlying LLM generating the plan, significantly influence web-agent robustness and task success.