AI Planning Framework for LLM-Based Web Agents
This work addresses the problem of improving transparency and evaluation for AI researchers and developers working on autonomous web agents, though it is incremental in building on existing planning concepts.
The paper tackles the challenge of diagnosing failures in LLM-based web agents by introducing a planning framework that maps agent architectures to traditional search paradigms, enabling principled analysis of issues like context drift. It validates this with a new dataset and metrics, showing that different architectures excel in different measures, such as a Step-by-Step agent achieving 38% success rate and a Full-Plan-in-Advance agent achieving 89% element accuracy.
Developing autonomous agents for web-based tasks is a core challenge in AI. While Large Language Model (LLM) agents can interpret complex user requests, they often operate as black boxes, making it difficult to diagnose why they fail or how they plan. This paper addresses this gap by formally treating web tasks as sequential decision-making processes. We introduce a taxonomy that maps modern agent architectures to traditional planning paradigms: Step-by-Step agents to Breadth-First Search (BFS), Tree Search agents to Best-First Tree Search, and Full-Plan-in-Advance agents to Depth-First Search (DFS). This framework allows for a principled diagnosis of system failures like context drift and incoherent task decomposition. To evaluate these behaviors, we propose five novel evaluation metrics that assess trajectory quality beyond simple success rates. We support this analysis with a new dataset of 794 human-labeled trajectories from the WebArena benchmark. Finally, we validate our evaluation framework by comparing a baseline Step-by-Step agent against a novel Full-Plan-in-Advance implementation. Our results reveal that while the Step-by-Step agent aligns more closely with human gold trajectories (38% overall success), the Full-Plan-in-Advance agent excels in technical measures such as element accuracy (89%), demonstrating the necessity of our proposed metrics for selecting appropriate agent architectures based on specific application constraints.