Detecting Pipeline Failures through Fine-Grained Analysis of Web Agents
This addresses the need for better diagnostic tools to improve web agents for researchers and developers, though it is incremental as it builds on existing frameworks and datasets.
The paper tackles the problem of limited insight into intermediate errors in web agent evaluations by proposing a modular evaluation framework that decomposes agent pipelines into interpretable stages for detailed error analysis. Using the SeeAct framework and Mind2Web dataset, it shows this approach reveals actionable weaknesses missed by standard metrics.
Web agents powered by large language models (LLMs) can autonomously perform complex, multistep tasks in dynamic web environments. However, current evaluations mostly focus on the overall success while overlooking intermediate errors. This limits insight into failure modes and hinders systematic improvement. This work analyzes existing benchmarks and highlights the lack of fine-grained diagnostic tools. To address this gap, we propose a modular evaluation framework that decomposes agent pipelines into interpretable stages for detailed error analysis. Using the SeeAct framework and the Mind2Web dataset as a case study, we show how this approach reveals actionable weaknesses missed by standard metrics - paving the way for more robust and generalizable web agents.