BrowserArena: Evaluating LLM Agents on Real-World Web Navigation Tasks
This addresses the need for better evaluation of LLM web agents for researchers and developers, though it is incremental in improving benchmarking methods.
The paper tackled the problem of evaluating LLM web agents in real-world settings by introducing BrowserArena, a live open-web platform that collects user-submitted tasks and uses human feedback to identify failure modes like captcha resolution and pop-up banner removal, finding that models like o4-mini and DeepSeek-R1 exhibit diverse but brittle behaviors.
LLM web agents now browse and take actions on the open web, yet current agent evaluations are constrained to sandboxed environments or artificial tasks. We introduce BrowserArena, a live open-web agent evaluation platform that collects user-submitted tasks, runs Arena-style head-to-head comparisons, and uses step-level human feedback to surface failure modes. Collecting and analyzing step-level annotations on the agent traces, we identify three consistent failure modes: captcha resolution, pop-up banner removal, and direct navigation to URLs. By constructing targeted datasets to further study these tasks, we discover variations in how different language models navigate these failure modes. We find, for example, that o4-mini deploys a wider variety of strategies to circumvent captcha resolution than other models and DeepSeek-R1 consistently misleads users about pop-up banner closure. Our findings surface both the diversity and brittleness of current web agents. More broadly, our benchmarking methodology provides an approach to evaluating and understanding web agent failure modes at scale.