WAREX: Web Agent Reliability Evaluation on Existing Benchmarks
This addresses the gap in robustness evaluation for web agents, which is crucial for developers and users relying on these agents for tasks like online shopping, though it is incremental as it builds on existing benchmarks.
The paper tackles the problem of evaluating browser-based LLM agents in real-world unreliable web environments, showing that introducing WAREX leads to significant drops in task success rates across three benchmarks.
Recent advances in browser-based LLM agents have shown promise for automating tasks ranging from simple form filling to hotel booking or online shopping. Current benchmarks measure agent performance in controlled environments, such as containers or stable networks, where websites behave deterministically. However, in the real world, users access websites over networks and HTTPS connections that introduce instability from multiple sources: client-side, server-side issues or broader system failures. Moreover, live websites are prone to web attacks such Cross-Site Scripting, as well as general site modifications which can cause unexpected or malicious pop-ups or improper functionality. To address this gap, we present WAREX: Web Agent Reliability Evaluation on Existing Benchmarks. We measure the impact of WAREX across three popular benchmarks: WebArena, WebVoyager, and REAL. Our experiments show that introducing WAREX leads to significant drops in task success rates, highlighting the limited robustness of state-of-the-art agents.