WARC-Bench: Web Archive Based Benchmark for GUI Subtask Executions
This addresses the need for better evaluation of web agents on realistic subtasks, which is incremental but important for robust web planning and navigation.
The authors tackled the problem of evaluating multimodal AI agents on web navigation subtasks by introducing WARC-Bench, a benchmark with 438 tasks using Web ARChive files, where the highest success rate observed was 64.8% for leading models, and they improved open-source models to 52.8% using reinforcement learning with verifiable rewards.
Training web agents to navigate complex, real-world websites requires them to master $\textit{subtasks}$ - short-horizon interactions on multiple UI components (e.g., choosing the correct date in a date picker, or scrolling in a container to extract information). We introduce WARC-Bench (Web Archive Benchmark), a novel web navigation benchmark featuring 438 tasks designed to evaluate multimodal AI agents on subtasks. WARC-Bench enables sandboxed interactions with dynamic and realistic webpages using Web ARChive files. We show that WARC-Bench is challenging for leading computer-use models, with the highest observed success rate being 64.8%. To improve open source models on subtask, we explore two common training techniques: supervised fine-tuning (SFT) and reinforcement learning with verifiable rewards (RLVR). Experiments show that SFT models obtain a 48.8% success rate on the benchmark. Training with RLVR over SFT checkpoints, even in data-scarce settings, improves the score to 52.8% on WARC-Bench, outperforming many frontier models. Our analysis concludes that mastering these subtasks is essential for robust web planning and navigation, and is a capability not extensively evaluated by existing benchmarks.