AssistantBench: Can Web Agents Solve Realistic and Time-Consuming Tasks?
This work addresses the challenge of assessing web agents for practical applications like real-estate monitoring, but it is incremental as it builds on existing benchmarks and methods.
The authors tackled the problem of evaluating language agents on realistic, time-consuming web tasks by introducing AssistantBench, a benchmark of 214 tasks, and found that current systems, including state-of-the-art web agents, perform poorly with no model exceeding 26% accuracy, while their new SeePlanAct agent significantly outperforms previous ones.
Language agents, built on top of language models (LMs), are systems that can interact with complex environments, such as the open web. In this work, we examine whether such agents can perform realistic and time-consuming tasks on the web, e.g., monitoring real-estate markets or locating relevant nearby businesses. We introduce AssistantBench, a challenging new benchmark consisting of 214 realistic tasks that can be automatically evaluated, covering different scenarios and domains. We find that AssistantBench exposes the limitations of current systems, including language models and retrieval-augmented language models, as no model reaches an accuracy of more than 26 points. While closed-book LMs perform well in terms of accuracy, they exhibit low precision and tend to hallucinate facts. State-of-the-art web agents reach a score of near zero. Additionally, we introduce SeePlanAct (SPA), a new web agent that significantly outperforms previous agents, and an ensemble of SPA and closed-book models reaches the best overall performance. Moreover, we analyze failures of current systems and highlight that open web navigation remains a major challenge.