Dual-View Visual Contextualization for Web Navigation
This work addresses the challenge of building more effective web agents for diverse real-world tasks, representing an incremental improvement over existing methods by integrating visual and textual features for better element contextualization.
The paper tackles the problem of automatic web navigation by addressing the lack of task-related context in HTML documents, proposing to contextualize HTML elements using dual views from webpage screenshots, and achieves consistent performance improvements over baselines on the Mind2Web dataset across cross-task, cross-website, and cross-domain scenarios.
Automatic web navigation aims to build a web agent that can follow language instructions to execute complex and diverse tasks on real-world websites. Existing work primarily takes HTML documents as input, which define the contents and action spaces (i.e., actionable elements and operations) of webpages. Nevertheless, HTML documents may not provide a clear task-related context for each element, making it hard to select the right (sequence of) actions. In this paper, we propose to contextualize HTML elements through their "dual views" in webpage screenshots: each HTML element has its corresponding bounding box and visual content in the screenshot. We build upon the insight -- web developers tend to arrange task-related elements nearby on webpages to enhance user experiences -- and propose to contextualize each element with its neighbor elements, using both textual and visual features. The resulting representations of HTML elements are more informative for the agent to take action. We validate our method on the recently released Mind2Web dataset, which features diverse navigation domains and tasks on real-world websites. Our method consistently outperforms the baseline in all the scenarios, including cross-task, cross-website, and cross-domain ones.