Webscraper: Leverage Multimodal Large Language Models for Index-Content Web Scraping
This addresses the challenge of brittle and manual web scraping for developers and researchers working with modern web applications, though it is incremental as it builds on existing MLLM capabilities.
The paper tackles the problem of web scraping for dynamic, interactive websites by introducing Webscraper, a framework that uses a Multimodal Large Language Model to navigate interfaces and extract data, achieving a significant improvement in extraction accuracy over a baseline agent on six news websites.
Modern web scraping struggles with dynamic, interactive websites that require more than static HTML parsing. Current methods are often brittle and require manual customization for each site. To address this, we introduce Webscraper, a framework designed to handle the challenges of modern, dynamic web applications. It leverages a Multimodal Large Language Model (MLLM) to autonomously navigate interactive interfaces, invoke specialized tools, and perform structured data extraction in environments where traditional scrapers are ineffective. Webscraper utilizes a structured five-stage prompting procedure and a set of custom-built tools to navigate and extract data from websites following the common ``index-and-content'' architecture. Our experiments, conducted on six news websites, demonstrate that the full Webscraper framework, equipped with both our guiding prompt and specialized tools, achieves a significant improvement in extraction accuracy over the baseline agent Anthropic's Computer Use. We also applied the framework to e-commerce platforms to validate its generalizability.