LiveWeb-IE: A Benchmark For Online Web Information Extraction
This addresses the need for more realistic evaluation of WIE systems for applications relying on dynamic web data, though it is incremental in improving benchmark design.
The authors tackled the problem that existing web information extraction (WIE) benchmarks use static HTML snapshots, which fail to generalize to dynamic real-world scenarios, by introducing LiveWeb-IE, a benchmark for evaluating WIE systems directly against live websites, and proposed VGS, a novel multi-stage agentic framework that demonstrates effectiveness and robustness in experiments.
Web information extraction (WIE) is the task of automatically extracting data from web pages, offering high utility for various applications. The evaluation of WIE systems has traditionally relied on benchmarks built from HTML snapshots captured at a single point in time. However, this offline evaluation paradigm fails to account for the temporally evolving nature of the web; consequently, performance on these static benchmarks often fails to generalize to dynamic real-world scenarios. To bridge this gap, we introduce \dataset, a new benchmark designed for evaluating WIE systems directly against live websites. Based on trusted and permission-granted websites, we curate natural language queries that require information extraction of various data categories, such as text, images, and hyperlinks. We further design these queries to represent four levels of complexity, based on the number and cardinality of attributes to be extracted, enabling a granular assessment of WIE systems. In addition, we propose Visual Grounding Scraper (VGS), a novel multi-stage agentic framework that mimics human cognitive processes by visually narrowing down web page content to extract desired information. Extensive experiments across diverse backbone models demonstrate the effectiveness and robustness of VGS. We believe that this study lays the foundation for developing practical and robust WIE systems.