Dr Web: a modern, query-based web data retrieval engine
This provides a tool for developers and researchers needing efficient web data extraction, but it is incremental as it builds on existing query-based retrieval methods.
The paper tackles the problem of extracting structured data from web pages by introducing Dr Web, a flexible and modular tool that uses a simple query language, addressing challenges like dynamic content handling and messy data extraction, and it is made publicly available as open source.
This article introduces the Data Retrieval Web Engine (also referred to as doctor web), a flexible and modular tool for extracting structured data from web pages using a simple query language. We discuss the engineering challenges addressed during its development, such as dynamic content handling and messy data extraction. Furthermore, we cover the steps for making the DR Web Engine public, highlighting its open source potential.