Cleaner Pretraining Corpus Curation with Neural Web Scraping
This addresses the challenge of curating high-quality pretraining corpora from complex webpages, which is incremental as it improves upon existing scraping techniques.
The paper tackles the problem of extracting clean text from webpages for language model pretraining by introducing NeuScraper, a neural web scraper that outperforms baseline methods by over 20% in improvement.
The web contains large-scale, diverse, and abundant information to satisfy the information-seeking needs of humans. Through meticulous data collection, preprocessing, and curation, webpages can be used as a fundamental data resource for language model pretraining. However, when confronted with the progressively revolutionized and intricate nature of webpages, rule-based/feature-based web scrapers are becoming increasingly inadequate. This paper presents a simple, fast, and effective Neural web Scraper (NeuScraper) to help extract primary and clean text contents from webpages. Experimental results show that NeuScraper surpasses the baseline scrapers by achieving more than a 20% improvement, demonstrating its potential in extracting higher-quality data to facilitate the language model pretraining. All of the code is available at https://github.com/OpenMatch/NeuScraper.